昨日在客户现场一套RAC在启动监听的时候报TNS-12545 TNS-12560 TNS-00515错误，经过处理后恢复正常，
今天在自己的虚拟机上还原了这个问题，记录下：
OS:Linux
DB:10.2.0.4 RAC
一、模拟恢复到事故发生时的情况
1、手工关闭监听后切换VIP
[Oracle@orlrac1 ~]$
[oracle@orlrac1 ~]$ srvctl stop listener -n orlrac1
[oracle@orlrac1 ~]$ srvctl stop listener -n orlrac2
[oracle@orlrac1 ~]$
[oracle@orlrac1 ~]$
[oracle@orlrac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcldb.db application ONLINE ONLINE orlrac1
ora....b1.inst application ONLINE ONLINE orlrac1
ora....b2.inst application ONLINE ONLINE orlrac2
ora....SM1.asm application ONLINE ONLINE orlrac1
ora....C1.lsnr application OFFLINE OFFLINE
ora....ac1.gsd application ONLINE ONLINE orlrac1
ora....ac1.ons application ONLINE ONLINE orlrac1
ora....ac1.vip application ONLINE ONLINE orlrac1
ora....SM2.asm application ONLINE ONLINE orlrac2
ora....C2.lsnr application OFFLINE OFFLINE
ora....ac2.gsd application ONLINE ONLINE orlrac2
ora....ac2.ons application ONLINE ONLINE orlrac2
ora....ac2.vip application ONLINE ONLINE orlrac2
[oracle@orlrac1 ~]$
2、手工切换vip ------问题的关键，昨天的问题是两个节点的VIP不知道什么原因互换了
[oracle@orlrac1 ~]$ crs_relocate ora.orlrac1.vip -c orlrac2
Attempting to stop `ora.orlrac1.vip` on member `orlrac1`
Stop of `ora.orlrac1.vip` on member `orlrac1` succeeded.
Attempting to start `ora.orlrac1.vip` on member `orlrac2`
Start of `ora.orlrac1.vip` on member `orlrac2` succeeded.
[oracle@orlrac1 ~]$
[oracle@orlrac1 ~]$
[oracle@orlrac1 ~]$ crs_relocate ora.orlrac2.vip -c orlrac1
Attempting to stop `ora.orlrac2.vip` on member `orlrac2`
Stop of `ora.orlrac2.vip` on member `orlrac2` succeeded.
Attempting to start `ora.orlrac2.vip` on member `orlrac1`
Start of `ora.orlrac2.vip` on member `orlrac1` succeeded.
[oracle@orlrac1 ~]$
[oracle@orlrac1 ~]$
[oracle@orlrac1 ~]$
[oracle@orlrac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcldb.db application ONLINE ONLINE orlrac1
ora....b1.inst application ONLINE ONLINE orlrac1
ora....b2.inst application ONLINE ONLINE orlrac2
ora....SM1.asm application ONLINE ONLINE orlrac1
ora....C1.lsnr application OFFLINE OFFLINE
ora....ac1.gsd application ONLINE ONLINE orlrac1
ora....ac1.ons application ONLINE ONLINE orlrac1
ora....ac1.vip application ONLINE ONLINE orlrac2
ora....SM2.asm application ONLINE ONLINE orlrac2
ora....C2.lsnr application OFFLINE OFFLINE
ora....ac2.gsd application ONLINE ONLINE orlrac2
ora....ac2.ons application ONLINE ONLINE orlrac2
ora....ac2.vip application ONLINE ONLINE orlrac1
[oracle@orlrac1 ~]$
----查看IP是否已经切换
[root@orlrac1 ~]#
[root@orlrac1 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 08:00:27:9F:BD:A4
inet addr:192.168.100.111 Bcast:192.168.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1485 errors:0 dropped:0 overruns:0 frame:0
TX packets:992 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:141378 (138.0 KiB) TX bytes:119168 (116.3 KiB)
eth0:1 Link encap:Ethernet HWaddr 08:00:27:9F:BD:A4
inet addr:192.168.100.114 Bcast:192.168.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1 Link encap:Ethernet HWaddr 08:00:27:05:6D:5E
inet addr:172.168.1.101 Bcast:172.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:103726 errors:0 dropped:0 overruns:0 frame:0
TX packets:69201 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:79639044 (75.9 MiB) TX bytes:26738291 (25.4 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:12392 errors:0 dropped:0 overruns:0 frame:0
TX packets:12392 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6477913 (6.1 MiB) TX bytes:6477913 (6.1 MiB)
[root@orlrac1 ~]#
[root@orlrac2 ~]#
[root@orlrac2 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 08:00:27:A2:A1:2A
inet addr:192.168.100.113 Bcast:192.168.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:471 errors:0 dropped:0 overruns:0 frame:0
TX packets:266 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:49614 (48.4 KiB) TX bytes:27620 (26.9 KiB)
eth0:2 Link encap:Ethernet HWaddr 08:00:27:A2:A1:2A
inet addr:192.168.100.112 Bcast:192.168.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1 Link encap:Ethernet HWaddr 08:00:27:25:23:D7
inet addr:172.168.1.102 Bcast:172.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:69257 errors:0 dropped:0 overruns:0 frame:0
TX packets:103814 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:26754117 (25.5 MiB) TX bytes:79662388 (75.9 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:10450 errors:0 dropped:0 overruns:0 frame:0
TX packets:10450 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6388467 (6.0 MiB) TX bytes:6388467 (6.0 MiB)
[root@orlrac2 ~]#
[root@orlrac2 ~]#
[oracle@orlrac1 admin]$
[oracle@orlrac1 admin]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.100.111 orlrac1 orlrac1.oracle.com
192.168.100.112 orlrac1-vip
192.168.100.113 orlrac2 orlrac2.oracle.com
192.168.100.114 orlrac2-vip
172.168.1.101 orlrac1-priv
172.168.1.102 orlrac2-priv
[oracle@orlrac1 admin]$
----两个节点的VIP已经交互