clusterware和Oracle10gr2软件升级到10.2.0.4时,重启系统后,节点一crs无法启动, crsctl start crs后系统立即重启。
以下是crs 和 css的日志记录。
crsd.log:
2012-12-25 08:11:56.757: [ CSSCLNT][1226828528]clsssInitNative: connect failed, rc 9
2012-12-25 08:11:56.757: [ CRSRTI][1226828528]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2012-12-25 08:11:58.252: [ COMMCRS][1099401536]clsc_connect: (0xe18010) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_crs))
2012-12-25 08:11:58.252: [ CSSCLNT][1226828528]clsssInitNative: connect failed, rc 9
2012-12-25 08:11:58.252: [ CRSRTI][1226828528]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2012-12-25 08:11:59.789: [ COMMCRS][1099401536]clsc_connect: (0xe18010) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_crs))
2012-12-25 08:11:59.789: [ CSSCLNT][1226828528]clsssInitNative: connect failed, rc 9
2012-12-25 08:11:59.789: [ CRSRTI][1226828528]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2012-12-25 08:12:01.586: [ COMMCRS][1099401536]clsc_connect: (0xe18010) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_crs))
2012-12-25 08:12:01.586: [ CSSCLNT][1226828528]clsssInitNative: connect failed, rc 9
2012-12-25 08:12:01.586: [ CRSRTI][1226828528]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2012-12-25 08:12:04.174: [ COMMCRS][1099401536]clsc_connect: (0xe18010) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_crs))
2012-12-25 08:12:04.174: [ CSSCLNT][1226828528]clsssInitNative: connect failed, rc 9
2012-12-25 08:12:04.175: [ CRSRTI][1226828528]0CSS is not ready. Received status 3 from CSS. Waiting for good status ..
ocssd.log:
[ CSSD]2012-12-25 09:58:03.233 >USER: Copyright 2012, Oracle version 10.2.0.4.0
[ CSSD]2012-12-25 09:58:03.233 >USER: CSS daemon log for node rac1, number 1, in cluster crs
[ clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_CSSD))
[ CSSD]2012-12-25 09:58:03.337 [547869936] >TRACE: clssscmain: local-only set to false
[ CSSD]2012-12-25 09:58:03.351 [547869936] >TRACE: clssnmReadNodeInfo: added node 1 (rac1) to cluster
[ CSSD]2012-12-25 09:58:03.386 [547869936] >TRACE: clssnmReadNodeInfo: added node 2 (rac2) to cluster
[ CSSD]2012-12-25 09:58:04.159 [1138325824] >TRACE: clssnm_skgxninit: Compatible vendor clusterware not in use
[ CSSD]2012-12-25 09:58:04.159 [1138325824] >TRACE: clssnm_skgxnmon: skgxn init failed
[ CSSD]2012-12-25 09:58:04.341 [547869936] >TRACE: clssnmNMInitialize: misscount set to (300)
[ CSSD]2012-12-25 09:58:04.342 [547869936] >TRACE: clssnmNMInitialize: Network heartbeat thresholds are: impending reconfig 150000 ms, reconfig start (misscount) 300000 ms
[ CSSD]2012-12-25 09:58:04.350 [547869936] >TRACE: clssnmDiskStateChange: state from 1 to 2 disk (0//dev/raw/raw4)
[ CSSD]2012-12-25 09:58:04.350 [1138325824] >TRACE: clssnmvDPT: spawned for disk 0 (/dev/raw/raw4)
[ CSSD]2012-12-25 09:58:06.389 [1138325824] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (0//dev/raw/raw4)
[ CSSD]2012-12-25 09:58:06.457 [547869936] >TRACE: clssnmFatalInit: fatal mode enabled
[ CSSD]2012-12-25 09:58:06.522 [1148815680] >TRACE: clssnmvKillBlockThread: spawned for disk 0 (/dev/raw/raw4) initial sleep interval (1000)ms
[ CSSD]2012-12-25 09:58:06.531 [1169795392] >TRACE: clssnmClusterListener: Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=rac1-priv)(PORT=49895))
[ CSSD]2012-12-25 09:58:06.542 [1169795392] >TRACE: clssnmClusterListener: Probing node rac2 (2), probcon(0x1422bd90)
[ CSSD]2012-12-25 09:58:06.582 [1169795392] >TRACE: clssnmConnComplete: MSGSRC 2, type 6, node 2, flags 0x0001, con 0x1422bd90, probe 0x1422bd90
[ CSSD]2012-12-25 09:58:06.582 [1169795392] >TRACE: clssnmConnComplete: node 2, rac2, con(0x1422bd90), probcon(0x1422bd90), ninfcon((nil)), node unique 1356444601, prev unique 0, msg unique 1356444601 node state 0
[ CSSD]2012-12-25 09:58:06.582 [1169795392] >TRACE: clssnmConnComplete: connected to node 2 (con 0x1422bd90), ninfcon (0x1422bd90), state (0), flag (1037)
[ CSSD]2012-12-25 09:58:06.594 [1138325824] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(2) wrtcnt(2797) LATS(207944) Disk lastSeqNo(2797)
[ CSSD]2012-12-25 09:58:06.756 [1092946240] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_1))
[ CSSD]2012-12-25 09:58:06.756 [1092946240] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_crs))
[ CSSD]2012-12-25 09:58:06.817 [1201264960] >TRACE: clssgmPeerListener: Listening on (ADDRESS=(PROTOCOL=tcp)(DEV=20)(HOST=10.0.0.154)(PORT=33670))
[ CSSD]2012-12-25 09:58:08.725 [1169795392] >TRACE: clssnmHandleSync: diskTimeout set to (297000)ms
[ CSSD]2012-12-25 09:58:08.725 [1169795392] >TRACE: clssnmHandleSync: Acknowledging sync: src[2] srcName[rac2] seq[0] sync[2]
[ CSSD]2012-12-25 09:58:08.725 [1232734528] >TRACE: clssnmRcfgMgrThread: initial lastleader(2) unique(1356444601)