光纤通道故障导致数据库崩溃

« 纪念我的第一次淘宝购物 | Blog首页 | Oracle中独一无二的Cache对象 »

早上一到公司，同事告知，昨晚一台SUN4500上的数据库Crash了。
故障是由于光纤通道问题导致，在系统日志中记录了如下信息：

socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.5010] socal1: port 1: Fibre Channel is OFFLINE
scsi: [ID 243001 kern.warning] WARNING: /sbus@3,0/SUNW,socal@0,0/sf@1,0 (sf3):
Offline Timeout
scsi: [ID 243001 kern.info] /sbus@3,0/SUNW,socal@0,0/sf@1,0 (sf3):
target 0x1 al_pa 0xe8 lun 0 offlined
scsi: [ID 107833 kern.warning] WARNING: /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w50020f2300007f86,0 (ssd0):
ssdrestart transport failed (fffffffe)
socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.6010] socal1: port 1: Fibre Channel Loop is ONLINE
socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.5010] socal1: port 1: Fibre Channel is OFFLINE
socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.6010] socal1: port 1: Fibre Channel Loop is ONLINE

数据库日志中记录如下信息：

Tue Apr 17 23:12:46 2007
Thread 1 advanced to log sequence 572945
Current log# 2 seq# 572945 mem# 0: /u01/oracle/oradata/hysms02/redo02.log
LGWR: terminating instance due to error 340
Wed Apr 18 01:36:08 2007
KCF: write/open error block=0x455f5 online=1
file=75 /u01/oracle8/oradata/hysms02/rbs02.dbf
error=27072 txt: 'SVR4 Error: 5: I/O error
Additional information: 284149'
Wed Apr 18 01:36:08 2007
Instance terminated by LGWR, pid = 527

故障过程是光纤通道Offline，导致LGWR写失败，LGWR中止了数据库。随后光纤通道自动恢复正常，数据库能够重新启动，经过如下恢复过程：

Wed Apr 18 02:44:37 2007
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
Wed Apr 18 02:44:40 2007
Thread recovery: start rolling forward thread 1
Recovery of Online Redo Log: Thread 1 Group 2 Seq 572945 Reading mem 0
Mem# 0 errs 0: /u01/oracle/oradata/hysms02/redo02.log
Wed Apr 18 02:46:35 2007
Thread recovery: finish rolling forward thread 1
Thread recovery: 5129 blocks read, 4998 blocks written
Crash recovery completed successfully
Picked broadcast on commit scheme to generate SCNs

这台SUN 4500 + T3已经服务了5年多，进入了故障多发期，故障发生时，该主机已经持续运行了497天左右：