« 纪念我的第一次淘宝购物 | Blog首页 | Oracle中独一无二的Cache对象 »
光纤通道故障导致数据库崩溃
作者：eygle |【转载时请以超链接形式标明文章出处和作者信息及本声明】链接：http://www.eygle.com/archives/2007/04/fibre_channel_lgwr.html
早上一到公司，同事告知，昨晚一台SUN4500上的数据库Crash了。
故障是由于光纤通道问题导致，在系统日志中记录了如下信息：
socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.5010] socal1: port 1: Fibre Channel is OFFLINE
scsi: [ID 243001 kern.warning] WARNING: /sbus@3,0/SUNW,socal@0,0/sf@1,0 (sf3):
Offline Timeout
scsi: [ID 243001 kern.info] /sbus@3,0/SUNW,socal@0,0/sf@1,0 (sf3):
target 0x1 al_pa 0xe8 lun 0 offlined
scsi: [ID 107833 kern.warning] WARNING: /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w50020f2300007f86,0 (ssd0):
ssdrestart transport failed (fffffffe)
socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.6010] socal1: port 1: Fibre Channel Loop is ONLINE
socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.5010] socal1: port 1: Fibre Channel is OFFLINE
socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.6010] socal1: port 1: Fibre Channel Loop is ONLINE
数据库日志中记录如下信息：
Tue Apr 17 23:12:46 2007
Thread 1 advanced to log sequence 572945
Current log# 2 seq# 572945 mem# 0: /u01/oracle/oradata/hysms02/redo02.log
LGWR: terminating instance due to error 340
Wed Apr 18 01:36:08 2007
KCF: write/open error block=0x455f5 online=1
file=75 /u01/oracle8/oradata/hysms02/rbs02.dbf
error=27072 txt: 'SVR4 Error: 5: I/O error
Additional information: 284149'
Wed Apr 18 01:36:08 2007
Instance terminated by LGWR, pid = 527
故障过程是光纤通道Offline，导致LGWR写失败，LGWR中止了数据库。随后光纤通道自动恢复正常，数据库能够重新启动，经过如下恢复过程：
Wed Apr 18 02:44:37 2007
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
Wed Apr 18 02:44:40 2007
Thread recovery: start rolling forward thread 1
Recovery of Online Redo Log: Thread 1 Group 2 Seq 572945 Reading mem 0
Mem# 0 errs 0: /u01/oracle/oradata/hysms02/redo02.log
Wed Apr 18 02:46:35 2007
Thread recovery: finish rolling forward thread 1
Thread recovery: 5129 blocks read, 4998 blocks written
Crash recovery completed successfully
Picked broadcast on commit scheme to generate SCNs
这台SUN 4500 + T3已经服务了5年多，进入了故障多发期，故障发生时，该主机已经持续运行了497天左右：
oracle:/oracle/oracle8>uptime
9:51am up 497 day(s), 10:50, 6 users, load average: 3.21, 3.06, 3.09
-The End-
历史上的今天...
>> 2006-04-18文章:
>> 2005-04-18文章:
------
这篇 【光纤通道故障导致数据库崩溃】来自 www.eygle.com | CSDN技术网摘| del.icio.us|365Key
By eygle on 2007-04-18 09:25 | Comments (6) | Posted to Case | Edit |Pageviews:
留言 (6)
问题不大哈,呵呵
只是通道的问题
那天控制器坏了就有事干了.
对了,你不是双通道的吗?
如果是双通道,应该可以避免一个通道挂,db就挂的问题的吧.
Posted by: David.Guo at April 18, 2007 11:07 AM
IOPS多少？俺们的SUN E250+ A1000，up1730 ，工作还是良好的。sun是值得托付的，除了io负载低点，比pcserver亦不如
Posted by: jacky at April 18, 2007 2:18 PM
这个是老设备了，只有单通道。T3比A1000要强一些，不过都是老设备了，容量很有限。
一般倒还稳定
Posted by: eygle at April 18, 2007 5:44 PM
由于存储一共只配了16 port，今天还在考虑到底是其中两个数据库共用4 port还是分别用2port，没想到还有 1 port 跑5年的
Posted by: biti_rainy at April 18, 2007 11:33 PM
唉，穷人也有穷人的活法。
Posted by: eygle at April 19, 2007 8:51 AM
这个,和ali系的暴发户不一样的哈
上次巴乔还问我为什么不用光线交换机
这个,这个不是这玩意贵吗.
咋穷人只能单光线通道直连.
Posted by: David.Guo at April 19, 2007 10:47 AM
