Oracle ERP数据库恢复案例一则-备份重于一切

近日，帮助某客户处理了一起Oracle ERP数据库恢复案例，恢复过程较为简单，客户拥有完整的全备份以及归档日志，可以执行安全恢复，最后遇到的问题是，中间某个归档日志损坏，Oracle无法识别，客户允许放弃几个小时的数据，这样就顺利的完成了恢复。

归档日志损坏，就是我之前提到的：
http://www.eygle.com/archives/2010/11/recover_archivelog_corruption.html

这个案例，唯一存在的疑点是，为什么数据库会出现问题？

从告警日志看来，数据库在完全正常的运行过程中，忽然出现了ORA-600 2662错误，以下是所有的错误信息，在多次出现错误之后，数据库Crash掉，此后未能启动成功：

Wed May 10 17:12:45 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005408990], [1388], [4005425099], [1484804288], [], []
Wed May 10 17:12:47 2010
Non-fatal internal error happenned while SMON was doing flushing of monitored table stats.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Wed May 10 17:12:47 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005408993], [1388], [4005425099], [1484804288], [], []
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 2 out of maximum 100 non-fatal internal errors.
Wed May 10 17:12:50 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005409007], [1388], [4005425099], [1484804288], [], []
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 3 out of maximum 100 non-fatal internal errors.
Wed May 10 17:12:52 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005409009], [1388], [4005425099], [1484804288], [], []
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 4 out of maximum 100 non-fatal internal errors.
Wed May 10 17:12:54 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005409012], [1388], [4005425099], [1484804288], [], []
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 5 out of maximum 100 non-fatal internal errors.
Wed May 10 17:13:06 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005409030], [1388], [4005425099], [1484804288], [], []
Wed May 10 17:13:07 2010
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 6 out of maximum 100 non-fatal internal errors.
Wed May 10 17:13:17 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005409048], [1388], [4005425099], [1484804288], [], []
Wed May 10 17:13:18 2010
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 7 out of maximum 100 non-fatal internal errors.
Wed May 10 17:13:28 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005409102], [1388], [4005425099], [1484804288], [], []
Wed May 10 17:13:29 2010
Non-fatal internal error happenned while SMON was doing extent coalescing.
SMON encountered 8 out of maximum 100 non-fatal internal errors.
Wed May 10 17:13:39 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_smon_14821.trc:
ORA-00600: internal error code, arguments: [2662], [1388], [4005409119], [1388], [4005425099], [1484804288], [], []
Wed May 10 17:13:41 2010
Errors in file /ORACLE/ERP/LOG/ADMIN/bdump/erp_pmon_14807.trc:
ORA-00474: SMON process terminated with error
Wed May 10 17:13:41 2010
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 14807

检查错误日志：

ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [2662], [1388], [4005408990], [1388], [4005425099], [1484804288], [], []
Current SQL statement for this session:
update sys.mon_mods$ set inserts = inserts + :ins, updates = updates + :upd, deletes = deletes + :del, flags = (decode(bitand(flags, :flag), :flag, flags, flags + :flag)), drop_segments = drop_segments + :dropseg, timestamp = :time where obj# = :objn

Oracle是在进行mon_mods表的维护时出现了不一致，导致2662错误的出现。
2662这个错误本质上仍然是好处理的，不过要看后台隐藏的深层原因，这个故障我怀疑是因为OS级别的写丢失、写异常导致的。
随后用户进行了一系列的恢复尝试，导致当前库不可用。

最后在进行实例恢复时，系统表空间块和Redo块出现了不一致：

Thu Nov 11 15:39:24 2010
Errors in file /bdump/pcerp_p005_27334.trc:
ORA-00600: internal error code, arguments: [3020], [1], [117393], [4311697], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 1, block# 117393)
ORA-10564: tablespace SYSTEM
ORA-01110: data file 1: '/data/system01.dbf'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 368
Thu Nov 11 15:39:24 2010
Errors in file /bdump/pcerp_p000_27324.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [2], [123902], [6101], [], [], [], []
Thu Nov 11 15:39:25 2010
Errors in file /bdump/pcerp_p005_27334.trc:
ORA-00600: internal error code, arguments: [3020], [1], [117393], [4311697], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 1, block# 117393)
ORA-10564: tablespace SYSTEM
ORA-01110: data file 1: '/data/system01.dbf'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 368

最为常见的，只有写丢失、写异常才可能导致此类问题。

记录一下这些错误信息供参考。