这两台服务器用的都是联志的国产低端PC Server，这些服务器的质量实在是差，上次一台备机的硬盘损坏，然后又有一台因为电源模块的问题反复重起，现在这一台服务器的硬盘再次出现问题。

Nov 24 10:27:48 wapcom1 kernel: attempt to access beyond end of device
Nov 24 10:27:48 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191
Nov 24 10:27:48 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:
 directory #128110 contains a hole at offset 2011258880
Nov 24 10:27:49 wapcom1 kernel: attempt to access beyond end of device
Nov 24 10:27:49 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191
Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:
 directory #128110 contains a hole at offset 2011262976
Nov 24 10:27:50 wapcom1 kernel: attempt to access beyond end of device
Nov 24 10:27:50 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191
Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:
 directory #128110 contains a hole at offset 2011267072
Nov 24 10:27:50 wapcom1 kernel: attempt to access beyond end of device
Nov 24 10:27:50 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191
Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:
 directory #128110 contains a hole at offset 2011271168

好在数据库通过DataGuard可以切换到另外一台，没有数据损失：

Thu Nov 23 18:46:18 2006
ARC0: Complete FAL archive (thread 1 sequence 6045 destination bmarksb)
ARC0: Begin FAL archive (thread 1 sequence 6047 destination bmarksb)
Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'
ARC0: Complete FAL archive (thread 1 sequence 6047 destination bmarksb)
ARC0: Begin FAL archive (thread 1 sequence 6048 destination bmarksb)
Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'
Thu Nov 23 18:46:18 2006
ARC1: Complete FAL archive (thread 1 sequence 6046 destination bmarksb)
ARC1: Begin FAL archive (thread 1 sequence 6049 destination bmarksb)
Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'
Thu Nov 23 18:46:18 2006
ARC0: Complete FAL archive (thread 1 sequence 6048 destination bmarksb)
Thu Nov 23 18:46:18 2006
ARC1: Complete FAL archive (thread 1 sequence 6049 destination bmarksb)

现在是主库所在的服务器出现问题:

SQL> select dbid,name,PROTECTION_MODE,DATABASE_ROLE,SWITCHOVER_STATUS from v$database;

      DBID NAME      PROTECTION_MODE      DATABASE_ROLE    SWITCHOVER_STATUS
---------- --------- -------------------- ---------------- ------------------
3520694939 BMARK     MAXIMUM PERFORMANCE  PRIMARY          SESSIONS ACTIVE

备库现在一切正常:

SQL> select dbid,name,PROTECTION_MODE,DATABASE_ROLE,SWITCHOVER_STATUS from v$database;

      DBID NAME      PROTECTION_MODE      DATABASE_ROLE    SWITCHOVER_STATUS
---------- --------- -------------------- ---------------- ------------------
3520694939 BMARK     MAXIMUM PERFORMANCE  PHYSICAL STANDBY SESSIONS ACTIVE

现在需要的是一点停机时间进行切换。

切换日志:

Fri Nov 24 11:30:43 2006
alter database commit to switchover to physical standby with session shutdown
Fri Nov 24 11:30:43 2006
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY
Fri Nov 24 11:30:43 2006
SMON: disabling tx recovery
Fri Nov 24 11:30:44 2006
Active process 26743 user 'oracle' program 'oracle@wapcom1.hawa.cn (CJQ0)'
Active process 9033 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
Active process 7655 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
...............
Active process 8944 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
Active process 29104 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
Active process 30750 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
Active process 9045 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
CLOSE: waiting for server sessions to complete.
Fri Nov 24 11:31:51 2006
CLOSE: all sessions shutdown successfully.
Fri Nov 24 11:32:09 2006
SMON: disabling cache recovery
Fri Nov 24 11:32:10 2006
Shutting down archive processes
Archiving is disabled
Fri Nov 24 11:32:10 2006
ARCH shutting down
Fri Nov 24 11:32:10 2006
ARCH shutting down
Fri Nov 24 11:32:10 2006
ARC0: Archival stopped
Fri Nov 24 11:32:10 2006
ARC1: Archival stopped
Fri Nov 24 11:32:10 2006
Thread 1 closed at log sequence 6076
Successful close of redo thread 1
Fri Nov 24 11:32:28 2006
ARCH: noswitch archival of thread 1, sequence 6076
ARCH: End-Of-Redo archival of thread 1 sequence 6076
ARCH: Evaluating archive log 3 thread 1 sequence 6076
ARCH: Beginning to archive log 3 thread 1 sequence 6076
Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'
Creating archive destination LOG_ARCHIVE_DEST_1: '/var/oradata/arch/1_6076.arc'
ARCH: Completed archiving log 3 thread 1 sequence 6076
ARCH: archiving is disabled due to current logfile archival
Clearing standby activation ID 3520937155 (0xd1dd3cc3)
The primary database controlfile was created using the
'MAXLOGFILES 5' clause.
The resulting standby controlfile will not have enough
available logfile entries to support an adequate number
of standby redo logfiles. Consider re-creating the
primary controlfile using 'MAXLOGFILES 8' (or larger).
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 10485760;
Archivelog for thread 1 sequence 6076 required for standby recovery
MRP0 started with pid=8
MRP0: Background Managed Standby Recovery process started
Media Recovery Log /var/oradata/arch/1_6076.arc
Identified end-of-REDO for thread 1 sequence 6076
Identified end-of-REDO for thread 1 sequence 6076
Media Recovery End-Of-Redo indicator encountered
Media Recovery Applied until change 194025715
MRP0: Media Recovery Complete: End-Of-REDO
Resetting standby activation ID 3520937155 (0xd1dd3cc3)
MRP0: Background Media Recovery process shutdown
Fri Nov 24 11:32:35 2006
Switchover: Complete - Database shutdown required
Completed: alter database commit to switchover to physical st
Fri Nov 24 11:32:53 2006
Shutting down instance: further logons disabled
Shutting down instance (immediate)
License high water mark = 140
Fri Nov 24 11:32:53 2006
ALTER DATABASE CLOSE NORMAL
ORA-1507 signalled during: ALTER DATABASE CLOSE NORMAL...
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Fri Nov 24 11:33:14 2006
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.6.0.
System parameters with non-default values:
processes = 150
timed_statistics = TRUE
shared_pool_size = 83886080
large_pool_size = 33554432
standby_archive_dest = /var/oradata/arch
fal_server = bmarksb
fal_client = bmark
log_archive_format = %t_%s.arc
...........
CJQ0 started with pid=8
Fri Nov 24 11:33:15 2006
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=9
ARC0: Archival started
ARC1 started with pid=10
Fri Nov 24 11:33:15 2006
ARCH: STARTING ARCH PROCESSES COMPLETE
Fri Nov 24 11:33:15 2006
ARC1: Archival started
Fri Nov 24 11:33:15 2006
ARC0: Thread not mounted
Fri Nov 24 11:33:15 2006
ARC1: Thread not mounted
Fri Nov 24 11:33:22 2006
alter database mount standby database
Fri Nov 24 11:33:26 2006
Successful mount of redo thread 1, with mount id 3559140162
Fri Nov 24 11:33:26 2006
Standby Database mounted.
Completed: alter database mount standby database
Fri Nov 24 11:33:29 2006
ALTER DATABASE RECOVER managed standby database disconnect
Attempt to start background Managed Standby Recovery process
MRP0 started with pid=12
MRP0: Background Managed Standby Recovery process started
Fri Nov 24 11:33:34 2006
Completed: ALTER DATABASE RECOVER managed standby database d
Fri Nov 24 11:33:34 2006
Media Recovery Waiting for thread 1 seq# 6077
Media Recovery Log /var/oradata/arch/1_6077.arc
Media Recovery Waiting for thread 1 seq# 6078
Media Recovery Log /var/oradata/arch/1_6078.arc
Media Recovery Waiting for thread 1 seq# 6079

看来以后不能再采购联志服务器了。

-The End-

作者提到：

　刚才同事告诉我，以前我的顶头上司，IT经理引咎辞职了，仔细一问，原来是我的继任没有做备份，资料全部损毁
　
　这也是今年我第2次见到这种情况

这种情况其实我们已经看到过很多次了。

如果拿这个具体案例来说，其实有很多环节可以避免出现这样的问题，比如:
1.良好的规范管理
2.严格的操作及上线流程
3.DBA的职责界定及监督检查机制
4.系统的日常监控及维护机制...

当然有很多很多可能防止问题出现的方法，可惜最终问题仍然发生了。

这又一次验证了墨菲定律，这个世界上没有永远的侥幸。
在我的新书《深入浅出Oracle》一书的序言中我也曾写到：
唯一一件会使DBA在梦中惊醒的事情就是：没有备份！

在我曾经授课的岁月里，我总是会在课程的最前面讲到DBA的四大守则，其中第一条就是：备份重于一切。

而在我的网站上，这句话已经重复了很多次，昨天看Tom的Blog，Tom提到了他的法则，在原文中，Tom用了一个词mantra，这个词在金山词霸上如下解释：

mantra
颂歌, 咒语(尤指四吠陀经典内作为咒文或祷告唱念的)

在昨天的文章中，被我翻译成法则，而我觉得如果翻译成咒语也满合适的，如果我一次一次的重复能够让所有的DBA们都记得，那么我仍然愿意重复我的DBA四大守则:

1.备份重于一切 我们必需知道,系统总是要崩溃的，没有有效的备份只是等哪一天死！我经常开玩笑的说,唯一会使DBA在梦中惊醒的就是,没有有效的备份.

2.三思而后行
think thrice before you act

任何时候都要清楚你所做的一切，否则宁可不做！有时候一个回车,一条命令就会造成不可恢复的灾难,所以,你必需清楚确认你所做的一切,并且在必要时保护现场.

3.rm是危险的
要知道在UNIX/Linux下，这个操作意味着你可能将永远失去后面的东西，所以，确认你的操作！！！
太多的人在 "rm -rf" 上悲痛欲绝,当年写下这条守则时,是一个凌晨被一个朋友吵醒,他说误操作rm -rf删除掉了200G的数据库,并且没有备份.

我当时能告诉他的只有一句话:要保持冷静.

4.你来制定规范
良好的规范是减少故障的基础。所以,做为一个DBA,你需要来制订规范,规范开发甚至系统人员,这样甚至可以规避有意或是无意的误操作.减少数据库的风险.

最早写下这四大守则时，还受到我们某位国家领导人的影响，在指导防火工作时，他曾经题写过以下名词:
隐患险于明火，防范胜于救灾，责任重于泰山

这句话对于DBA来说，同样适用。在某种程度上，DBA就是消防队员。：）

最后，我们还可以来看看DCBA对这件事情的看法。

-The End-

首先启动实例:

[oracle@stat ~]$ export ORACLE_SID=order
[oracle@stat ~]$ rman target /

Recovery Manager: Release 10.2.0.2.0 - Production on Thu Sep 14 22:43:50 2006

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database (not started)

RMAN> set DBID=1341966532

executing command: SET DBID

RMAN> startup nomount;

Oracle instance started

Total System Global Area 2483027968 bytes

Fixed Size 1262344 bytes
Variable Size 654314744 bytes
Database Buffers 1811939328 bytes
Redo Buffers 15511552 bytes

恢复数据文件并加载(mount)数据库:

RMAN> restore controlfile from autobackup;

Starting restore at 14-SEP-06
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=541 devtype=DISK

channel ORA_DISK_1: looking for autobackup on day: 20060914
channel ORA_DISK_1: autobackup found: c-1341966532-20060914-02
channel ORA_DISK_1: control file restore from autobackup complete
output filename=/oradata/controlfile/o1_mf_28spy45z_.ctl
output filename=/oradata/controlfile/o2_mf_28spy45z_.ctl
Finished restore at 14-SEP-06

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

Restore数据库:

RMAN> restore database;

Starting restore at 14-SEP-06
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=541 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /oradata/datafile/o1_mf_system_28spy7kl_.dbf
restoring datafile 00002 to /oradata/datafile/o1_mf_undotbs1_28spykdh_.dbf
restoring datafile 00003 to /oradata/datafile/o1_mf_sysaux_28spyo9s_.dbf
restoring datafile 00004 to /oradata/datafile/o1_mf_users_28spyvm8_.dbf
restoring datafile 00005 to /oradata/datafile/o1_mf_vascms_2c444bhj_.dbf
restoring datafile 00006 to /oradata/datafile/o1_mf_wapgame_2c44gz55_.dbf
restoring datafile 00007 to /oradata/datafile/o1_mf_vascms_2c4kn0b2_.dbf
channel ORA_DISK_1: reading from backup piece /data3/ordrbak/full_ORDER_20060913_169
channel ORA_DISK_1: restored backup piece 1
piece handle=/data3/ordrbak/orderfullback_ORDER_20060913_169 tag=order
channel ORA_DISK_1: restore complete, elapsed time: 00:03:06
Finished restore at 14-SEP-06

进行基于时间点的恢复：

RMAN> recover database until time '2006-09-14 19:00:00'
2> ;

Starting recover at 14-SEP-06
using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 09/14/2006 22:49:54
ORA-01861: literal does not match format string

RMAN> exit

Recovery Manager complete.

这个错误是由于时间日期格式设置的问题。

设置正确的时间格式，进行基于时间点的不完全恢复:

[oracle@stat ~]$ export NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'
[oracle@stat ~]$ rman target /

Recovery Manager: Release 10.2.0.2.0 - Production on Thu Sep 14 22:50:22 2006

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database: order (DBID=1341966532, not open)

RMAN> recover database until time '2006-09-14 19:00:00'
2> ;

Starting recover at 2006-09-14 22:50:26
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=544 devtype=DISK

starting media recovery

archive log thread 1 sequence 303 is already on disk as file
/oradata/archive/1_303_592917188.dbf
archive log thread 1 sequence 304 is already on disk as file
/oradata/archive/1_304_592917188.dbf
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=299
channel ORA_DISK_1: reading from backup piece /data3/ordrbak/arch_order_20060913_171
channel ORA_DISK_1: restored backup piece 1
piece handle=/data3/ordrbak/orderarch_order_20060913_171 tag=order
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
archive log filename=/oradata/archive/1_299_592917188.dbf thread=1 sequence=299
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=300
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=301
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=302
channel ORA_DISK_1: reading from backup piece /data3/ordrbak/arch_order_20060914_173
channel ORA_DISK_1: restored backup piece 1
piece handle=/data3/ordrbak/orderarch_order_20060914_173 tag=TAG20060914T033004
channel ORA_DISK_1: restore complete, elapsed time: 00:00:08
archive log filename=/oradata/archive/1_300_592917188.dbf thread=1 sequence=300
archive log filename=/oradata/archive/1_301_592917188.dbf thread=1 sequence=301
archive log filename=/oradata/archive/1_302_592917188.dbf thread=1 sequence=302
archive log filename=/oradata/archive/1_303_592917188.dbf thread=1 sequence=303
archive log filename=/oradata/archive/1_304_592917188.dbf thread=1 sequence=304
media recovery complete, elapsed time: 00:00:57
Finished recover at 2006-09-14 22:51:39

Resetlogs打开数据库:

RMAN> alter database open resetlogs;

database opened

RMAN>

此时可以检查数据的正确性，如果无误就可以通过exp导出数据，再imp进生产数据库，完成恢复。

在有了充分的备份的前提下，这样的恢复是非常容易的。
这个故事再次告诉我们：备份重于一切。

-The End-

 动手试一下，大概就是如下步骤:

-The End-

仔细看了一下帖子内容,发现作者执行了如下几条命令:

然后数据库无法打开.

我们注意到,在第一条命令中,作者offline drop掉了SYSTEM表空间中的数据文件.不管之前怎样,现在的数据库肯定是无法启动了.

在作者提供的日志中,可以看到之前数据库的错误提示:

是因为磁盘空间不足导致数据库错误.

而作者在不清楚具体原因时,竟然贸然offline drop了重要的数据文件.这一行为非常草率.做为DBA我们不仅不能想当然,也不能无知者无畏.

当然,offline drop的文件,可以通过重建控制文件的方式重新加入数据库,再尝试正常的恢复.

最后仍然要重复之前那句话:无知者不能无畏.

请问sys.file$数据字典文件被truncate掉后是否能够恢复
昨天太大意了，一不小心把file$中的内容给删除了，造成tablespace里的数据文件列表看不到了，请各位大侠帮忙！是否能用那几个建库脚本将这个表的内容恢复？

我们知道数据字典对于数据库来说至关重要,通常建议不要手工对数据字典进行任何修改和变更.因为一个简单的修改可能引发数据库内部很多潜在的问题.

除非在Oracle技术支持的指导下,对字典的手工修改实在没有必要.对于DBA来说,大家需要谨记:

绝对不要手工修改数据字典

如果发生如上意外(当然这个意外也太离奇了点),最好能够从备份中恢复(当然有些字典表是能够通过insert等简单操作恢复的),如果不存在备份,一个需要知道的经验是,千万不要关闭数据库,尝试导出数据,最后如果无法恢复,可以通过重建,imp导入数据恢复.

后来这位朋友电话给我,因为是测试环境,同时没有备份,我建议他导出重建数据库来解决.

回顾错误的过程，这位朋友说：

我还以为 props$这个表可以在mount 下面update 呢

对"以为"这两个字，我是深恶痛绝的，作为DBA，一定要严谨，千万不能想当然。想当然的轻率对于数据库来说可能是灾难。

把这个故事记录在这里，为大家稍作警戒。

数据库为 9.2.0.7.0 ,OS : Solaris Operating System (SPARC 64-bit)

起因是这样的，我的一客户那里UPS出现故障导致系统宕机，
然后起来，大约过了10来分钟，突然操作系统找不到磁盘又一次宕机，
然后再起来，有用户报一个SQL用不上索引.

这个SQL是这样的：

select * from ww.test20060504 dg where dg.user_number='7290'

第一个想法是给那个索引做分析，但还是不行，我们就对这个表做了一次分析，但执行计划没有什么改变 。
我们尝试加提示（包括加 rule )，但也不行，用户反映是有一批这样类似的都用不到索引。
然后通过 10053 做 trace 居然发现优化器根本没有考虑索引。开始怀疑这个数据库的数据字典可能有问题。
我们只好用一个笨方法，将其中一个表导到测试库上去测试，在导出的过程中居然发现系统报错

EXP-00056: ORACLE error 6552 encountered
ORA-06552: PL/SQL: Compilation unit analysis terminated
ORA-06553: PLS-553: character set name is not recognized

居然系统报字符集的错！
.....

真是晕呀！我们马上仔细检查了一个 alter 文件发现了一条信息，系统在第一次宕机起来后就自已将 controlfile 中的字符集给更改了。最后我们将系统中的字符集改回来系统就恢复正常了。

警告日志中的信息是这样的:

SMON: enabling tx recovery
Mon Jun 5 09:52:52 2006
Updating character set in controlfile to ZHS16CGB231280
replication_dependency_tracking turned off (no async multimaster replication found)

其实这个信息是手工更新过数据库字符集后,重新启动,数据库比较数据库和控制文件信息,根据数据库字符集修改控制文件字符集导致的.

我在以前作过这样的测试,参考:

http://www.eygle.com/special/NLS_CHARACTER_SET_03.htm

通过更新props$的方式修改字符集是非常危险的,我在以上的文章中有过详细说明,在Oracle8i中,如果修改了错误的字符集,那么重新启动后数据库将无法启动.

如果需要提醒的话,我们需要再警世一次:

绝对不要用update系统表(props$)的方式来修改数据库字符集.

但是从Oracle9i开始,Oracle在启动时跳过了这个检查,即使修改了错误的字符集,也仍然可以启动,数据库启动时会将控制文件中的字符集更改为缺省的US7ASCII.

具体可以看看以下的测试:

此时警告日志中会记录如下信息:

不同版本中,Oracle行为已经不同.

这就是DBA警世录的由来.

今天看到有朋友记下了这样一个案例:

因为要导两个表的数据到测试库,结果在产品库上用了Truncate......
更糟的是客户首先发现了问题 而不是自己 自己以为目标是
测试库............

总结:
1. 谨慎&细心
操作涉及产品库慎之再慎
2. 产品库和测试库有相同的user/pw(这在某种程度上造成了假象)

ps:此次事件被定性为生产事故 严重

这样的案例很多见，因为测试环境和生产环境混淆而导致的误Delete,误Truncate操作经常发生。除了DBA不够严谨之外，制度上没有保证也是问题之一。

这位同学总结的很好，通常我们的测试库和产品库应该设置不同的用户密码，不同的SID，在进行重要操作时，应该先select instance_name from v$instance命令验证一下当前连接的例程：

这就如同我们在Unix/Linux主机上应该经常用hostname来确认一下当前连接的主机一样。

如果在本地登陆，我们还可以通过修改本地glogin.sql文件，显示当前连接的实例等信息。

总之，在执行任何数据变更操作之前，我们都应当谨慎。这是对于DBA的基本要求之一。

参考连接:
生产事故 http://www.itpub.net/533262.html  

Sat Jan 21 13:55:21 2006
Errors in file /opt/oracle/admin/conner/bdump/conner_smon_17113.trc:
ORA-00600: internal error code, arguments: [4194], [43], [46], [], [], [], [], []
Sat Jan 21 13:55:21 2006
Errors in file /opt/oracle/admin/conner/udump/conner_ora_17121.trc:
ORA-00600: internal error code, arguments: [4194], [45], [44], [], [], [], [], [] 

4194错误通常说明UNDO段出现问题,最好的办法是通过备份进行恢复,如果没有备份,那么可以通过特殊的初始化参数进行强制启动,本文就Oracle的隐含参数进行恢复说明(由于实际情况可能各不相同，进行测试前请先行备份)，仅供参考。

首先确定当前回滚段名称，这可以从alert文件中获得:

Sat Jan 21 13:55:21 2006
Undo Segment 11 Onlined
Undo Segment 12 Onlined
Undo Segment 13 Onlined
Successfully onlined Undo Tablespace 16. 

对应的AUM (auto undo management) 下的回滚段名称为: 

'_SYSSMU11$','_SYSSMU12$','_SYSSMU13$'

 修改init<sid>.ora参数文件，使用Oracle隐含参数_corrupted_rollback_segments将回滚段标记为损坏，此时启动数据库，Oracle会跳过对于这些回滚段的相关操作，强制启动数据库。

._corrupted_rollback_segments='_SYSSMU11$','_SYSSMU12$','_SYSSMU13$'

误删表空间
System损坏
又一个System损坏

再加上今日,dcba又报道了一例redo损坏的重大事故:

在open时删除了所有的联机log后能起来吗?

在这里我不想就具体技术细节进行评论,我只想说说我对这些问题的看法.在我反复面试我的DBA们时,经常我会谈到:

在我们这里,你不会遇到太复杂的恢复情况,一个管理良好的数据库始终能够从容的从事故或灾难中恢复出来,所以比较起来,在复杂的技术和严谨的态度之间,我们更需要你的严谨.

这也是我对DBA的一个基本要求,如果你足够严谨,以上的情况你可能都不需要面对,最严重的情况,你还有有效的备份可以恢复.在我主讲的DBA课程中,我曾经提到的DBA四大守则里,第一守则就是:

备份重于一切
          系统总是要崩溃的，没有有效的备份只是等哪一天死！

然而可惜的是大多数人都在运行无备份的数据库系统,这是DBA的悲哀.

Fri Dec 16 22:37:27 2005
Errors in file /opt/oracle/admin/conner/bdump/conner_smon_22817.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4193], [1171], [1187], [], [], [], [], []
Fri Dec 16 23:28:40 2005
Errors in file /opt/oracle/admin/conner/bdump/conner_smon_22817.trc:
ORA-00600: internal error code, arguments: [4193], [1171], [1187], [], [], [], [], []

4193错误通常是因为恢复时redo与undo不一致所导致。

Oracle的解释如下:

  While backing out an undo record (i.e. at the time of rollback) we found a  transaction id mis-match indicating either a corruption in the rollback   segment or corruption in an object which the rollback segment is trying to  apply undo records on.

  This would indicate a corrupted rollback segment.

检查具体的Trace文件，可以发现类似如下错误：

*** 2005-12-16 20:54:53.496
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [4193], [1171], [1187], [], [], [], [], []
Current SQL statement for this session:
UPDATE SMON_SCN_TIME SET SCN_WRP=:1, SCN_BAS=:2, TIME_MP=:3, TIME_DP=:4 
WHERE TIME_MP = :5  AND   THREAD = :6  AND   ROWNUM <= 1

当然模拟2662错误需要技巧，本文并不会涉及这个内容。

通过正常方式启动数据库时，从alert文件中，我们可以看到ora-00600 2662号错误。

Sun Dec 11 18:02:25 2005
Errors in file /opt/oracle/admin/conner/udump/conner_ora_13349.trc:
ORA-00600: internal error code, arguments: [2662], [0], [547743994], [0], [898092653], [8388617], [], []
Sun Dec 11 18:02:27 2005
Errors in file /opt/oracle/admin/conner/udump/conner_ora_13349.trc:
ORA-00600: internal error code, arguments: [2662], [0], [547743994], [0], [898092653], [8388617], [], []
Sun Dec 11 18:02:27 2005
Error 600 happened during db open, shutting down database
USER: terminating instance due to error 600

此时我们可以通过Oracle的内部事件来调整SCN:

增进SCN有两种常用方法:

1.通过immediate trace name方式(在数据库Open状态下)

alter session set events 'IMMEDIATE trace name ADJUST_SCN level x';

2.通过10015事件(在数据库无法打开，mount状态下)

alter session set events '10015 trace name adjust_scn level x';

注:level 1为增进SCN 10亿 (1 billion) (1024*1024*1024),通常Level 1已经足够。也可以根据实际情况适当调整。

本例由于数据库无法打开，只能使用的二种方法。

[oracle@jumper dbs]$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.4.0 - Production on Sun Dec 11 18:26:18 2005

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile=initconner.ora
ORACLE instance started.

Total System Global Area   97588504 bytes
Fixed Size                   451864 bytes
Variable Size              33554432 bytes
Database Buffers           62914560 bytes
Redo Buffers                 667648 bytes
Database mounted.

SQL> alter session set events '10015 trace name adjust_scn level 10';

Session altered.

SQL> alter database open;

Database altered.

注意,由于我使用了10015事件，使得SCN增进了10 billion，稍后我们可以验证。 

[oracle@jumper dbs]$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.4.0 - Production on Sun Dec 11 18:26:18 2005

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile=initconner.ora
ORACLE instance started.

Total System Global Area   97588504 bytes
Fixed Size                   451864 bytes
Variable Size              33554432 bytes
Database Buffers           62914560 bytes
Redo Buffers                 667648 bytes
Database mounted.

SQL> alter session set events '10015 trace name adjust_scn level 10';

Session altered.

SQL> alter database open;

Database altered.

此时数据库可以打开，从alert文件中我们可以看到如下提示:

Sun Dec 11 18:27:04 2005 SMON: enabling cache recovery Sun Dec 11 18:27:05 2005 Debugging event used to advance scn to 10737418240

SCN被增进了10 billion,即 10 * (1024*1024*1024) = 10737418240,正好是日志里记录的数量。

ORA-19815: WARNING: db_recovery_file_dest_size of 53687091200 bytes is 85.00% used,
            and has 8052259328 remaining bytes available.
            *************************************************************
            You have the following choices to free up space from
flash recovery area:
1. Consider changing your RMAN retention policy.
   If you are using dataguard, then consider changing your
   RMAN archivelog deletion policy.
2. Backup files to tertiary device such as tape using the
   RMAN command BACKUP RECOVERY AREA.
3. Add disk space and increase the db_recovery_file_dest_size
   parameter to reflect the new space.
4. Delete unncessary files using the RMAN DELETE command.
   If an OS command was used to delete files, then use
   RMAN CROSSCHECK and DELETE EXPIRED commands.

db_recovery_file_dest_size设置的是50G,在当前的备份策略下已经不足够.只好临时扩展一下恢复区:

 SQL> show parameter recov

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /msflsh
db_recovery_file_dest_size           big integer 50G
recovery_parallelism                 integer     0
SQL> alter system set db_recovery_file_dest_size=65G scope=both;

System altered.

SQL> show parameter recov

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /msflsh
db_recovery_file_dest_size           big integer 65G
recovery_parallelism                 integer     0

再修改下冗余策略,释放部分磁盘空间: