这两台服务器用的都是联志的国产低端PC Server，这些服务器的质量实在是差，上次一台备机的硬盘损坏，然后又有一台因为电源模块的问题反复重起，现在这一台服务器的硬盘再次出现问题。

Nov 24 10:27:48 wapcom1 kernel: attempt to access beyond end of device
Nov 24 10:27:48 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191
Nov 24 10:27:48 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:
 directory #128110 contains a hole at offset 2011258880
Nov 24 10:27:49 wapcom1 kernel: attempt to access beyond end of device
Nov 24 10:27:49 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191
Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:
 directory #128110 contains a hole at offset 2011262976
Nov 24 10:27:50 wapcom1 kernel: attempt to access beyond end of device
Nov 24 10:27:50 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191
Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:
 directory #128110 contains a hole at offset 2011267072
Nov 24 10:27:50 wapcom1 kernel: attempt to access beyond end of device
Nov 24 10:27:50 wapcom1 kernel: 08:08: rw=0, want=1564747716, limit=5245191
Nov 24 10:27:50 wapcom1 kernel: EXT3-fs error (device sd(8,8)): ext3_readdir:
 directory #128110 contains a hole at offset 2011271168

好在数据库通过DataGuard可以切换到另外一台，没有数据损失：

Thu Nov 23 18:46:18 2006
ARC0: Complete FAL archive (thread 1 sequence 6045 destination bmarksb)
ARC0: Begin FAL archive (thread 1 sequence 6047 destination bmarksb)
Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'
ARC0: Complete FAL archive (thread 1 sequence 6047 destination bmarksb)
ARC0: Begin FAL archive (thread 1 sequence 6048 destination bmarksb)
Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'
Thu Nov 23 18:46:18 2006
ARC1: Complete FAL archive (thread 1 sequence 6046 destination bmarksb)
ARC1: Begin FAL archive (thread 1 sequence 6049 destination bmarksb)
Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'
Thu Nov 23 18:46:18 2006
ARC0: Complete FAL archive (thread 1 sequence 6048 destination bmarksb)
Thu Nov 23 18:46:18 2006
ARC1: Complete FAL archive (thread 1 sequence 6049 destination bmarksb)

现在是主库所在的服务器出现问题:

SQL> select dbid,name,PROTECTION_MODE,DATABASE_ROLE,SWITCHOVER_STATUS from v$database;

      DBID NAME      PROTECTION_MODE      DATABASE_ROLE    SWITCHOVER_STATUS
---------- --------- -------------------- ---------------- ------------------
3520694939 BMARK     MAXIMUM PERFORMANCE  PRIMARY          SESSIONS ACTIVE

备库现在一切正常:

SQL> select dbid,name,PROTECTION_MODE,DATABASE_ROLE,SWITCHOVER_STATUS from v$database;

      DBID NAME      PROTECTION_MODE      DATABASE_ROLE    SWITCHOVER_STATUS
---------- --------- -------------------- ---------------- ------------------
3520694939 BMARK     MAXIMUM PERFORMANCE  PHYSICAL STANDBY SESSIONS ACTIVE

现在需要的是一点停机时间进行切换。

切换日志:

Fri Nov 24 11:30:43 2006
alter database commit to switchover to physical standby with session shutdown
Fri Nov 24 11:30:43 2006
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY
Fri Nov 24 11:30:43 2006
SMON: disabling tx recovery
Fri Nov 24 11:30:44 2006
Active process 26743 user 'oracle' program 'oracle@wapcom1.hawa.cn (CJQ0)'
Active process 9033 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
Active process 7655 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
...............
Active process 8944 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
Active process 29104 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
Active process 30750 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
Active process 9045 user 'oracle' program 'oracle@wapcom1.hawa.cn (TNS V1-V3)'
CLOSE: waiting for server sessions to complete.
Fri Nov 24 11:31:51 2006
CLOSE: all sessions shutdown successfully.
Fri Nov 24 11:32:09 2006
SMON: disabling cache recovery
Fri Nov 24 11:32:10 2006
Shutting down archive processes
Archiving is disabled
Fri Nov 24 11:32:10 2006
ARCH shutting down
Fri Nov 24 11:32:10 2006
ARCH shutting down
Fri Nov 24 11:32:10 2006
ARC0: Archival stopped
Fri Nov 24 11:32:10 2006
ARC1: Archival stopped
Fri Nov 24 11:32:10 2006
Thread 1 closed at log sequence 6076
Successful close of redo thread 1
Fri Nov 24 11:32:28 2006
ARCH: noswitch archival of thread 1, sequence 6076
ARCH: End-Of-Redo archival of thread 1 sequence 6076
ARCH: Evaluating archive log 3 thread 1 sequence 6076
ARCH: Beginning to archive log 3 thread 1 sequence 6076
Creating archive destination LOG_ARCHIVE_DEST_2: 'bmarksb'
Creating archive destination LOG_ARCHIVE_DEST_1: '/var/oradata/arch/1_6076.arc'
ARCH: Completed archiving log 3 thread 1 sequence 6076
ARCH: archiving is disabled due to current logfile archival
Clearing standby activation ID 3520937155 (0xd1dd3cc3)
The primary database controlfile was created using the
'MAXLOGFILES 5' clause.
The resulting standby controlfile will not have enough
available logfile entries to support an adequate number
of standby redo logfiles. Consider re-creating the
primary controlfile using 'MAXLOGFILES 8' (or larger).
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 10485760;
Archivelog for thread 1 sequence 6076 required for standby recovery
MRP0 started with pid=8
MRP0: Background Managed Standby Recovery process started
Media Recovery Log /var/oradata/arch/1_6076.arc
Identified end-of-REDO for thread 1 sequence 6076
Identified end-of-REDO for thread 1 sequence 6076
Media Recovery End-Of-Redo indicator encountered
Media Recovery Applied until change 194025715
MRP0: Media Recovery Complete: End-Of-REDO
Resetting standby activation ID 3520937155 (0xd1dd3cc3)
MRP0: Background Media Recovery process shutdown
Fri Nov 24 11:32:35 2006
Switchover: Complete - Database shutdown required
Completed: alter database commit to switchover to physical st
Fri Nov 24 11:32:53 2006
Shutting down instance: further logons disabled
Shutting down instance (immediate)
License high water mark = 140
Fri Nov 24 11:32:53 2006
ALTER DATABASE CLOSE NORMAL
ORA-1507 signalled during: ALTER DATABASE CLOSE NORMAL...
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Fri Nov 24 11:33:14 2006
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.6.0.
System parameters with non-default values:
processes = 150
timed_statistics = TRUE
shared_pool_size = 83886080
large_pool_size = 33554432
standby_archive_dest = /var/oradata/arch
fal_server = bmarksb
fal_client = bmark
log_archive_format = %t_%s.arc
...........
CJQ0 started with pid=8
Fri Nov 24 11:33:15 2006
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=9
ARC0: Archival started
ARC1 started with pid=10
Fri Nov 24 11:33:15 2006
ARCH: STARTING ARCH PROCESSES COMPLETE
Fri Nov 24 11:33:15 2006
ARC1: Archival started
Fri Nov 24 11:33:15 2006
ARC0: Thread not mounted
Fri Nov 24 11:33:15 2006
ARC1: Thread not mounted
Fri Nov 24 11:33:22 2006
alter database mount standby database
Fri Nov 24 11:33:26 2006
Successful mount of redo thread 1, with mount id 3559140162
Fri Nov 24 11:33:26 2006
Standby Database mounted.
Completed: alter database mount standby database
Fri Nov 24 11:33:29 2006
ALTER DATABASE RECOVER managed standby database disconnect
Attempt to start background Managed Standby Recovery process
MRP0 started with pid=12
MRP0: Background Managed Standby Recovery process started
Fri Nov 24 11:33:34 2006
Completed: ALTER DATABASE RECOVER managed standby database d
Fri Nov 24 11:33:34 2006
Media Recovery Waiting for thread 1 seq# 6077
Media Recovery Log /var/oradata/arch/1_6077.arc
Media Recovery Waiting for thread 1 seq# 6078
Media Recovery Log /var/oradata/arch/1_6078.arc
Media Recovery Waiting for thread 1 seq# 6079

看来以后不能再采购联志服务器了。

-The End-

这次损坏的是Redo日志,数据库警告日志给出Redo相关的错误信息:

Mon Nov 13 11:42:54 2006
Errors in file /opt/oracle/admin/mydb/udump/mydb_ora_16682.trc:
ORA-00333: redo log read error block 186498 count 6144
ORA-00312: online log 2 thread 1: '/opt/oracle/oradata/mydb/redo02.log'
ORA-27072: skgfdisp: I/O error
Linux Error: 2: No such file or directory
Additional information: 186497
Mon Nov 13 11:42:58 2006
Errors in file /opt/oracle/admin/mydb/udump/mydb_ora_16682.trc:
ORA-00333: redo log read error block 184450 count 8192
ORA-00312: online log 2 thread 1: '/opt/oracle/oradata/mydb/redo02.log'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
Linux Error: 2: No such file or directory
Additional information: 186498
Mon Nov 13 11:43:03 2006
Errors in file /opt/oracle/admin/mydb/udump/mydb_ora_16682.trc:
ORA-00333: redo log read error block 184450 count 8192
ORA-00312: online log 2 thread 1: '/opt/oracle/oradata/mydb/redo02.log'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
Linux Error: 2: No such file or directory
Additional information: 186498

相关的跟踪文件记录了类似的错误信息:

[oracle@gdmstest bdump]$ cat /opt/oracle/admin/mydb/udump/mydb_ora_16682.trc
/opt/oracle/admin/mydb/udump/mydb_ora_16682.trc
Oracle9i Enterprise Edition Release 9.2.0.4.0 - Production
With the Partitioning option
JServer Release 9.2.0.4.0 - Production
ORACLE_HOME = /opt/oracle/product/9.2.0
System name: Linux
Node name: gdmstest.hurray.com.cn
Release: 2.4.21-15.EL
Version: #1 Thu Apr 22 00:27:41 EDT 2004
Machine: i686
Instance name: mydb
Redo thread mounted by this instance: 1
Oracle process number: 11
Unix process pid: 16682, image: oracle@gdmstest.hurray.com.cn (TNS V1-V3)

*** SESSION ID:(9.3) 2006-11-13 11:41:23.555
Thread checkpoint rba:0x00001d.00000002.0010 scn:0x0000.000f94cd
On-disk rba:0x00001d.0002dc60.0000 scn:0x0000.000f9b4e
Use incremental checkpoint cache-low RBA
Thread 1 recovery from rba:0x00001d.00029082.0000 scn:0x0000.00000000
*** 2006-11-13 11:42:54.830
ORA-00333: redo log read error block 186498 count 6144
ORA-00312: online log 2 thread 1: '/opt/oracle/oradata/mydb/redo02.log'
ORA-27072: skgfdisp: I/O error
Linux Error: 2: No such file or directory
Additional information: 186497
ORA-00333: redo log read error block 184450 count 8192
ORA-00312: online log 2 thread 1: '/opt/oracle/oradata/mydb/redo02.log'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
Linux Error: 2: No such file or directory
Additional information: 186498
ORA-00333: redo log read error block 184450 count 8192
ORA-00312: online log 2 thread 1: '/opt/oracle/oradata/mydb/redo02.log'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
Linux Error: 2: No such file or directory
Additional information: 186498
ORA-00333: redo log read error block 184450 count 8192
ORA-00312: online log 2 thread 1: '/opt/oracle/oradata/mydb/redo02.log'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
Linux Error: 2: No such file or directory
Additional information: 186498

察看系统提示，发现存在问题的扇区（Sector）和上次相同(sector=14266880)，看来真的是物理损坏，只能更换硬盘了:

[oracle@gdmstest bdump]$ dmesg
or=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880

-The End-

发现以下两个命令很有用:

LSNRCTL> set current_listener <listener name>
LSNRCTL> set log_file <sid name>.log

使用set current_listener可以访问非缺省监听器，使用set log_file更改名称后，原来有问题的日志文件可以清除或实现日志重定位:

[oracle@jumper admin]$ lsnrctl

LSNRCTL for Linux: Version 9.2.0.4.0 - Production on 10-NOV-2006 16:54:16

Copyright (c) 1991, 2002, Oracle Corporation.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set current_listener LISTENER1
Current Listener is LISTENER1
LSNRCTL> set log_file
Parameter Value: a.log
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
LISTENER1 parameter "log_file" set to a.log
The command completed successfully
LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER1
Version                   TNSLSNR for Linux: Version 9.2.0.4.0 - Production
Start Date                10-NOV-2006 16:54:12
Uptime                    0 days 0 hr. 2 min. 6 sec
Trace Level               off
Security                  OFF
SNMP                      OFF
Listener Parameter File   /opt/oracle/product/9.2.0/network/admin/listener.ora
Listener Log File         /opt/oracle/product/9.2.0/network/log/a.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.16.33.11)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "conner" has 1 instance(s).
  Instance "conner", status UNKNOWN, has 1 handler(s) for this service...
Service "eygle" has 1 instance(s).
  Instance "eygle", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
LSNRCTL>

如果需要将这个修改永久化，需要使用save_config命令保存一下：

LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
Saved LISTENER1 configuration parameters.
Listener Parameter File /opt/oracle/product/9.2.0/network/admin/listener.ora
Old Parameter File /opt/oracle/product/9.2.0/network/admin/listener.bak
The command completed successfully

此时listener.ora文件会被增加如下记录：

[oracle@jumper oracle]$ tail -5 /opt/oracle/product/9.2.0/network/admin/listener.ora

#----ADDED BY TNSLSNR 14-NOV-2006 16:39:12---
LOG_FILE_LISTENER1 = a.log
#--------------------------------------------

记录一下。

-The End-

同样是硬件故障，存放数据库软件及数据文件的磁盘出现问题，导致数据库Down机。
登陆数据库服务器检查可以发现:

$ df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t10d0s0 494235 95149 349663 22% /
/dev/dsk/c0t10d0s6 4384710 2160661 2180202 50% /usr
/proc 0 0 0 0% /proc
mnttab 0 0 0 0% /etc/mnttab
fd 0 0 0 0% /dev/fd
/dev/dsk/c0t10d0s1 1018191 586987 370113 62% /var
swap 3703192 96 3703096 1% /var/run
swap 4133440 430344 3703096 11% /tmp
/dev/dsk/c4t1d0s0 120514012 100868307 18440565 85% /data1
/dev/dsk/c0t10d0s5 8261393 2365474 5813306 29% /opt
/dev/dsk/c0t11d0s2 17348866 17229 17158149 1% /backup
/dev/dsk/c0t10d0s4 586515 21157 506707 5% /export/home
$ cd /data1
$ ls
.: I/O error

数据库Mount点data1已经不可以访问，I/O error的提示一般意味着磁盘出现问题。

这时我们可以通过一个系统命令dmesg来进行系统信息察看。
dmesg - collect system diagnostic messages to form error log

dmesg主要发现如下错误：

$ dmesg

Nov 1 23:58:10 stat socal: [ID 403145 kern.info] ID[SUNWssa.socal.link.5010] socal1: port 1: Fibre Channel is OFFLINE
Nov 1 23:58:56 stat scsi: [ID 243001 kern.warning] WARNING: /sbus@3,0/SUNW,socal@0,0/sf@1,0 (sf3):
Nov 1 23:58:56 stat Offline Timeout
Nov 1 23:58:56 stat scsi: [ID 243001 kern.info] /sbus@3,0/SUNW,socal@0,0/sf@1,0 (sf3):
Nov 1 23:58:56 stat target 0x1 al_pa 0xe8 lun 0 offlined
Nov 1 23:58:56 stat scsi: [ID 107833 kern.warning] WARNING: /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w50020f2300009321,0 (ssd0):
Nov 1 23:58:56 stat SCSI transport failed: reason 'reset': retrying command
Nov 1 23:58:56 stat scsi: [ID 107833 kern.warning] WARNING: /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w50020f2300009321,0 (ssd0):
Nov 1 23:58:56 stat transport rejected fatal error
Nov 1 23:58:56 stat ufs: [ID 702911 kern.warning] WARNING: Error writing master during ufs log roll
Nov 1 23:58:56 stat ufs: [ID 127457 kern.warning] WARNING: ufs log for /data1 changed state to Error
Nov 1 23:58:56 stat ufs: [ID 616219 kern.warning] WARNING: Please umount(1M) /data1 and run fsck(1M)

至此我们已经可以看到这是IO通道出现问题，最后导致IO操作失败。

这已经不是数据库层面的问题，我们通过重新启动主机及阵列，进行磁盘检查后，系统恢复正常。
还算幸运！

-The End-

登陆检查主库，警告日志记录了错误信息:

*** 2006-10-30 07:32:10.614
kcrrfail: dest:2 err:12560 force:0
ORA-12560: TNS:protocol adapter error
*** 2006-10-30 07:34:10.615
Error 12541 connecting to destination LOG_ARCHIVE_DEST_2 standby host 'bmarksb'
Error 12541 attaching to destination LOG_ARCHIVE_DEST_2 standby host 'bmarksb'
Heartbeat failed to connect to standby 'bmarksb'. Error is 12541.
*** 2006-10-30 07:34:10.615
kcrrfail: dest:2 err:12541 force:0
ORA-12541: TNS:no listener
*** 2006-10-30 07:36:10.615
Error 12541 connecting to destination LOG_ARCHIVE_DEST_2 standby host 'bmarksb'
Error 12541 attaching to destination LOG_ARCHIVE_DEST_2 standby host 'bmarksb'
Heartbeat failed to connect to standby 'bmarksb'. Error is 12541.

马上登陆从库主机，手工启动备用数据库：

[oracle@wapcom2 bdump]$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.6.0 - Production on Mon Oct 30 08:17:24 2006

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Connected to an idle instance.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 470881780 bytes
Fixed Size 452084 bytes
Variable Size 167772160 bytes
Database Buffers 301989888 bytes
Redo Buffers 667648 bytes
SQL> alter database mount standby database;

Database altered.

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.6.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production
[oracle@wapcom2 bdump]$ lsnrctl start

观察从库的日志信息，发现归档可以自动应用:

[oracle@wapcom2 bdump]$ tail -f alert_bmark.log
Standby Database mounted.
Completed: alter database mount standby database
Mon Oct 30 08:19:23 2006
alter database recover managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process
MRP0 started with pid=12
MRP0: Background Managed Standby Recovery process started
Media Recovery Waiting for thread 1 seq# 5151
Mon Oct 30 08:19:29 2006
Completed: alter database recover managed standby database di
Mon Oct 30 08:22:58 2006
Media Recovery Log /opt/oracle/oradata/bmark/stdarch/1_5151.arc
Media Recovery Log /opt/oracle/oradata/bmark/stdarch/1_5152.arc
Media Recovery Log /opt/oracle/oradata/bmark/stdarch/1_5153.arc
Media Recovery Log /opt/oracle/oradata/bmark/stdarch/1_5154.arc
Media Recovery Log /opt/oracle/oradata/bmark/stdarch/1_5155.arc
Media Recovery Waiting for thread 1 seq# 5156

再检查原因，发现原来是主机出现问题，在夜间不断重起：

-bash-2.05b$ last |grep reboot
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 08:10 (02:14)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 07:51 (02:32)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 07:38 (02:45)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 07:35 (02:48)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 07:21 (03:02)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 07:18 (03:05)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 06:39 (03:44)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 06:37 (03:46)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 06:32 (03:51)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 06:03 (04:21)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 01:48 (08:36)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 01:23 (09:01)
reboot system boot 2.4.21-15.ELsmp Mon Oct 30 00:39 (09:44)

初步看来是硬件出现了故障，最近的硬件故障极为频繁，年底也到了事故多发期。
提醒大家也多多注意。

参考文档：
http://www.eygle.com/ha/dataguard-step-by-step.htm

-The End-

[oracle@gdmstest bdump]$ tail -20 alert_mydb.log
Linux Error: 4: Interrupted system call
Additional information: 23710
Wed Oct 25 16:47:44 2006
Errors in file /opt/oracle/admin/mydb/bdump/mydb_smon_19646.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01115: IO error reading block from file 1 (block # 23712)
ORA-01110: data file 1: '/opt/oracle/oradata/mydb/system01.dbf'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
Linux Error: 4: Interrupted system call
Additional information: 23710
Wed Oct 25 16:47:59 2006
Errors in file /opt/oracle/admin/mydb/bdump/mydb_smon_19646.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01115: IO error reading block from file 1 (block # 23712)
ORA-01110: data file 1: '/opt/oracle/oradata/mydb/system01.dbf'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
Linux Error: 4: Interrupted system call
Additional information: 23710

而通过dbv检查又没有报数据块损坏：

[oracle@gdmstest mydb]$ dbv file=system01.dbf blocksize=8192

DBVERIFY: Release 9.2.0.4.0 - Production on Thu Oct 26 11:36:42 2006

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

DBVERIFY - Verification starting : FILE = system01.dbf

DBVERIFY - Verification complete

Total Pages Examined : 23709
Total Pages Processed (Data) : 13000
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 2090
Total Pages Failing (Index): 0
Total Pages Processed (Other): 1377
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 7242
Total Pages Marked Corrupt : 0
Total Pages Influx : 0

我们一起来看看这个问题，首先从错误日志来看，其实这并不是一个数据块损坏的问题：
ORA-01115: IO error reading block from file 1 (block # 23712)

这是个IO错误，数据块不能读取。

而DBV的提示也只是说检查了23709个数据块，这些数据块没有问题，而我们真正报错的数据块是23712号数据块，也就是说DBV检查到这个块附近，无法继续读取，进而退出。

而系统表空间远远大于 23709 * 8k / 1024 = 185M。

此时检查系统日志，dmesg日志中有大量的寻址错误，也就是说硬件出现了故障：

[maintain@gdmstest bdump]$ dmesg
: error=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=58847319, high=3, low=8515671, sector=14266880
end_request: I/O error, dev 03:06 (hda), sector 14266880
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }

至此问题被定位。

如果我们尝试cp系统表空间文件，同样会收到硬件的错误提示信息:

[oracle@gdmstest mydb]$ cp system01.dbf system01.dbf.bk
cp: 正在读入‘system01.dbf’: 输入/输出错误
[oracle@gdmstest mydb]$ ll
总用量 2173060
....
-rw-r----- 1 oracle dba 524296192 10月 25 16:49 system01.dbf
-rw-r----- 1 oracle dba 194236416 10月 25 17:00 system01.dbf.bk
...............

只能复制194236416 Bytes，也就是 194236416 / 8192 = 23710.5，同样是读到23709个数据块左右，硬件的损坏就要通过系统的其它手段去解决了。

-The End-

sql*plus无法连接,想了一下才记起,还有svrmgrl可以用:

在Oracle8.1.7.0.0中,ORA-04031的问题是由来已久的,使用svrmgrl也不能执行shutdown immediate了.只能通过shutdown abort关闭数据库后重起.

进一步检查发现这个数据库处于初始态运行,共享池设置的只有30M,过小的共享池设置也是导致ORA-04031的原因之一:

shared_pool_size = 31457280
db_block_buffers = 2048

对这两个参数进行了放大调整,主机毕竟有4G内存,调整后,ORA-04031错误应该会少很多了.

数据库关闭后,共享内存并未及时释放:

杀掉残余的Oracle进程后,共享内存释放:

此时数据库可以成功启动.

4CPU,8G内存,的SUN Fire 480R主机:

# ./prtdiag
System Configuration:  Sun Microsystems  sun4u Sun Fire 480R
System clock frequency: 150 MHz
Memory size: 8192 Megabytes

========================= CPUs ===============================================

          Run   E$  CPU     CPU  
Brd  CPU  MHz   MB  Impl.   Mask 
--- ----- ---- ---- ------- ---- 
 A     0  1050  8.0 US-III+  11.0
 A     2  1050  8.0 US-III+  11.0
 B     1  1050  8.0 US-III+  11.0
 B     3  1050  8.0 US-III+  11.0 

 现在负载:

load averages: 13.12, 12.60, 12.23      db480-4.hurray.com.cn    09:50:32
184 processes: 166 sleeping, 12 running, 1 stopped, 5 on cpu
CPU states:  0.0% idle, 93.6% user,  6.4% kernel,  0.0% iowait,  0.0% swap
Memory: 8.0G real, 2.8G free, 4.1G swap in use, 18.1G swap free

   PID USERNAME THR PR NCE  SIZE   RES STATE   TIME FLTS    CPU COMMAND
 11834 oracle     2 22   0  3.1G  3.0G run    12:51    0  7.08% oracle
  3876 oracle     2 22   0  3.1G  3.0G run    38.5H    0  6.93% oracle
 15876 oracle     2 21   0  3.1G  3.0G run   514:20    0  6.85% oracle
  4042 oracle     2 21   0  3.1G  3.0G run    41.3H    0  6.78% oracle
 29532 oracle     2 21   0  3.1G  3.0G run    19.4H    0  6.67% oracle
  3703 oracle     2 21   0  3.1G  3.0G run    38.5H    0  6.66% oracle
 29704 oracle     2 21   0  3.1G  3.0G run    20.9H    0  6.60% oracle
 15537 oracle     2 21   0  3.1G  3.0G run   496:48    0  6.57% oracle
 15680 oracle     2 31   0  3.1G  3.0G run   496:44    0  6.50% oracle
 29375 oracle     2 31   0  3.1G  3.0G run    19.5H    0  6.31% oracle
  4033 oracle     2 22   0  3.1G  3.0G run     8:51    0  4.76% oracle
  4035 oracle     2 51   0  3.1G  3.0G sleep   8:55    0  4.70% oracle
  4046 oracle     2 52   0  3.1G  3.0G cpu03   8:59    0  4.68% oracle
  7349 oracle    11 53   0  3.1G  3.0G sleep   8:38    0  4.44% oracle
  4055 oracle     2 42   0  3.1G  3.0G cpu02   8:43    0  4.39% oracle 

当然模拟2662错误需要技巧，本文并不会涉及这个内容。

通过正常方式启动数据库时，从alert文件中，我们可以看到ora-00600 2662号错误。

Sun Dec 11 18:02:25 2005
Errors in file /opt/oracle/admin/conner/udump/conner_ora_13349.trc:
ORA-00600: internal error code, arguments: [2662], [0], [547743994], [0], [898092653], [8388617], [], []
Sun Dec 11 18:02:27 2005
Errors in file /opt/oracle/admin/conner/udump/conner_ora_13349.trc:
ORA-00600: internal error code, arguments: [2662], [0], [547743994], [0], [898092653], [8388617], [], []
Sun Dec 11 18:02:27 2005
Error 600 happened during db open, shutting down database
USER: terminating instance due to error 600

此时我们可以通过Oracle的内部事件来调整SCN:

增进SCN有两种常用方法:

1.通过immediate trace name方式(在数据库Open状态下)

alter session set events 'IMMEDIATE trace name ADJUST_SCN level x';

2.通过10015事件(在数据库无法打开，mount状态下)

alter session set events '10015 trace name adjust_scn level x';

注:level 1为增进SCN 10亿 (1 billion) (1024*1024*1024),通常Level 1已经足够。也可以根据实际情况适当调整。

本例由于数据库无法打开，只能使用的二种方法。

[oracle@jumper dbs]$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.4.0 - Production on Sun Dec 11 18:26:18 2005

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile=initconner.ora
ORACLE instance started.

Total System Global Area   97588504 bytes
Fixed Size                   451864 bytes
Variable Size              33554432 bytes
Database Buffers           62914560 bytes
Redo Buffers                 667648 bytes
Database mounted.

SQL> alter session set events '10015 trace name adjust_scn level 10';

Session altered.

SQL> alter database open;

Database altered.

注意,由于我使用了10015事件，使得SCN增进了10 billion，稍后我们可以验证。 

[oracle@jumper dbs]$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.4.0 - Production on Sun Dec 11 18:26:18 2005

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile=initconner.ora
ORACLE instance started.

Total System Global Area   97588504 bytes
Fixed Size                   451864 bytes
Variable Size              33554432 bytes
Database Buffers           62914560 bytes
Redo Buffers                 667648 bytes
Database mounted.

SQL> alter session set events '10015 trace name adjust_scn level 10';

Session altered.

SQL> alter database open;

Database altered.

此时数据库可以打开，从alert文件中我们可以看到如下提示:

Sun Dec 11 18:27:04 2005 SMON: enabling cache recovery Sun Dec 11 18:27:05 2005 Debugging event used to advance scn to 10737418240

SCN被增进了10 billion,即 10 * (1024*1024*1024) = 10737418240,正好是日志里记录的数量。

他查询SYSDATE的值一直停留在2005-03-01 11:41:15。感觉很奇怪。

忍不住指导他研究一下，先是从系统级诊断，发现没有问题。

再从数据库角度来诊断，发现:

select current_date from dual; 的输出是正确的，而

select sysdate from dual; 却是不正确的。

猜测是某个参数导致了系统日期被固化，让他传来alert文件，果然发现了一个此前未注意到的参数: FIXED_DATE，

  core_dump_dest           = /u01/app/oracle/admin/unicode/cdump
  fixed_date               = 01-MAR-05
  sort_area_size           = 524288
  db_name                  = unicode

文档上的解释为：

FIXED_DATE

Parameter type	String
Syntax	FIXED_DATE = YYYY-MM-DD-HH24:MI:SS (or the default Oracle date format)
Default value	There is no default value.
Parameter class	Dynamic: ALTER SYSTEM

FIXED_DATE enables you to set a constant date that SYSDATE will always return instead of the current date. This parameter is useful primarily for testing. The value can be in the format shown above or in the default Oracle date format, without a time.

找到了这个参数也就找到了答案！

开始介入处理该事故.
系统环境:
SunOS DB 5.8 Generic_108528-21 sun4u sparc SUNW,Ultra-4
Oracle9i Enterprise Edition Release 9.2.0.3.0 - Production

1.首先介入检查数据库任务

$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.3.0 - Production on Wed Nov 17 20:23:53 2004

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.


Connected to:
Oracle9i Enterprise Edition Release 9.2.0.3.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.3.0 - Production

SQL> select job,last_date,last_sec,next_date,next_sec,broken,failures from dba_jobs;

       JOB LAST_DATE LAST_SEC         NEXT_DATE NEXT_SEC         B   FAILURES INTERVAL
---------- --------- ---------------- --------- ---------------- - ----------   ----------------------------
        31 16-NOV-04 01:00:02         17-NOV-04 01:00:00         N          0 trunc(sysdate+1)+1/24
        27 16-NOV-04 00:00:04         17-NOV-04 00:00:00         N          0 TRUNC(SYSDATE) + 1
        35 16-NOV-04 01:00:02         17-NOV-04 01:00:00         N          0 trunc(sysdate+1)+1/24
        29 16-NOV-04 00:00:04         17-NOV-04 00:00:00         N          0 TRUNC(SYSDATE) + 1
        30 01-NOV-04 06:00:01         01-DEC-04 06:00:00         N          0 trunc(add_months(sysdate,1),'MM')+6/24
        65 16-NOV-04 04:00:03         17-NOV-04 04:00:00         N          0 trunc(sysdate+1)+4/24
        46 16-NOV-04 02:14:27         17-NOV-04 02:14:27         N          0 sysdate+1
        66 16-NOV-04 03:00:02         17-NOV-04 18:14:49         N          0 trunc(sysdate+1)+3/24

8 rows selected.                      

发现JOB任务是都没有正常执行，最早一个应该在17-NOV-04 01:00:00执行。但是没有执行。

2.建立测试JOB

create or replace PROCEDURE pining
  IS
BEGIN
         NULL;
 END;
/

variable jobno number;
variable instno number;
begin
  select instance_number into :instno from v$instance;
  dbms_job.submit(:jobno, 'pining;', trunc(sysdate+1/288,'MI'), 'trunc(SYSDATE+1/288,''MI'')', TRUE, :instno);
end;
/

发现同样的，不执行。
但是通过dbms_job.run(<job>)执行没有任何问题。

3.进行恢复尝试

怀疑是CJQ0进程失效，首先设置JOB_QUEUE_PROCESSES为0，Oracle会杀掉CJQ0及相应job进程
SQL> ALTER SYSTEM SET JOB_QUEUE_PROCESSES = 0;

等2~3分钟，重新设置

SQL> ALTER SYSTEM SET JOB_QUEUE_PROCESSES = 5;

此时PMON会重起CJQ0进程

在警报日志中可以看到以下信息:

Thu Nov 18 11:59:50 2004
ALTER SYSTEM SET job_queue_processes=0 SCOPE=MEMORY;
Thu Nov 18 12:01:30 2004
ALTER SYSTEM SET job_queue_processes=10 SCOPE=MEMORY;
Thu Nov 18 12:01:30 2004
Restarting dead background process CJQ0
CJQ0 started with pid=8      

但是Job仍然不执行，而且在再次修改的时候，CJQ0直接死掉了。

Thu Nov 18 13:52:05 2004
ALTER SYSTEM SET job_queue_processes=0 SCOPE=MEMORY;
Thu Nov 18 14:09:30 2004
ALTER SYSTEM SET job_queue_processes=10 SCOPE=MEMORY;
Thu Nov 18 14:10:27 2004
ALTER SYSTEM SET job_queue_processes=0 SCOPE=MEMORY;
Thu Nov 18 14:10:42 2004
ALTER SYSTEM SET job_queue_processes=10 SCOPE=MEMORY;
Thu Nov 18 14:31:07 2004
ALTER SYSTEM SET job_queue_processes=0 SCOPE=MEMORY;
Thu Nov 18 14:40:14 2004
ALTER SYSTEM SET job_queue_processes=10 SCOPE=MEMORY;
Thu Nov 18 14:40:28 2004
ALTER SYSTEM SET job_queue_processes=0 SCOPE=MEMORY;
Thu Nov 18 14:40:33 2004
ALTER SYSTEM SET job_queue_processes=1 SCOPE=MEMORY;
Thu Nov 18 14:40:40 2004
ALTER SYSTEM SET job_queue_processes=10 SCOPE=MEMORY;
Thu Nov 18 15:00:42 2004
ALTER SYSTEM SET job_queue_processes=0 SCOPE=MEMORY;
Thu Nov 18 15:01:36 2004
ALTER SYSTEM SET job_queue_processes=15 SCOPE=MEMORY;
      

4.尝试重起数据库
这个必须在晚上进行

PMON started with pid=2
DBW0 started with pid=3
LGWR started with pid=4
CKPT started with pid=5
SMON started with pid=6
RECO started with pid=7
CJQ0 started with pid=8
QMN0 started with pid=9
....
      

CJQ0正常启动，但是Job仍然不执行。

5.没办法了...

继续研究...居然发现Oralce有这样一个bug

1. Clear description of the problem encountered:
slgcsf() / slgcs() on Solaris will stop incrementing after
497 days 2 hrs 28 mins (approx) machine uptime.

2. Pertinent configuration information
No special configuration other than long machine uptime. .

3. Indication of the frequency and predictability of the problem
100% but only after 497 days.

4. Sequence of events leading to the problem
If the gethrtime() OS call returns a value > 42949672950000000
nanoseconds then slgcs() stays at 0xffffffff. This can
cause some problems in parts of the code which rely on
slgcs() to keep moving.
eg: In kkjssrh() does "now = slgcs(&se)" and compares that
to a previous timestamp. After 497 days uptime slgcs()
keeps returning 0xffffffff so "now - kkjlsrt" will
always return 0. .

5. Technical impact on the customer. Include persistent after effects.
In this case DBMS JOBS stopped running after 497 days uptime.
Other symptoms could occur in various places in the code.

好么，原来是计时器溢出了，一检查我的主机:

bash-2.03$ uptime
 10:00pm  up 500 day(s), 14:57,  1 user,  load average: 1.31, 1.09, 1.08
bash-2.03$ date
Fri Nov 19 22:00:14 CST 2004      

刚好到事发时是497天多一点.ft.

6.安排重起主机系统..

这个问题够郁闷的，NND，谁曾想Oracle这都成...

Oracle最后声称:

fix made it into 9.2.0.6 patchset

在Solaris上的9206尚未发布...晕.

好了，就当是个经历吧，如果有问题非常不可思议的话，那么大胆怀疑Oracle吧，是Bug，可能就是Bug。

重起以后问题解决，状态如下:

$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.3.0 - Production on Fri Nov 26 09:21:21 2004

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.


Connected to:
Oracle9i Enterprise Edition Release 9.2.0.3.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.3.0 - Production

SQL> select job,last_date,last_sec,next_date,next_sec from user_jobs;

       JOB LAST_DATE LAST_SEC         NEXT_DATE NEXT_SEC
---------- --------- ---------------- --------- ----------------
        70 26-NOV-04 09:21:04         26-NOV-04 09:26:00


SQL> /

       JOB LAST_DATE LAST_SEC         NEXT_DATE NEXT_SEC
---------- --------- ---------------- --------- ----------------
        70 26-NOV-04 09:26:01         26-NOV-04 09:31:00

SQL> 
SQL> select * from v$timer;

     HSECS
----------
   3388153

SQL> select * from v$timer;

     HSECS
----------
   3388319

SQL>       
                      

7.FAQ

一些朋友在Pub上问的问题
Q:对于不同平台，是否存在同样的问题?

A:对于不同平台，存在同样的问题
因为Oracle使用了标准C函数gethrtime
参考:
http://www.eygle.com/unix/Man.Page.Of.gethrtime.htm

使用了该函数的代码都会存在问题.

在Metalink Note:3427424.8 文档中，Oracle定义的平台影响为:Generic (all / most platforms affected)

Q.计数器溢出，看了看job 中基本都是1天左右执行一次，如果设置 3 天执行一次的 job , 是否出问题的uptime 应该是 497*3 之后呢 ？

A:不会

Oracle内部通过计时器来增进相对时间.
由于Oracle内部hrtime_t使用了32位计数

那么最大值也就是0xffffffff
0xffffffff = 4294967295

slgcs()是10亿分之一秒，溢出在42949672950000000这个点上.

注意，这里0xffffffff，达到这个值时，本来是无符号整型，现在变成了-1，那么这个值递增时，+1 = 0了。
时间就此停住了。

 

[oracle@jumper oracle]$ cat unsign.c
#include 
int main(void){
unsigned int num = 0xffffffff;

printf("num is %d bits long\n", sizeof(num) * 8);
printf("num = 0x%x\n", num);
printf("num + 1 = 0x%x\n", num + 1);

return 0;
}

[oracle@jumper oracle]$ gcc -o unsign.sh unsign.c
[oracle@jumper oracle]$ ./unsign.sh
num is 32 bits long
num = 0xffffffff
num + 1 = 0x0
[oracle@jumper oracle]$

Q:内部时钟之一应该就是这个吧： v$timer 精确到1/100 秒的数据

没错!

注意前面说的:

4. Sequence of events leading to the problem
If the gethrtime() OS call returns a value > 42949672950000000
nanoseconds then slgcs() stays at 0xffffffff. This can
cause some problems in parts of the code which rely on
slgcs() to keep moving.

也就是说如果gethrtime() 操作系统调用返回值大于42949672950000000（单位10亿分之一秒）

也就是说Oracle将得到一个cs值为4294967295的时间值

而4294967295值就是0xffffffff

所以当时v$timer的计时也就是:

SQL> select * from v$timer;

     HSECS
----------
4294967295

SQL> /                   

     HSECS
----------
4294967295

SQL> /

     HSECS
----------
4294967295

SQL>      
                      

一、 基础介绍

(a) SQL_TRACE说明

SQL_TRACE可以作为初始化参数在全局启用，也可以通过命令行方式在具体session启用。
1． 在全局启用
在参数文件(pfile/spfile)中指定:

sql_trace =true

在全局启用SQL_TRACE会导致所有进程的活动被跟踪，包括后台进程及所有用户进程，这通常会导致比较严重的性能问题，所以在生产环境
中要谨慎使用.
提示: 通过在全局启用sql_trace，我们可以跟踪到所有后台进程的活动，很多在文档中的抽象说明，通过跟踪文件的实时变化，我们可以清晰
的看到各个进程之间的紧密协调.

2． 在当前session级设置
大多数时候我们使用sql_trace跟踪当前进程.通过跟踪当前进程可以发现当前操作的后台数据库递归活动(这在研究数据库新特性时尤其有效)，
研究SQL执行，发现后台错误等.
在session级启用和停止sql_trace方式如下:

启用当前session的跟踪:
SQL> alter session set sql_trace=true;

Session altered.

此时的SQL操作将被跟踪:
SQL> select count(*) from dba_users;

  COUNT(*)
----------
        34
结束跟踪:
SQL> alter session set sql_trace=false;

Session altered.
       

3． 跟踪其他用户进程
在很多时候我们需要跟踪其他用户的进程，而不是当前用户，这可以通过Oracle提供的系统包DBMS_SYSTEM. SET_SQL_TRACE_IN_SESSION
来完成

SET_SQL_TRACE_IN_SESSION过程序要提供三个参数:

SQL> desc dbms_system
…
PROCEDURE SET_SQL_TRACE_IN_SESSION
 Argument Name                     Type                    In/Out Default?
 ------------------------------           -----------------------   ------ --------
 SID                               NUMBER                  IN
 SERIAL#                          NUMBER                  IN
 SQL_TRACE                        BOOLEAN                 IN
…

通过v$session我们可以获得sid、serial#等信息:

获得进程信息，选择需要跟踪的进程:

SQL> select sid,serial#,username from v$session
  2  where username is not null;

       SID    SERIAL#  USERNAME
---------- ---------- ------------------------------
         8       2041  SYS
         9        437  EYGLE

设置跟着:
SQL> exec dbms_system.set_sql_trace_in_session(9,437,true)

PL/SQL procedure successfully completed.

….
可以等候片刻，跟踪session执行任务,捕获sql操作…
….

停止跟踪:
SQL> exec dbms_system.set_sql_trace_in_session(9,437,false)

PL/SQL procedure successfully completed.
      

(b) 10046事件说明
10046事件是Oracle提供的内部事件，是对SQL_TRACE的增强.
10046事件可以设置以下四个级别:
1 - 启用标准的SQL_TRACE功能,等价于sql_trace
4 - Level 1 加上绑定值(bind values)
8 - Level 1 + 等待事件跟踪
12 - Level 1 + Level 4 + Level 8
类似sql_trace，10046事件可以在全局设置，也可以在session级设置。
1． 在全局设置
在参数文件中增加:

此设置对所有用户的所有进程生效、包括后台进程.

2． 对当前session设置
通过alter session的方式修改，需要alter session的系统权限:

SQL> alter session set events '10046 trace name context forever';

Session altered.

SQL> alter session set events '10046 trace name context forever, level 8';

Session altered.

SQL> alter session set events '10046 trace name context off';

Session altered.

      

3． 对其他用户session设置
通过DBMS_SYSTEM.SET_EV系统包来实现:

SQL> desc dbms_system
...
PROCEDURE SET_EV
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 SI                             BINARY_INTEGER          IN
 SE                             BINARY_INTEGER          IN
 EV                             BINARY_INTEGER          IN
 LE                             BINARY_INTEGER          IN
 NM                             VARCHAR2                IN

...

                      

其中的参数SI、SE来自v$session视图:

查询获得需要跟踪的session信息:
SQL> select sid,serial#,username from v$session where username is not null;

(c) 获取跟踪文件
以上生成的跟踪文件位于user_dump_dest目录中，位置及文件名可以通过以下SQL查询获得:

SQL> select
  2    d.value||'/'||lower(rtrim(i.instance, chr(0)))||'_ora_'||p.spid||'.trc' trace_file_name
  3  from
  4    ( select p.spid
  5      from sys.v$mystat m,sys.v$session s,sys.v$process p
  6      where m.statistic# = 1 and s.sid = m.sid and p.addr = s.paddr) p,
  7    ( select t.instance from sys.v$thread  t,sys.v$parameter  v
  8      where v.name = 'thread' and (v.value = 0 or t.thread# = to_number(v.value))) i,
  9    ( select value from sys.v$parameter where name = 'user_dump_dest') d
 10  /

(d) 读取当前session设置的参数
当我们通过alter session的方式设置了sql_trace,这个设置是不能通过show parameter的方式得到的,我们需要通过dbms_system.read_ev来获取：

SQL> set feedback off
SQL> set serveroutput on