帮助用户恢复数据块损坏的海量数据库

« Oracle统计信息的收集、管理与清除 | Blog首页 | COL_USAGE$表的信息记录与存储 »

上周在上海出差时，有用户找到我们，请求帮住恢复一个因为硬盘损坏导致的数据库故障，数据库是Oracle 10g的RAC集群，存储使用的是ASM管理。

主要问题是SYSTEM表空间存在大量的坏块，DBV检查，很多关键数据块出现问题，收录一下DBV的检测输出，看一看RAID硬盘损坏类故障会导致多少故障吧：

DBVERIFY: Release 10.2.0.4.0 - Production on Sat Oct 23 13:12:28 2010

Copyright (c) 1982, 2007, Oracle. All rights reserved.

DBVERIFY - Verification starting : FILE = SYSTEM
Page 83 is influx - most likely media corrupt
Corrupt block relative dba: 0x00400053 (file 1, block 83)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x00400053
last change scn: 0x0000.00001612 seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x06920608
check value in block header: 0x1755
computed block checksum: 0x7b6d

Block Checking: DBA = 4194389, Block Type = KTB-managed data block
**** kdxconro = 4416 exceeds max number of rows = 2337
---- end index block validation
Page 85 failed with check code 6401
Block Checking: DBA = 4194390, Block Type = KTB-managed data block
**** kdxcofbo = 17611 != 1620
---- end index block validation
Page 86 failed with check code 6401
Page 87 is influx - most likely media corrupt
Corrupt block relative dba: 0x00400057 (file 1, block 87)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x00400057
last change scn: 0x0000.0529f57c seq: 0x7 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xe4a30601
check value in block header: 0x84ae
computed block checksum: 0x76e9

Page 128 is marked corrupt
Corrupt block relative dba: 0x00400080 (file 1, block 128)
Bad header found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00400090
last change scn: 0x0000.000a3050 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x30500001
check value in block header: 0x81a1
computed block checksum: 0x0

Page 129 is marked corrupt
Corrupt block relative dba: 0x00400081 (file 1, block 129)
Bad header found during dbv:
Data in bad block:
type: 16 format: 2 rdba: 0x00400091
last change scn: 0x0000.00000042 seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x00421001
check value in block header: 0xe6fe
computed block checksum: 0x0

Page 130 is marked corrupt
Corrupt block relative dba: 0x00400082 (file 1, block 130)
Bad header found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00400092
last change scn: 0x0000.00000000 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x00000001
check value in block header: 0xa7d2
computed block checksum: 0x0

Page 787 is influx - most likely media corrupt
Corrupt block relative dba: 0x00400313 (file 1, block 787)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x00400313
last change scn: 0x0000.d9ac8879 seq: 0x1 flg: 0x06
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x0e300601
check value in block header: 0x5e47
computed block checksum: 0x9a4b

Block Checking: DBA = 4195978, Block Type = KTB-managed data block
**** row 0: row offset 1878 out of valid range
**** row 1: committed with rsl and/or ras flag
**** actual rows locked by itl 2 = 0 != # in trans. header = 2
**** actual rows marked deleted = 0 != kdxlende = 1
---- end index block validation
Page 1674 failed with check code 6401
Block Checking: DBA = 4196250, Block Type = KTB-managed data block
data header at 0x60025c
kdbchk: bad row offset slot 20 offs 16675 fseo 2034 dtl 8168 bhs 72
Page 1946 failed with check code 6135
Page 3802 is influx - most likely media corrupt
Corrupt block relative dba: 0x00400eda (file 1, block 3802)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x00400eda
last change scn: 0x0ade.e37c82e2 seq: 0x1 flg: 0x06
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x3ba50601
check value in block header: 0x1d27
computed block checksum: 0xe87d

Page 3953 is influx - most likely media corrupt
Corrupt block relative dba: 0x00400f71 (file 1, block 3953)
Fractured block found during dbv:
Data in bad block:
type: 16 format: 2 rdba: 0x00400f71
last change scn: 0x0000.04e2970e seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x484f1001
check value in block header: 0xa3dd
computed block checksum: 0x6817

Block Checking: DBA = 4198258, Block Type = KTB-managed data block
data header at 0x6b025c
kdbchk: row locked by non-existent transaction
        table=0   slot=0
        lockid=50   ktbbhitc=2
Page 3954 failed with check code 6101
Page 3955 is influx - most likely media corrupt
Corrupt block relative dba: 0x00400f73 (file 1, block 3955)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x00400f73
last change scn: 0x0000.050b6741 seq: 0x1 flg: 0x06
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xa25e0601
check value in block header: 0x85cd
computed block checksum: 0xc86f

Block Checking: DBA = 4198260, Block Type = KTB-managed data block
data header at 0x6b425c
kdbchk: bad row offset slot 2 offs 417 fseo 1040 dtl 8168 bhs 72
Page 3956 failed with check code 6135
Block Checking: DBA = 4198262, Block Type = KTB-managed data block
data header at 0x6b825c
kdbchk: bad row offset slot 71 offs 977 fseo 1008 dtl 8168 bhs 72
Page 3958 failed with check code 6135
Block Checking: DBA = 4198263, Block Type = KTB-managed data block
data header at 0x6ba25c
kdbchk: bad row offset slot 68 offs 1052 fseo 1177 dtl 8168 bhs 72
Page 3959 failed with check code 6135
Block Checking: DBA = 4198264, Block Type = KTB-managed data block
data header at 0x6bc25c
kdbchk: row locked by non-existent transaction
        table=0   slot=0
        lockid=1   ktbbhitc=2
Page 3960 failed with check code 6101
Page 3961 is influx - most likely media corrupt
Corrupt block relative dba: 0x00400f79 (file 1, block 3961)
Fractured block found during dbv:
Data in bad block:
type: 16 format: 2 rdba: 0x00400f79
last change scn: 0x0000.051514df seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x97001001
check value in block header: 0xb3c2
computed block checksum: 0xdaf6

Page 4169 is influx - most likely media corrupt
Corrupt block relative dba: 0x00401049 (file 1, block 4169)
Fractured block found during dbv:
Data in bad block:
type: 16 format: 2 rdba: 0x00401049
last change scn: 0x0000.0000191a seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x69dd1001
check value in block header: 0xf436
computed block checksum: 0x2fe7

Block Checking: DBA = 4198474, Block Type = KTB-managed data block
**** actual rows locked by itl 1 = 0 != # in trans. header = 5
**** actual rows locked by itl 2 = 1 != # in trans. header = 0
---- end index block validation
Page 4170 failed with check code 6401
Page 4171 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040104b (file 1, block 4171)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x0040104b
last change scn: 0x0000.01d8855b seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xd6fc0001
check value in block header: 0x1d86
computed block checksum: 0x6d96

Page 4173 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040104d (file 1, block 4173)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x0040104d
last change scn: 0x0000.003f5a45 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x12910001
check value in block header: 0xd869
computed block checksum: 0xd40b

Page 4175 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040104f (file 1, block 4175)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x0040104f
last change scn: 0x0000.01f2cd94 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x00000001
check value in block header: 0x3814
computed block checksum: 0xdb95

Page 4181 is influx - most likely media corrupt
Corrupt block relative dba: 0x00401055 (file 1, block 4181)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00401055
last change scn: 0x0000.01f2e7e3 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xb3280001
check value in block header: 0xdc22
computed block checksum: 0x54cb

Page 4185 is influx - most likely media corrupt
Corrupt block relative dba: 0x00401059 (file 1, block 4185)
Fractured block found during dbv:
Data in bad block:
type: 16 format: 2 rdba: 0x00401059
last change scn: 0x0000.00092c6c seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x19441001
check value in block header: 0x93aa
computed block checksum: 0xa0d

Page 4901 is influx - most likely media corrupt
Corrupt block relative dba: 0x00401325 (file 1, block 4901)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00401325
last change scn: 0x0000.00026d1b seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x4ccb0001
check value in block header: 0x3460
computed block checksum: 0x21d0

Page 4903 is influx - most likely media corrupt
Corrupt block relative dba: 0x00401327 (file 1, block 4903)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00401327
last change scn: 0x0000.00190d98 seq: 0x1 flg: 0x07
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xd9ed0001
check value in block header: 0x2d5f
computed block checksum: 0xb9d8

Page 4979 is influx - most likely media corrupt
Corrupt block relative dba: 0x00401373 (file 1, block 4979)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00401373
last change scn: 0x0000.00000000 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x2eb20025
check value in block header: 0xb433
computed block checksum: 0x2e96

Block Checking: DBA = 4203242, Block Type = KTB-managed data block
data header at 0x6a025c
kdbchk: row does not end within block
        table=0 slot=0
        len=65 offset=8051 dtl=8096
Page 8938 failed with check code 6103
Block Checking: DBA = 4205770, Block Type = KTB-managed data block
**** kdxcofbo = 36 != 266
---- end index block validation
Page 11466 failed with check code 6401
Block Checking: DBA = 4221095, Block Type = KTB-managed data block
data header at 0x61a25c
kdbchk: avsp(7814) > tosp(7448)
Page 26791 failed with check code 6128
Page 56945 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040de71 (file 1, block 56945)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x0040de71
last change scn: 0x0000.00079c2f seq: 0x1 flg: 0x06
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xb6090601
check value in block header: 0x5259
computed block checksum: 0x2f40

Block Checking: DBA = 4251252, Block Type = KTB-managed data block
**** row 215: key out of order
**** row 215: lock value 6 is larger than maximum itl 2
**** row 215: bad flag value 195
**** row 216: row length 65537 past end of block
**** row 216: row skipped so other stats may be wrong
**** row 217: row length 65737 past end of block
**** row 217: row skipped so other stats may be wrong
**** row 218: row length 65537 past end of block
**** row 218: row skipped so other stats may be wrong
**** row 219: row length 567 past end of block
**** row 219: row skipped so other stats may be wrong
**** row 220: key out of order
**** row 220: lock value 4 is larger than maximum itl 2
**** row 220: bad flag value 193
**** row 221: bad flag value 193
**** row 222: key out of order
**** row 222: lock value 4 is larger than maximum itl 2
**** row 222: bad flag value 193
**** row 223: lock value 193 is larger than maximum itl 2
**** row 224: key out of order
**** row 224: lock value 118 is larger than maximum itl 2
**** row 225: key out of order
**** row 225: lock value 26 is larger than maximum itl 2
**** row 226: row length 1386 past end of block
**** row 226: row skipped so other stats may be wrong
**** row 227: rsl is 0 with ras flag
**** row 228: row length 1220 past end of block
**** row 228: row skipped so other stats may be wrong
**** row 229: lock value 108 is larger than maximum itl 2
**** row 230: row offset 1070 out of valid range
**** row 231: row offset 1088 out of valid range
**** row 232: row length 65538 past end of block
**** row 232: row skipped so other stats may be wrong
**** actual rows locked by itl 2 = 37 != # in trans. header = 39
**** actual free space credit for itl 2 = 13 != # in trans. hdr = 0
**** actual rows marked deleted = 1 != kdxlende = 0
**** key (begin=0x1bfd, len=17) overlaps with another
        begin = 0x1c0c len = 9
---- end index block validation
Page 56948 failed with check code 6401
Page 59913 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040ea09 (file 1, block 59913)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x0040ea09
last change scn: 0x0000.00079c40 seq: 0x1 flg: 0x06
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xa9b70002
check value in block header: 0x9088
computed block checksum: 0x5c69

Page 59921 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040ea11 (file 1, block 59921)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x0040ea11
last change scn: 0x0a20.65b3f778 seq: 0x3 flg: 0x07
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x69360601
check value in block header: 0xdc6d
computed block checksum: 0x92de

Page 59925 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040ea15 (file 1, block 59925)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x0040ea15
last change scn: 0x0add.97765338 seq: 0x1 flg: 0x06
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x66730002
check value in block header: 0xd22
computed block checksum: 0x3d9d

Page 59929 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040ea19 (file 1, block 59929)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x0040ea19
last change scn: 0x0ade.b7aa3820 seq: 0x2 flg: 0x07
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x0dd70601
check value in block header: 0x8e2a
computed block checksum: 0x5c69

Page 61034 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040ee6a (file 1, block 61034)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x0040ee6a
last change scn: 0x0000.00001cbc seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x19f10001
check value in block header: 0xf453
computed block checksum: 0x3010

Page 61036 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040ee6c (file 1, block 61036)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x0040ee6c
last change scn: 0x0000.46b298bc seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x00000001
check value in block header: 0x89f5
computed block checksum: 0x5558

Page 63921 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040f9b1 (file 1, block 63921)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x0040f9b1
last change scn: 0x0ada.cb9269b0 seq: 0x1 flg: 0x06
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x13ed0601
check value in block header: 0xcaa
computed block checksum: 0xfba6

Block Checking: DBA = 4258226, Block Type = KTB-managed data block
**** row 195: row length 65546 past end of block
**** row 195: row skipped so other stats may be wrong
**** row 196: row length 65545 past end of block
**** row 196: row skipped so other stats may be wrong
**** row 230: row offset 2205 out of valid range
**** row 250: key out of order
**** row 251: key out of order
**** key (begin=0x1f4b, len=21) overlaps with another
        begin = 0x1f4b len = 21
---- end index block validation
Page 63922 failed with check code 6401
Page 63923 is influx - most likely media corrupt
Corrupt block relative dba: 0x0040f9b3 (file 1, block 63923)
Fractured block found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x0040f9b3
last change scn: 0x0a0e.70b10d63 seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x41800601
check value in block header: 0x8da0
computed block checksum: 0x4ce3

Block Checking: DBA = 4258228, Block Type = KTB-managed data block
**** kdxcofbo = 602 != 422
---- end index block validation
Page 63924 failed with check code 6401
Block Checking: DBA = 4258229, Block Type = KTB-managed data block
**** kdxcofbo = 538 != 646
---- end index block validation
Page 63925 failed with check code 6401
Page 66868 is influx - most likely media corrupt
Corrupt block relative dba: 0x00410534 (file 1, block 66868)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00410534
last change scn: 0x0000.00000000 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xf4560001
check value in block header: 0xa275
computed block checksum: 0xdf49

Page 66870 is influx - most likely media corrupt
Corrupt block relative dba: 0x00410536 (file 1, block 66870)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00410536
last change scn: 0x0000.018a6974 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xdd130001
check value in block header: 0x4a1
computed block checksum: 0xb467

Page 66872 is influx - most likely media corrupt
Corrupt block relative dba: 0x00410538 (file 1, block 66872)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x00410538
last change scn: 0x0000.01b50edf seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x9d100001
check value in block header: 0xeeac
computed block checksum: 0x93cf

Page 66876 is influx - most likely media corrupt
Corrupt block relative dba: 0x0041053c (file 1, block 66876)
Fractured block found during dbv:
Data in bad block:
type: 0 format: 2 rdba: 0x0041053c
last change scn: 0x0000.01b544d1 seq: 0x1 flg: 0x05
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xdd340001
check value in block header: 0x7a0b
computed block checksum: 0xbfeb

DBVERIFY - Verification complete

Total Pages Examined         : 131072
Total Pages Processed (Data) : 43322
Total Pages Failing   (Data) : 8
Total Pages Processed (Index): 10883
Total Pages Failing   (Index): 9
Total Pages Processed (Other): 1843
Total Pages Processed (Seg) : 0
Total Pages Failing   (Seg) : 0
Total Pages Empty            : 74992
Total Pages Marked Corrupt   : 32
Total Pages Influx           : 29
Highest block SCN            : 3827128212 (2782.3827128212)