linux ora-00322,ASM丢失disk header导致ORA-15032、ORA-15040、ORA-15042 Diskgroup无法mount

SQL> select * from v$version;

BANNER

——————————————————————————–

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

PL/SQL Release 11.2.0.3.0 – Production

CORE 11.2.0.3.0 Production

TNS for Linux: Version 11.2.0.3.0 – Production

NLSRTL Version 11.2.0.3.0 – Production

SQL> alter diskgroup datadg mount;

alter diskgroup datadg mount

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk “5” is missing from group number “1”

ERROR: alter diskgroup datadg mount

Wed Mar 13 07:42:03 2013

SQL> alter diskgroup datadg mount

NOTE: cache registered group DATADG number=1 incarn=0xccb845cd

NOTE: cache began mount (first) of group DATADG number=1 incarn=0xccb845cd

NOTE: Assigning number (1,2) to disk (/dev/asm-diskg)

NOTE: Assigning number (1,1) to disk (/dev/asm-diskf)

NOTE: Assigning number (1,0) to disk (/dev/asm-diske)

Wed Mar 13 16:42:09 2013

NOTE: GMON heartbeating for grp 1

GMON querying group 1 at 20 for pid 27, osid 5439

NOTE: Assigning number (1,5) to disk ()

GMON querying group 1 at 21 for pid 27, osid 5439

NOTE: cache dismounting (clean) group 1/0xCCB845CD (DATADG)

NOTE: messaging CKPT to quiesce pins Unix process pid: 5439, image: oracle@vmac1 (TNS V1-V3)

NOTE: dbwr not being msg’d to dismount

NOTE: lgwr not being msg’d to dismount

NOTE: cache dismounted group 1/0xCCB845CD (DATADG)

NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xccb845cd

NOTE: cache deleting context for group DATADG 1/0xccb845cd

GMON dismounting group 1 at 22 for pid 27, osid 5439

NOTE: Disk in mode 0x8 marked for de-assignment

ERROR: diskgroup DATADG was not mounted

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk “5” is missing from group number “1”

ERROR: alter diskgroup datadg mount

Wed Mar 13 16:42:10 2013

ASM Health Checker found 1 new failures

[grid@vmac1 ~]$ kfed read /dev/asm-diskh

kfbh.endian: 0 ; 0x000: 0x00

kfbh.hard: 0 ; 0x001: 0x00

kfbh.type: 0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt: 0 ; 0x003: 0x00

kfbh.block.blk: 0 ; 0x004: blk=0

kfbh.block.obj: 0 ; 0x008: file=0

kfbh.check: 0 ; 0x00c: 0x00000000

kfbh.fcn.base: 0 ; 0x010: 0x00000000

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

7FA1DA233400 00000000 00000000 00000000 00000000 [................]

Repeat 255 times

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

col path for a20

set linesize 200 pagesize 1400

select path,header_status,state from v$asm_disk;

PATH HEADER_STATUS STATE

——————– ———————————— ————————

/dev/asm-diskh CANDIDATE NORMAL

/dev/asm-diskg MEMBER NORMAL

/dev/asm-diskf MEMBER NORMAL

/dev/asm-diske MEMBER NORMAL

/dev/asm-diskc MEMBER NORMAL

/dev/asm-diskd MEMBER NORMAL

/dev/asm-diskb MEMBER NORMAL

7 rows selected.

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh

KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576

KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]

关闭ASM实例

优先备份问题ASM的header

dd if= of= bs=4096 count=1

3. 检查现有磁盘找出拥有file 1 block 1的

[grid@vmac1 ~]$ kfed read /dev/asm-diske |grep f1b1

kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002

[grid@vmac1 ~]$

[grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1

kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

[grid@vmac1 ~]$ kfed read /dev/asm-diskg |grep f1b1

kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

这里asm-diske上出现了f1b1非零值。则其拥有file 1 block 1，能够通过检查第二个au的类型是否是KFBTYP_LISTHEAD来确认

[grid@vmac1 ~]$ kfed read /dev/asm-diske aun=2|grep kfbh.type

kfbh.type: 5 ; 0x002: KFBTYP_LISTHEAD

若丢失的磁盘包括了”file 1 block 1 F1B1″则扫描该磁盘上全部的 AU直到找到KFBTYP_LISTHEAD ，假设找不到LISTHEAD 那么别无选择仅仅能重建diskgroup。

从版本号11.1.0.7開始(10g是从10.2.0.5開始，所以尽量别用10.2.0.5之前的版本号上的ASM)。当每个I/O写提交向ASM disk header(AU 0 blocknum 0)，都会拷贝到 AU 1中，最后第二个块。基于不同的AU size，该块的位置不同

Allocation Unit Size Block Number on AU 1

1048576 254

4194304 1022

8388608 2046

16777216 4094

首先使用kfed 验证该位置是否有正确的disk header，否则手动找到合适的header。表1如參考：

[grid@vmac1 ~]$ kfed read /dev/asm-diske ausz=1048576 aun=1 blkn=254|less

kfbh.endian: 1 ; 0x000: 0x01

kfbh.hard: 130 ; 0x001: 0x82

kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD

kfbh.datfmt: 1 ; 0x003: 0x01

kfbh.block.blk: 254 ; 0x004: blk=254

kfbh.block.obj: 2147483648 ; 0x008: disk=0

kfbh.check: 2086475720 ; 0x00c: 0x7c5d17c8

kfbh.fcn.base: 31322 ; 0x010: 0x00007a5a

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

之后使用kfed repair命令修复disk header

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576

假设disk header的自己主动备份也丢失了。那么会报例如以下错误

KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]

若上述kfed repair无效则须要通过手动恢复disk header的办法了：

对于版本号10.2.0.5之前没有kred repair可用的情况：

找一个与问题disk 在同一个diskgroup中的，不包括f1b1 的好的disk header。比如这里的asm-diskf

[grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1

kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

使用kfed read > fix.txt 命令保存其头部

[grid@vmac1 ~]$ kfed read /dev/asm-diskf > fix.txt

编辑fix.txt 改动kfdhdb.dsknum 、kfdhdb.dskname 、 kfdhdb.fgname 三个信息：

同一时候參考alert.log中的信息：

[grid@vmac1 trace]$ grep “cache opening” alert_+ASM1.log

NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb

NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc

NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd