昨天中午接到一位朋友紧急求救电话,大概场景如下,asm data磁盘组一共把个asm disk,但是使用4个lun实现的(也就是说每个lun使用fdisk进行分区),该主机上还有一个lun是用来存放备份的挂载在/xifenfei目录.客户记得他们的/xifenfei目录是/dev/sdh1这个分区,上次主机重启之后,rac工作正常,但是/xifenfei这个目录对应的/dev/sdh1无法挂载上去(提示无法找到超级块).然后我这位朋友上去也不管三七二一直接执行了mkfs.ext4 /dev/sdh1.结果整个asm 磁盘组异常了.最后他通过fdisk命令一看发现完蛋了,以前的/dev/sdh 已经变为了/dev/sdc,而现在的/dev/sdh是以前的asm disk.也就是说,他把asm 磁盘组中的一个disk进行了格式化为ext4文件系统操作.

ext4文件系统大概结构

通过这里大概可以发现在格式化为ext4文件系统并非把所有的磁盘数据全部重写主要就是覆盖一些ext4必要的一些元数据信息,理论上没有覆盖部分的数据依旧可以恢复

fdisk 分析现在磁盘情况

[root@db3 ~]# fdisk -l

Disk /dev/sda: 171.8 GB, 171798691840 bytes

255 heads, 63 sectors/track, 20886 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00055b12

Device Boot Start End Blocks Id System

/dev/sda1 * 1 64 512000 83 Linux

Partition 1 does not end on cylinder boundary.

/dev/sda2 64 8223 65536000 82 Linux swap / Solaris

/dev/sda3 8223 20887 101723136 8e Linux LVM

Disk /dev/sdb: 751.6 GB, 751619276800 bytes

255 heads, 63 sectors/track, 91379 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00b6888f

Device Boot Start End Blocks Id System

/dev/sdb1 1 91379 734001786 5 Extended

/dev/sdb5 1 15000 120487437 83 Linux

Disk /dev/sdd: 1073 MB, 1073741824 bytes

34 heads, 61 sectors/track, 1011 cylinders

Units = cylinders of 2074 * 512 = 1061888 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00083b68

Device Boot Start End Blocks Id System

Disk /dev/sdf: 526.1 GB, 526133493760 bytes

255 heads, 63 sectors/track, 63965 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000c10d8

Device Boot Start End Blocks Id System

/dev/sdf1 1 13054 104856223+ 83 Linux

/dev/sdf2 13055 63965 408942607+ 83 Linux

Disk /dev/sde: 1073 MB, 1073741824 bytes

34 heads, 61 sectors/track, 1011 cylinders

Units = cylinders of 2074 * 512 = 1061888 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000b120f

Device Boot Start End Blocks Id System

Disk /dev/sdc: 1795.3 GB, 1795296329728 bytes

255 heads, 63 sectors/track, 218265 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x78c377f7

Device Boot Start End Blocks Id System

/dev/sdc1 1 218265 1753213581 83 Linux

Disk /dev/sdg: 526.1 GB, 526133493760 bytes

255 heads, 63 sectors/track, 63965 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000ee648

Device Boot Start End Blocks Id System

/dev/sdg1 1 13054 104856223+ 83 Linux

/dev/sdg2 13055 63965 408942607+ 83 Linux

Disk /dev/sdi: 526.1 GB, 526133493760 bytes

255 heads, 63 sectors/track, 63965 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00027e4b

Device Boot Start End Blocks Id System

/dev/sdi1 1 13054 104856223+ 83 Linux

/dev/sdi2 13055 63965 408942607+ 83 Linux

Disk /dev/sdh: 526.1 GB, 526133493760 bytes

255 heads, 63 sectors/track, 63965 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000938dc

Device Boot Start End Blocks Id System

/dev/sdh1 1 13054 104856223+ 83 Linux

/dev/sdh2 13055 63965 408942607+ 83 Linux

Disk /dev/mapper/rootvg-rootfs: 31.7 GB, 31708938240 bytes

255 heads, 63 sectors/track, 3855 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

Disk /dev/mapper/rootvg-lv01: 32.2 GB, 32212254720 bytes

255 heads, 63 sectors/track, 3916 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

通过asm alert日志确定asm disk信息

这里可以确定asm disk是VOL01-VOL08一共8个盘

SQL> CREATE DISKGROUP DATA EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/VOL01',

'/dev/oracleasm/disks/VOL02',

'/dev/oracleasm/disks/VOL03',

'/dev/oracleasm/disks/VOL04' ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M' /* ASMCA */

SQL> ALTER DISKGROUP DATA ADD DISK '/dev/oracleasm/disks/VOL05' SIZE 399358M ,

'/dev/oracleasm/disks/VOL06' SIZE 399358M ,

'/dev/oracleasm/disks/VOL07' SIZE 399358M ,

'/dev/oracleasm/disks/VOL08' SIZE 399358M /* ASMCA */

SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:2} */

NOTE: Diskgroup used for Voting files is:

DATA

Diskgroup with spfile:DATA

Diskgroup used for OCR is:DATA

NOTE: cache registered group DATA number=1 incarn=0x6f3f6eb8

NOTE: cache began mount (not first) of group DATA number=1 incarn=0x6f3f6eb8

NOTE: Assigning number (1,7) to disk (/dev/oracleasm/disks/VOL08)

NOTE: Assigning number (1,6) to disk (/dev/oracleasm/disks/VOL07)

NOTE: Assigning number (1,5) to disk (/dev/oracleasm/disks/VOL06)

NOTE: Assigning number (1,4) to disk (/dev/oracleasm/disks/VOL05)

NOTE: Assigning number (1,3) to disk (/dev/oracleasm/disks/VOL04)

NOTE: Assigning number (1,1) to disk (/dev/oracleasm/disks/VOL02)

NOTE: Assigning number (1,2) to disk (/dev/oracleasm/disks/VOL03)

NOTE: Assigning number (1,0) to disk (/dev/oracleasm/disks/VOL01)

GMON querying group 1 at 2 for pid 23, osid 32026

NOTE: cache opening disk 0 of grp 1: DATA_0000 path:/dev/oracleasm/disks/VOL01

NOTE: cache opening disk 1 of grp 1: DATA_0001 path:/dev/oracleasm/disks/VOL02

NOTE: cache opening disk 2 of grp 1: DATA_0002 path:/dev/oracleasm/disks/VOL03

NOTE: cache opening disk 3 of grp 1: DATA_0003 path:/dev/oracleasm/disks/VOL04

NOTE: cache opening disk 4 of grp 1: DATA_0004 path:/dev/oracleasm/disks/VOL05

NOTE: cache opening disk 5 of grp 1: DATA_0005 path:/dev/oracleasm/disks/VOL06

NOTE: cache opening disk 6 of grp 1: DATA_0006 path:/dev/oracleasm/disks/VOL07

NOTE: cache opening disk 7 of grp 1: DATA_0007 path:/dev/oracleasm/disks/VOL08

NOTE: cache mounting (not first) external redundancy group 1/0x6F3F6EB8 (DATA)

查询asm对应的disk信息

这部分信息,可以确定VOL03被格式化,而且应该就是朋友反馈的被他格式化成ext4的/dev/sdh1

[root@db3 disks]# ls

VOL01 VOL02 VOL04 VOL05 VOL06 VOL07 VOL08

[root@db3 disks]# oracleasm querydisk -p 'VOL01'

Disk "VOL01" is a valid ASM disk

/dev/sdf1: LABEL="VOL01" TYPE="oracleasm"

[root@db3 disks]# oracleasm querydisk -p 'VOL02'

Disk "VOL02" is a valid ASM disk

/dev/sdg1: LABEL="VOL02" TYPE="oracleasm"

[root@db3 disks]# oracleasm querydisk -p 'VOL03'

Disk "VOL03" does not exist or is not instantiated

[root@db3 disks]# oracleasm querydisk -p 'VOL04'

Disk "VOL04" is a valid ASM disk

/dev/sdi1: LABEL="VOL04" TYPE="oracleasm"

[root@db3 disks]# oracleasm querydisk -p 'VOL05'

Disk "VOL05" is a valid ASM disk

/dev/sdf2: LABEL="VOL05" TYPE="oracleasm"

[root@db3 disks]# oracleasm querydisk -p 'VOL06'

Disk "VOL06" is a valid ASM disk

/dev/sdg2: LABEL="VOL06" TYPE="oracleasm"

[root@db3 disks]# oracleasm querydisk -p 'VOL07'

Disk "VOL07" is a valid ASM disk

/dev/sdh2: LABEL="VOL07" TYPE="oracleasm"

[root@db3 disks]# oracleasm querydisk -p 'VOL08'

Disk "VOL08" is a valid ASM disk

/dev/sdi2: LABEL="VOL08" TYPE="oracleasm"

通过kfed分析被格式化成ext4的磁盘信息

[grid@db3 ~]$ kfed read /dev/sdh1

kfbh.endian: 0 ; 0x000: 0x00

kfbh.hard: 0 ; 0x001: 0x00

kfbh.type: 0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt: 0 ; 0x003: 0x00

kfbh.block.blk: 0 ; 0x004: blk=0

kfbh.block.obj: 0 ; 0x008: file=0

kfbh.check: 810307429 ; 0x00c: 0x304c4f65

kfbh.fcn.base: 0 ; 0x010: 0x00000000

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

000000000 00000000 00000000 00000000 304C4F65 [............eOL0]

000000010 00000000 00000000 00000000 00000000 [................]

000000020 4C43524F 44524C43 00000000 00000000 [ORCLCLRD........]

000000030 00000000 00000000 00000000 00000000 [................]

Repeat 252 times

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[grid@db3 ~]$ kfed read /dev/sdh1 aun=1

kfbh.endian: 0 ; 0x000: 0x00

kfbh.hard: 129 ; 0x001: 0x81

kfbh.type: 0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt: 0 ; 0x003: 0x00

kfbh.block.blk: 98560 ; 0x004: blk=98560

kfbh.block.obj: 164096 ; 0x008: file=164096

kfbh.check: 229632 ; 0x00c: 0x00038100

kfbh.fcn.base: 295168 ; 0x010: 0x00048100

kfbh.fcn.wrap: 819456 ; 0x014: 0x000c8100

kfbh.spare1: 884992 ; 0x018: 0x000d8100

kfbh.spare2: 1605888 ; 0x01c: 0x00188100

000100000 00008100 00018100 00028100 00038100 [................]

000100010 00048100 000C8100 000D8100 00188100 [................]

000100020 00288100 003E8100 00798100 00AB8100 [..(...>...y.....]

000100030 01388100 016C8100 00000000 00000000 [..8...l.........]

000100040 00000000 00000000 00000000 00000000 [................]

Repeat 251 times

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[grid@db3 ~]$ kfed read /dev/sdh1 aun=1 blkn=254

kfbh.endian: 254 ; 0x000: 0xfe

kfbh.hard: 129 ; 0x001: 0x81

kfbh.type: 0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt: 0 ; 0x003: 0x00

kfbh.block.blk: 98814 ; 0x004: blk=98814

kfbh.block.obj: 164350 ; 0x008: file=164350

kfbh.check: 229886 ; 0x00c: 0x000381fe

kfbh.fcn.base: 295422 ; 0x010: 0x000481fe

kfbh.fcn.wrap: 819710 ; 0x014: 0x000c81fe

kfbh.spare1: 885246 ; 0x018: 0x000d81fe

kfbh.spare2: 1606142 ; 0x01c: 0x001881fe

0001FE000 000081FE 000181FE 000281FE 000381FE [................]

0001FE010 000481FE 000C81FE 000D81FE 001881FE [................]

0001FE020 002881FE 003E81FE 007981FE 00AB81FE [..(...>...y.....]

0001FE030 013881FE 016C81FE 00000000 00000000 [..8...l.........]

0001FE040 00000000 00000000 00000000 00000000 [................]

Repeat 251 times

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[grid@db3 ~]$ kfed read /dev/sdh1 aun=9

kfbh.endian: 0 ; 0x000: 0x00

kfbh.hard: 0 ; 0x001: 0x00

kfbh.type: 0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt: 0 ; 0x003: 0x00

kfbh.block.blk: 0 ; 0x004: blk=0

kfbh.block.obj: 0 ; 0x008: file=0

kfbh.check: 0 ; 0x00c: 0x00000000

kfbh.fcn.base: 0 ; 0x010: 0x00000000

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

000900000 00000000 00000000 00000000 00000000 [................]

Repeat 255 times

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

从这里可以看出来该asm的元数据信息大量被破坏(asm 备份的磁盘头也损坏),直接修复的可能性基本上为0,现在为了最大限度的恢复数据,通过底层扫描asm au结合数据库block信息,进行重组出来数据文件(这次的恢复有点难度,因为该asm磁盘组中有两个数据库,需要通过结合asm file+datafile来区分文件)

恢复出来两套库的数据文件信息

/orabak/xifenfei/oradb/1.dbf

/orabak/xifenfei/oradb/2.dbf

/orabak/xifenfei/oradb/3.dbf

/orabak/xifenfei/oradb/4.dbf

/orabak/xifenfei/oradb/5.dbf

/orabak/xifenfei/oradb/6.dbf

/orabak/xifenfei/oradb/7.dbf

/orabak/xifenfei/oradb/8.dbf

/orabak/xifenfei/oradb/9.dbf

/orabak/xifenfei/oradb/10.dbf

/orabak/xifenfei/oradb/11.dbf

/orabak/xifenfei/oradb/12.dbf

/orabak/xifenfei/oradb/13.dbf

/orabak/xifenfei/oradb/14.dbf

/orabak/xifenfei/oradb/15.dbf

/orabak/xifenfei/oradb/16.dbf

/orabak/xifenfei/oradb/17.dbf

/orabak/xifenfei/oradb/18.dbf

/orabak/xifenfei/oradb/19.dbf

/orabak/xifenfei/xgdb/1.dbf

/orabak/xifenfei/xgdb/2.dbf

/orabak/xifenfei/xgdb/3.dbf

/orabak/xifenfei/xgdb/4.dbf

/orabak/xifenfei/xgdb/5.dbf

/orabak/xifenfei/xgdb/6.dbf

/orabak/xifenfei/xgdb/7.dbf

/orabak/xifenfei/xgdb/8.dbf

/orabak/xifenfei/xgdb/9.dbf

/orabak/xifenfei/xgdb/10.dbf

/orabak/xifenfei/xgdb/11.dbf

/orabak/xifenfei/xgdb/12.dbf

/orabak/xifenfei/xgdb/13.dbf

/orabak/xifenfei/xgdb/14.dbf

/orabak/xifenfei/xgdb/15.dbf

/orabak/xifenfei/xgdb/16.dbf

然后使用工具拷贝出来redo信息,最后通过重建控制文件.其中一套库完美直接open,另外一套库system中的c_obj$被覆盖,不过使用一个多月以前的备份的system文件强制打开库成功,数据基本上完美导出,实现完美恢复.由于在格式化为ext4的时候,会在磁盘中部分位置写入一些

数据文件恢复参考:asm disk header 彻底损坏恢复

另外有一次win平台类似恢复经历:asm disk格式化为ntfs恢复

如果您遇到此类情况,无法解决请联系我们，提供专业ORACLE数据库恢复技术支持

Phone:13429648788    Q Q:107644445    E-Mail:dba@xifenfei.com