一、 问题描述

用户VA7110 有一硬盘故障，VA自动rebuild失败，11月25更换硬盘后rebuild仍然失败，之后VA做balance一周都未完成，I/O比较慢，数据库checkpoint时间最高达到200多秒，业务严重受到影响

二、 告警信息

#armdsp -a va

Vendor ID:______________________________HP

Product ID:_____________________________A6189B

Array World Wide Name:__________________50060b00001535a8

Array Serial Number:____________________00SG324J0103

Alias:__________________________________va

Software Revision:______________________1.09.02 - 0191 - 060113

Command execution timestamp:____________Nov 24, 2008 5:31:30 PM

------------------------------------------------------------

ARRAY INFORMATION

Array Status:_________________________Warning

Firmware Revision:____________________38370A140P0513051631

Product Revision:_____________________A140

Local Controller Product Revision:____A140

Remote Controller Product Revision:___A140

Last Event Log Entry for Page 1:______140908

Last Event Log Entry for Page 2:______140896

Last Event Log Entry for Page 5:______131219

ENCLOSURES

Enclosure at M

Enclosure ID__________________________0

Enclosure Status______________________Failed

Enclosure Type________________________HP StorageWorks Virtual Array 7110

Node WWN______________________________50060b00001535a8

FRU HW COMPONENT IDENTIFICATION ID STATUS

===========================================================================

M Enclosure 00SG324J0103 Failed

M/P1 Power Supply 94030JD01148 Good

M/P2 Power Supply 94030JD01145 Good

M/MP1 MidPlane 000617570117 Good

M/C2 Controller 00PR00D50084 Good

M/C2.H1 Host Port Good

M/C2.J1 BackEnd Port Good

M/C2.B1 Battery 44298:MOLTECHPS:NI2040:2003/3/12 Good

M/C2.PM1 Processor HP:A6189B:A140 Good

M/C2.M1 DIMM 512 Good

M/C2.M2 DIMM 512 Good

M/C1 Controller 00PR00D50060 Good

M/C1.H1 Host Port Good

M/C1.J1 BackEnd Port Good

M/C1.B1 Battery 44304:MOLTECHPS:NI2040:2003/3/12 Good

M/C1.PM1 Processor HP:A6189B:A140 Good

M/C1.M1 DIMM 512 Good

M/C1.M2 DIMM 512 Good

M/D1 Disk 3HX0WX3G Good

M/D2 Disk 3HX0X2RN Good

M/D3 Disk 3HX0XBBG Failed

M/D4 Disk 3HX0X625 Good

CONTROLLERS

Controller At M/C2:

Status:_______________________________Good

Serial Number:________________________00PR00D50084

Vendor ID:____________________________HP

Product ID:___________________________A6189B

Product Revision:_____________________A140

Firmware Revision:____________________38370A140P0513051631

Manufacturing Product Code:___________IJMTU00016

Controller Type:______________________HP StorageWorks Virtual Array 7110

Battery Charger Firmware Revision:____5.0

Front Port At M/C2.H1:

Status:_____________________________Good

Port Instance:______________________0

Hard Address:_______________________126

Link State:_________________________Link Up

Node WWN:___________________________50060b00001535a8

Port WWN:___________________________50060b00001b7a28

Topology:___________________________Point To Point, Fabric Attached

Data Rate:__________________________2 GBit/sec

Port ID:____________________________0x10000

Device Host Name:___________________zhsmp1

Hardware Path:______________________0/6/2/0.1.0.0.0.0.0

Device Path:________________________/dev/dsk/c6t0d0

Back Port At M/C2.J1:

Status:_____________________________Good

Port Instance:______________________0

Hard Address:_______________________125

Link State:_________________________Link Up

Node WWN:___________________________50060b00001535a8

Port WWN:___________________________50060b00001b7a29

Topology:___________________________Private Loop

Data Rate:__________________________2 GBit/sec

Port ID:____________________________125

Battery at M/C2.B1:

Status:_____________________________Good

Identification:_____________________44298:MOLTECHPS:NI2040:2003/3/12

Manufacturer Name:__________________MOLTECHPS

Device Name:________________________NI2040

Manufacturer Date:__________________March 12, 2003

Remaining Capacity:_________________5700 mAh

Remaining Capacity:_________________95 %

Voltage:____________________________12349 mVolts

Discharge Cycles:___________________2

Processor at M/C2.PM1:

Status:_____________________________Good

Identification:_____________________HP:A6189B:A140

DIMM at M/C2.M1:

Status:_____________________________Good

Identification:_____________________512

Capacity:___________________________512 MB

DIMM at M/C2.M2:

Status:_____________________________Good

Identification:_____________________512

Capacity:___________________________512 MB

Controller At M/C1:

Status:_______________________________Good

Serial Number:________________________00PR00D50060

Vendor ID:____________________________HP

Product ID:___________________________A6189B

Product Revision:_____________________A140

Firmware Revision:____________________38370A140P0513051631

Manufacturing Product Code:___________IJMTU00016

Controller Type:______________________HP StorageWorks Virtual Array 7110

Battery Charger Firmware Revision:____5.0

Front Port At M/C1.H1:

Status:_____________________________Good

Port Instance:______________________0

Hard Address:_______________________126

Link State:_________________________Link Up

Node WWN:___________________________50060b00001535a8

Port WWN:___________________________50060b00001b67ac

Topology:___________________________Point To Point, Fabric Attached

Data Rate:__________________________2 GBit/sec

Port ID:____________________________0x10000

Device Host Name:___________________zhsmp1

Hardware Path:______________________0/4/0/0.1.0.0.0.0.0

Device Path:________________________/dev/dsk/c4t0d0

Back Port At M/C1.J1:

Status:_____________________________Good

Port Instance:______________________0

Hard Address:_______________________125

Link State:_________________________Link Up

Node WWN:___________________________50060b00001535a8

Port WWN:___________________________50060b00001b67ad

Topology:___________________________Private Loop

Data Rate:__________________________2 GBit/sec

Port ID:____________________________125

Battery at M/C1.B1:

Status:_____________________________Good

Identification:_____________________44304:MOLTECHPS:NI2040:2003/3/12

Manufacturer Name:__________________MOLTECHPS

Device Name:________________________NI2040

Manufacturer Date:__________________March 12, 2003

Remaining Capacity:_________________5821 mAh

Remaining Capacity:_________________97 %

Voltage:____________________________12575 mVolts

Discharge Cycles:___________________2

Processor at M/C1.PM1:

Status:_____________________________Good

Identification:_____________________HP:A6189B:A140

DIMM at M/C1.M1:

Status:_____________________________Good

Identification:_____________________512

Capacity:___________________________512 MB

DIMM at M/C1.M2:

Status:_____________________________Good

Identification:_____________________512

Capacity:___________________________512 MB

PORTS

Settings for port M/C2.H1:

Port ID:______________________________108

Behavior:_____________________________HPUX

Topology:_____________________________Point To Point, Fabric Attached

Queue Full Threshold:_________________4

Data Rate:____________________________2 GBit/sec

Settings for port M/C2.J1:

Data Rate:____________________________2 GBit/sec

Settings for port M/C1.H1:

Port ID:______________________________110

Behavior:_____________________________HPUX

Topology:_____________________________Point To Point, Fabric Attached

Queue Full Threshold:_________________4

Data Rate:____________________________2 GBit/sec

Settings for port M/C1.J1:

Data Rate:____________________________2 GBit/sec

DISKS

Disk at M/D1:

Status:_______________________________Good

Disk State:___________________________Included

Vendor ID:____________________________HP 36.4G

Product ID:___________________________ST336753FC

Product Revision:_____________________HP03

Data Capacity:________________________33.378 GB (70000000 blocks)

Block Length:_________________________520 bytes

Address:______________________________111

Node WWN:_____________________________2000000c5029bcf8

Initialize State:_____________________Ready

Redundancy Group:_____________________1

Volume Set Serial Number:_____________0000C38C0000000A

Serial Number:________________________3HX0WX3G

Firmware Revision:____________________HP03

Recovery Maps are on this disk.

Space is reserved on this disk for subsystem metadata and

may be a map disk.

Disk at M/D2:

Status:_______________________________Good

Disk State:___________________________Included

Vendor ID:____________________________HP 36.4G

Product ID:___________________________ST336753FC

Product Revision:_____________________HP03

Data Capacity:________________________33.378 GB (70000000 blocks)

Block Length:_________________________520 bytes

Address:______________________________112

Node WWN:_____________________________2000000c5029bcca

Initialize State:_____________________Ready

Redundancy Group:_____________________1

Volume Set Serial Number:_____________0000C38C0000000A

Serial Number:________________________3HX0X2RN

Firmware Revision:____________________HP03

Recovery Maps are on this disk.

Space is reserved on this disk for subsystem metadata and

may be a map disk.

Disk at M/D3:

Status:_______________________________Failed

Disk State:___________________________Failed

Vendor ID:____________________________HP 36.4G

Product ID:___________________________ST336753FC

Product Revision:_____________________HP03

Data Capacity:________________________33.378 GB (70000000 blocks)

Block Length:_________________________520 bytes

Address:______________________________113

Node WWN:_____________________________2000000c5029886b

Initialize State:_____________________Ready

Redundancy Group:_____________________1

Volume Set Serial Number:_____________0000C38C0000000A

Serial Number:________________________3HX0XBBG

Firmware Revision:____________________HP03

Disk at M/D4:

Status:_______________________________Good

Disk State:___________________________Included

Vendor ID:____________________________HP 36.4G

Product ID:___________________________ST336753FC

Product Revision:_____________________HP03

Data Capacity:________________________33.378 GB (70000000 blocks)

Block Length:_________________________520 bytes

Address:______________________________114

Node WWN:_____________________________2000000c5029924c

Initialize State:_____________________Ready

Redundancy Group:_____________________1

Volume Set Serial Number:_____________0000C38C0000000A

Serial Number:________________________3HX0X625

Firmware Revision:____________________HP03

LUNS

LUN 0:

Redundancy Group:_____________________1

Active:_______________________________True

Data Capacity:________________________20 MB

WWN:__________________________________60060b00001535a80000000000000010

Number Of Business Copies:____________0

LUN 1:

Redundancy Group:_____________________1

Active:_______________________________True

Data Capacity:________________________11 GB

WWN:__________________________________60060b00001535a80001000000000011

Number Of Business Copies:____________0

LUN 2:

Redundancy Group:_____________________1

Active:_______________________________True

Data Capacity:________________________16 GB

WWN:__________________________________60060b00001535a80002000000000012

Number Of Business Copies:____________0

LUN 3:

Redundancy Group:_____________________1

Active:_______________________________True

Data Capacity:________________________11 GB

WWN:__________________________________60060b00001535a80003000000000013

Number Of Business Copies:____________0

LUN 4:

Redundancy Group:_____________________1

Active:_______________________________True

Data Capacity:________________________16 GB

WWN:__________________________________60060b00001535a80004000000000014

Number Of Business Copies:____________0

CAPACITY Totals for Redundancy Group 1:

REGULAR LUNs:_________________________54.019 GB

BUSINESS COPIES:______________________0 bytes

CAPACITY USAGE

Total Disk Enclosures:________________1

Redundancy Group:_____________________1

Total Disks:________________________3

Total Physical Size:________________100.135 GB

Allocated to Regular LUNs:__________54.019 GB

Allocated as Business Copies:_______0 bytes

Used as Active Hot Spare:___________0 bytes

Used for Redundancy:________________46.116 GB

Unallocated (Available for LUNs):___0 bytes

Used by Non-Included Disks:___________33.378 GB

VFP

Settings for VFP Serial Port M/C1.VFP:

VFP Baud Rate:________________________9600

VFP Paging Value:_____________________24

Settings for VFP Serial Port M/C2.VFP:

VFP Baud Rate:________________________9600

VFP Paging Value:_____________________24

SUB-SYSTEM SETTINGS

RAID Level:___________________________RAID1+0

Auto Format Drive:____________________On

Hang Detection:_______________________On

Capacity Depletion Threshold:_________100%

Queue Full Threshold Maximum:_________4096

Enable Optimize Policy:_______________True

Enable Manual Override:_______________False

Manual Override Destination:__________False

Read Cache Disable:___________________False

Rebuild Priority:_____________________Low

Security Enabled:_____________________False

Shutdown Completion:__________________0

Subsystem Type ID:____________________0

Unit Attention:_______________________True

Volume Set Partition (VSpart):________False

Write Cache Enable:___________________True

Write Working Set Interval:___________8640

Enable Prefetch:______________________True

Disable Secondary Path Presentation:__False

Backend Diagnostics:__________________On

RESILIENCY SETTINGS

Simplified Resiliency Setting:________Normal Performance (Default)

Enable Secure Mode:___________________True

Disable NVRAM on UPS Absent:__________False

Disable NVRAM on WCE False:___________False

Disable Read Hits:____________________False

Force Unit Access Response:___________1

Lock Write Cache On:__________________True

Performance Goal Configuration:_______Normal Performance

Resiliency Threshold:_________________4

Single Controller Warning:____________True

DISK SETTINGS

Auto Include:_________________________On

Auto Rebuild:_________________________On

Hot Spare:____________________________None

Max Drives per Loop Pair:_____________45

Max Drives per Subsystem:_____________45

ENCLOSURE SETTINGS

Max Enclosures per Loop Pair:_________2

Max Enclosures per Subsystem:_________2

LUN SETTINGS

LUN Creation Limit:___________________1024

Maximum LUN Creation Limit:___________1024

Migrating Write Destination:__________False

CRUB SETTINGS

Scrub Restart Period:_________________0 minutes

Scrub State:__________________________Not running, system in warning state

OPERATIONS IN PROGRESS

None

WARNINGS

WARNING: Unallocated capacity has fallen below the threshold specified by the capacity threshold mode parameter.

WARNING: A physical drive has failed, failed initialization, been downed or is in the previously used state.

WARNING: A rebuild operation failed.

WARNING: Some data in the device lacks redundancy, and is exposed to becoming unavailable if further drive removals or failures occur.

三、 分析问题原因及解决办法

原因：

此次造成rebuild失败的原因是VA的剩余空间不足，一共4个硬盘做RAID1+0，没有热备盘，所以更换硬盘也不能Rebuild成功，接着VA做balance动作，每当前端有I/O读写时balance动作就暂停，经过一周的balance仍然未完成。

解决办法：

由于造成rebuild失败的原因是VA本身剩余空间不足，因此根据HP的建议有以下办法：

1、备份数据，格式化VA，重建所有的LUN并恢复数据。

2、删除无用的LUN，以便腾出空间加快balance完成。

3、停止前端I/O读写，加快balance完成后手工rebuild。

4、新增两块硬盘，. 以便有更多的剩余空间，加快balance完成。

5、将raid1+0转换为raid5，以便腾出空间加快balance完成。

四、 处理过程

以上5种处理办法中办法2和办法5显然是不可取的，考虑到业务可以暂时停下来，首先采取了办法3，即暂时停掉业务和数据库，经过12月3日一个晚上balcnce仍未完成，业务还是受影响，效果不明显。于是12月4日采取办法2，处理过程如下：

1、备份卷组配置信息

#vgcfgbackup vg01

#vgcfgbackup vg02

#vgcfgbackup vglock

2、停止业务，数据库

#cmhaltcl

3、 格式化VA

#armfmt -f va

4、重建VA的LUN，跟原来一样

#armcfg -L 0 -a 20M -g 1 va

#armcfg -L 1 -a 11G -g 1 va

#armcfg -L 2 -a 16G -g 1 va

#armcfg -L 3 -a 11G -g 1 va

#armcfg -L 4 -a 16G -g 1 va

5 、扫描硬盘，以认到以上几个lun

#insf -C disk

6、恢复卷组、逻辑卷信息

#vgcfgrestore -n vg01 /dev/rdsk/c4t0d1

#vgcfgrestore -n vg01 /dev/rdsk/c4t0d3

#vgcfgrestore -n vg02 /dev/rdsk/c4t0d2

#vgcfgrestore -n vg02 /dev/rdsk/c4t0d4

#vgcfgrestore -n vglock /dev/rdsk/c4t0d0

7、起双机服务，激活卷组

#cmruncl –v

#vgchange -a y vg01

#vgchange -a y vg02

#vgchange -a y vglock

8、 恢复数据库

#su – Informix

$ontape –r

y

n

n

n

9、起双机包、切换测试……

#cmrunpkg –e –v pkg_smp

[@more@]

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/9479798/viewspace-1050072/，如需转载，请注明出处，否则将追究法律责任。