0 CRS                                                      0DISMOUNTED

2 DATA1                          EXTERN              4194304 MOUNTED

查看asm实例alert日志，返现CRS磁盘组被强制卸载了

SQL> show parameter dump

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

background_core_dump                 string      partial

background_dump_dest                 string      /app/gridbase/diag/asm/+asm/+A

SM1/trace

cd /app/gridbase/diag/asm/+asm/+ASM1/trace

$ vim alert_+ASM1.log

-------------------------------------------

Tue Dec 10 11:13:57 2013

WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.

WARNING: Waited 15 secs for write IO to PST disk 1 in group 1.

WARNING: Waited 15 secs for write IO to PST disk 2 in group 1.

WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.

WARNING: Waited 15 secs for write IO to PST disk 1 in group 1.

WARNING: Waited 15 secs for write IO to PST disk 2 in group 1.

Tue Dec 10 11:13:57 2013

NOTE: process _b000_+asm1 (15822) initiating offline of disk 0.3916226472 (CRS_0000) with mask 0x7e in group 1

NOTE: process _b000_+asm1 (15822) initiating offline of disk 1.3916226471 (CRS_0001) with mask 0x7e in group 1

NOTE: process _b000_+asm1 (15822) initiating offline of disk 2.3916226470 (CRS_0002) with mask 0x7e in group 1

NOTE: checking PST: grp = 1

GMON checking disk modes for group 1 at 12 for pid 37, osid 15822

ERROR: no read quorum in group: required 2, found 0 disks

NOTE: checking PST for grp 1 done.

NOTE: initiating PST update: grp = 1, dsk = 0/0xe96cdfa8, mask = 0x6a, op = clear

NOTE: initiating PST update: grp = 1, dsk = 1/0xe96cdfa7, mask = 0x6a, op = clear

NOTE: initiating PST update: grp = 1, dsk = 2/0xe96cdfa6, mask = 0x6a, op = clear

GMON updating disk modes for group 1 at 13 for pid 37, osid 15822

ERROR: no read quorum in group: required 2, found 0 disks

Tue Dec 10 11:13:57 2013

NOTE: cache dismounting (not clean) group 1/0x165C2F6D (CRS)

WARNING: Offline for disk CRS_0000 in mode 0x7f failed.

WARNING: Offline for disk CRS_0001 in mode 0x7f failed.

NOTE: messaging CKPT to quiesce pins Unix process pid: 15824, image: oracle@node1 (B001)

WARNING: Offline for disk CRS_0002 in mode 0x7f failed.

Tue Dec 10 11:13:57 2013

NOTE: halting all I/Os to diskgroup 1 (CRS)

Tue Dec 10 11:13:57 2013

NOTE: LGWR doing non-clean dismount of group 1 (CRS)

NOTE: LGWR sync ABA=3.42 last written ABA 3.42

Tue Dec 10 11:13:57 2013

kjbdomdet send to inst 2

detach from dom 1, sending detach message to inst 2

Tue Dec 10 11:13:57 2013

List of instances:

1 2

Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)

Global Resource Directory partially frozen for dirty detach

* dirty detach - domain 1 invalid = TRUE

Tue Dec 10 11:13:57 2013

NOTE: No asm libraries found in the system

520 GCS resources traversed, 0 cancelled

Dirty Detach Reconfiguration complete

Tue Dec 10 11:13:57 2013

WARNING: dirty detached from domain 1

NOTE: cache dismounted group 1/0x165C2F6D (CRS)

SQL> alter diskgroup CRS dismount force /* ASM SERVER:375140205 */

Tue Dec 10 11:13:57 2013

NOTE: cache deleting context for group CRS 1/0x165c2f6d

GMON dismounting group 1 at 14 for pid 41, osid 15824

NOTE: Disk CRS_0000 in mode 0x7f marked for de-assignment

NOTE: Disk CRS_0001 in mode 0x7f marked for de-assignment

NOTE: Disk CRS_0002 in mode 0x7f marked for de-assignment

NOTE:Waiting for all pending writes to complete before de-registering: grpnum 1

Tue Dec 10 11:14:27 2013

NOTE:Waiting for all pending writes to complete before de-registering: grpnum 1

Tue Dec 10 11:14:29 2013

ASM Health Checker found 1 new failures

Tue Dec 10 11:14:57 2013

SUCCESS: diskgroup CRS was dismounted

SUCCESS: alter diskgroup CRS dismount force /* ASM SERVER:375140205 */

SUCCESS: ASM-initiated MANDATORY DISMOUNT of group CRS

--------------------------------------

挂载CRS 磁盘组

su - grid

sqlplus / assysasm--！！！一定是sysasm

SQL> alter diskgroup crs mount;

SQL> select GROUP_NUMBER,NAME,TYPE,ALLOCATION_UNIT_SIZE,STATE from v$asm_diskgroup;

GROUP_NUMBER NAME                           TYPE   ALLOCATION_UNIT_SIZE STATE

------------ ------------------------------ ------ -------------------- -----------

1 CRS                            NORMAL              4194304 MOUNTED

2 DATA1                          EXTERN              4194304 MOUNTED

启动CRS

但是常用的start crs命令执行不成功

# /app/grid/bin/crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

使用该命令启动成功

[root@node1 ~]#/app/grid/bin/crsctl start res ora.crsd -init

CRS-2672: Attempting to start 'ora.crsd' on 'node1'

CRS-2676: Start of 'ora.crsd' on 'node1' succeeded

# /app/grid/bin/crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

$ crsctl check cluster -all

**************************************************************

node1:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

node2:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

解决路线图:

crsd_log-->asm_instance_alert_log-->mount crs diskgroup -->start crs

虽然node1的CRS恢复正常，但CRS磁盘组会被强制dismount的原因还没找到，找到后会贴在这里。