oracle crs 4535,11gRAC报错CRS-4535,CRS-4000解决

1.故障现象

使用crsctl查看集群各资源状态，在任一节点都会直接报错CRS-4535,

CRS-4000；但此时数据库是可以被正常访问的。

具体故障现象如下：

＃节点1查询

grid@bjdb1:/home/grid>crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Status failed, or completed with errors.

＃节点2查询

root@bjdb2:/>crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Status failed, or completed with errors.

同样的，crs_stat -t 查看一样报错，错误码是CRS－0184：

root@bjdb1:/>crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

节点2也一样！

确定此时数据库是可以被正常访问的。如下：

＃节点2模拟客户端登录RAC集群，使用SCAN

IP访问，发现可以正常访问到数据库 oracle@bjdb2:/home/oracle>sqlplus

jingyu/jingyu@192.168.103.31/bjdb

SQL*Plus: Release

11.2.0.4.0

Production on Mon Oct 10

14:24:47

2016

2013, Oracle. All rights

reserved.

Connected to: Oracle Database

11g Enterprise Edition Release

11.2.0.4.0 -

64bit Production With the

Partitioning, Real Application Clusters, Automatic Storage

Management, OLAP, Data Mining and

Real Application Testing options

SQL>

2.定位问题

首先查看节点1的集群相关日志：

Clusterware(GI)的日志存放在$GRID_HOME/log/nodename下；

Clusterware(GI)对应几个关键的后台进程css，crs，evm，它们的日志分别存在cssd，crsd，evmd目录下；

节点1查看相关日志：

＃查看GI的alert日志文件，最近的记录只是提示GI所在存储空间使用率高，稍后清理下即可，而且目前还有一定空间剩余，显然并非是此次故障的原因。

root@bjdb1:/opt/u01/app/11.2.0/grid/log/bjdb1>tail -f alert*.log

2016-10-10 14:18:26.125:

[crflogd(39190674)]CRS-9520:The storage of Grid Infrastructure Management Repository is 93% full. The storage location is '/opt/u01/app/11.2.0/grid/crf/db/bjdb1'.

2016-10-10 14:23:31.125:

[crflogd(39190674)]CRS-9520:The storage of Grid Infrastructure Management Repository is 93% full. The storage location is '/opt/u01/app/11.2.0/grid/crf/db/bjdb1'.

2016-10-10 14:28:36.125:

[crflogd(39190674)]CRS-9520:The storage of Grid Infrastructure Management Repository is 93% full. The storage location is '/opt/u01/app/11.2.0/grid/crf/db/bjdb1'.

2016-10-10 14:33:41.125:

[crflogd(39190674)]CRS-9520:The storage of Grid Infrastructure Management Repository is 93% full. The storage location is '/opt/u01/app/11.2.0/grid/crf/db/bjdb1'.

2016-10-10 14:38:46.125:

[crflogd(39190674)]CRS-9520:The storage of Grid Infrastructure Management Repository is 93% full. The storage location is '/opt/u01/app/11.2.0/grid/crf/db/bjdb1'.

＃因为crsctl不可以使用，进而查看crs的日志信息，发现3号已经有报错，无法打开裸设备，从而导致无法初始化OCR；继续看错误信息，发现是这个时候访问共享存储时无法成功。怀疑此刻存储出现问题，需要进一步和现场人员确定此时间点是否有存储相关的施工。

root@bjdb1:/opt/u01/app/11.2.0/grid/log/bjdb1/crsd>tail -f crsd.log

2016-10-03 18:04:40.248:

[OCRRAW][1]proprinit: Could not open raw device

2016-10-03 18:04:40.248:

[OCRASM][1]proprasmcl: asmhandle is NULL

2016-10-03 18:04:40.252:

[OCRAPI][1]a_init:16!: Backend init unsuccessful : [26]

2016-10-03 18:04:40.253:

[CRSOCR][1] OCR context init failure. Error: PROC-26: Error while accessing the physical storage

2016-10-03 18:04:40.253:

[CRSD][1] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage

2016-10-03 18:04:40.253:

[CRSD][1][PANIC] CRSD exiting: Could not init OCR, code: 26

2016-10-03 18:04:40.253: [CRSD][1] Done.

节点2查看相关日志：

＃查看GI的alert日志，发现节点2的ctss有CRS-2409的报错，虽然根据MOS文档 ID 1135337.1

说明，This is not an error. ctssd is reporting that there is a time difference and it is not doing anything about it as it is running in observer mode.

只需要查看两个节点的时间是否一致，但实际上查询节点时间一致:

root@bjdb2:/opt/u01/app/11.2.0/grid/log/bjdb2>tail -f alert*.log

2016-10-10 12:29:22.145: