RAC环境报错ORA-00239: timeout waiting for control file enqueue排查

近期，有一用户的RAC环境反映业务缓慢；排查数据库日志发现有报错ORA-00239: timeout waiting for control file enqueue: held by 'inst 2, osid 7591' for more than 900 seconds；检查当时系统负载非常低；通过alert日志中报错的进程（这里是inst 2, osid 7591），查出持有control file enqueue的为LGWR进程；进一步排查LGWR进程TRACE日志，可以发现有大量IO缓慢的提示。

在此前提下，排查OS日志，也有些磁盘的报错信息；进一步与硬件维护人员排查确认，通过更换主机的HBA卡，之后系统恢复正常。

相关日志如下：

1.ALERT日志中的ORA-00239: timeout waiting for control file enqueue信息
Sat Feb 19 21:47:36 2022
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc0_8317.trc  (incident=1154998):
ORA-00239: timeout waiting for control file enqueue: held by 'inst 2, osid 7591' for more than 900 seconds
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_1154998/orcl1_arc0_8317_i1154998.trc
opidrv aborting process ARC0 ospid (8317) as a result of ORA-603
Sat Feb 19 21:47:37 2022
Sweep [inc][1154998]: completed
Sweep [inc2][1154998]: completed
Sat Feb 19 21:52:52 2022
Suspending MMON slave action ketbgact_ for 82800 seconds
Sat Feb 19 21:53:55 2022
Suspending MMON action 'undo usage' for 82800 seconds2.检查数据库中持有锁的进程信息[oracle@CRXJ-DB-2 ~]$ cd /u01/app/oracle/diag/rdbms/orcl/orcl2/trace
[oracle@CRXJ-DB-2 trace]$ ls *7591*
orcl2_lgwr_7591.trc  orcl2_lgwr_7591.trm
[oracle@CRXJ-DB-2 trace]$ ls -al orcl2_lgwr_7591.trc
-rw-r----- 1 oracle asmadmin 748774 Feb 19 20:00 orcl2_lgwr_7591.trc
3.查看LGWR进程TRACE信息
[oracle@CRXJ-DB-2 trace]$ tail -n 50 orcl2_lgwr_7591.trc
…………*** 2022-02-19 17:02:47.221
Warning: log write elapsed time 837ms, size 1KB(set event 10468 level 4 to disable this warning)*** 2022-02-19 19:10:58.951
Warning: log write elapsed time 516ms, size 1058KB(set event 10468 level 4 to disable this warning)*** 2022-02-19 19:22:50.852
Warning: log write elapsed time 534ms, size 0KB(set event 10468 level 4 to disable this warning)*** 2022-02-19 20:00:55.692
Warning: log write broadcast wait time 3354ms(set event 10468 level 8 to disable this warning)4.查看操作系统日志：Feb 20 13:00:48 crxj-db-2 kernel: 6>sd 3:0:0:16: [sdq] Sense Key : Illegal Request [current]
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:16: [sdq]
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:10: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:10: [sdk] Sense Key : Illegal Request [current]
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:10: [sdk] <<vendor>> ASC=0x94 ASCQ=0x1<<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:16: [sdq] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Feb 20 13:00:48 crxj-db-2 kernel: end_request: I/O error, dev sdq, sector 0
Feb 20 13:00:48 crxj-db-2 kernel: ASC=0x94 ASCQ=0x1
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:10: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Feb 20 13:00:48 crxj-db-2 kernel: end_request: I/O error, dev sdk, sector 0
…………
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:18: [sds] Sense Key : Illegal Request [current]
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:18: [sds] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:18: [sds] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Feb 20 13:00:48 crxj-db-2 kernel: end_request: I/O error, dev sds, sector 0
Feb 20 13:00:48 crxj-db-2 kernel: [current]
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:2: [sdc] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
Feb 20 13:00:48 crxj-db-2 kernel: sd 3:0:0:2: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00

RAC环境报错ORA-00239: timeout waiting for control file enqueue排查相关推荐

STARTUP报错：ORA-00205: error in identifying control file, check alert log for more info
问题描述 startup nomount没有异常 startup mount则出现异常,说明控制文件出错
Oracle 11G R2 RAC 启动报错：ORA-01078 ORA-01565 ORA-17503 ORA-12547 处理方法
Oracle 11G R2 RAC 启动报错:ORA-01078 ORA-01565 ORA-17503 ORA-12547 处理方法前几天搭建的RAC测试环境:RedHat 6.8 grid ...
mysql 5.6 主主复制_Percona MySQL 5.6 主主复制环境报错Got fatal error 1236 from master.....
Percona MySQL 5.6 主主复制环境报错: mysql> show slave status\G *************************** 1. row ******* ...
php5.6.30环境报错Call to undefined function ImageCreate() 编译安装 gd库
php5.6.30环境报错Call to undefined function ImageCreate() 编译安装 gd库发现php5.6.30没有加载gd库[root@cn_vs_web04:/u ...
Gym导入Atari游戏环境报错
由于Gym近期版本的更新,导致程序中 import Atari的游戏环境报错.Gym最新版本 0.21.0 虽然修复了一系列更新后的Bug,但是貌似只支持 Python 3.7及以上版本.于是为了能够 ...
oracle 10g rac数据库,oracle 10g rac安装报错集锦
oracle 10g rac安装报错汇总删除crs 1. 停止资源:关闭crs资源:crsctl stop resources 关闭crs: crsctl stop crs 2 ...
vue3 全局注册app.config.globalProperties，如何处理getCurrentInstance 上下文线上环境报错
vue3挂载全局属性和方法,使用app.config.globalProperties, 但是在组件中获取全局上下文的时候getCurrentInstance会在线上环境报错. 搜索了一遍,看了好多文 ...
设备通过国标GB28181协议接入EasyCVR，报错“504 Gateway Time-out”是什么原因？
EasyCVR基于云边端一体化架构,能支持海量视频的轻量化接入与汇聚管理.在视频能力上,可提供视频监控直播.视频轮播.视频录像.云存储.回放与检索.智能告警.服务器集群.语音对讲.云台控制.电子地图. ...
ESP32 VScode环境报错：Something went wrong while trying to build the project。解决办法
配置ESP32- Vscode环境报错如图项目目录包含了中文字符,我给换成全英文路径,再次尝试成功一些使用技巧不要学我当星际玩家,找修改端口号找了半天,最后发群里丢人才发现
【数据库】报错ora 12154 解决方法
Oracle报错 ora 12154 解决办法: 1.确认服务器监听正确打开: 2.确认别名前无空格.

RAC环境报错ORA-00239: timeout waiting for control file enqueue排查

RAC环境报错ORA-00239: timeout waiting for control file enqueue排查相关推荐

最新文章

热门文章