ges resource dynamic和ges enqueues占用较高导致实例终止问题处理

数据库告警日志分析分析
通过对数据库告警日志查看，节点1出现ORA-29970错误，LMD进程无法响应，被LMHB进程将实例终止，实例终止后自动启动。节点2被LMS0进程将实例终止，实例终止后自动启动。

1节点：
Mon Jun 07 01:22:14 2021
LMD0 (ospid: 10613) has not called a wait for sub 0 secs.
LMD1 (ospid: 10617) has not called a wait for sub 0 secs.
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_lmhb_10637.trc  (incident=688375) (PDBNAME=CDB$ROOT):
ORA-29770: global enqueue process LMD0 (OSID 10613) is hung for more than 70 seconds
Incident details in: /u01/app/oracle/diag/rdbms/cdb/cdb1/incident/incdir_688375/cdb1_lmhb_10637_i688375.trc
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_lmhb_10637.trc  (incident=688376) (PDBNAME=CDB$ROOT):
ORA-29770: global enqueue process LMD1 (OSID 10617) is hung for more than 70 seconds
Incident details in: /u01/app/oracle/diag/rdbms/cdb/cdb1/incident/incdir_688376/cdb1_lmhb_10637_i688376.trc
LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+DB_CDB by memno 0
ERROR: Some process(s) is not making progress.
LMHB (ospid: 10637) is terminating the instance.
Please check LMHB trace file for more details.
Please also check the CPU load, I/O load and other system properties for anomalous behavior
ERROR: Some process(s) is not making progress.
LMHB (ospid: 10637): terminating the instance due to error 29770
Mon Jun 07 01:22:24 2021
System state dump requested by (instance=1, osid=10637 (LMHB)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_diag_10575_20210607012224.trc
Mon Jun 07 01:22:26 2021
License high water mark = 591
Mon Jun 07 01:22:29 2021
Instance terminated by LMHB, pid = 10637
Mon Jun 07 01:22:29 2021
USER (ospid: 21601): terminating the instance
Mon Jun 07 01:22:29 2021
Instance terminated by USER, pid = 21601
Mon Jun 07 01:22:32 2021
Starting ORACLE instance (normal) (OS id: 21660)
Mon Jun 07 01:22:32 2021
CLI notifier numLatches:37 maxDescs:3986
Mon Jun 07 01:22:32 2021
**********************************************************************
Mon Jun 07 01:22:32 2021
Dump of system resources acquired for SHARED GLOBAL AREA (SGA) Mon Jun 07 01:22:32 2021Per process system memlock (soft) limit = UNLIMITED
Mon Jun 07 01:22:32 2021Expected per process system memlock (soft) limit to lockSHARED GLOBAL AREA (SGA) into memory: 128G
Mon Jun 07 01:22:32 2021Available system pagesizes:4K, 2048K
Mon Jun 07 01:22:32 2021Supported system pagesize(s):
Mon Jun 07 01:22:32 2021PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
Mon Jun 07 01:22:32 20212048K            66823           65538           65538        NONE
Mon Jun 07 01:22:32 2021Reason for not supporting certain system pagesizes:
Mon Jun 07 01:22:32 20214K - Large pagesizes only
Mon Jun 07 01:22:32 2021
**********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 242节点：
Mon Jun 07 01:25:05 2021Set master node info
Mon Jun 07 01:26:05 2021
Auto-tuning: Shutting down background process GTXb
Mon Jun 07 01:27:53 2021
IPC Send timeout detected. Sender: ospid 62152 [oracle@node1 (PING)]
Receiver: inst 1 binc 912546309 ospid 10589
Mon Jun 07 01:29:25 2021
LMD0 (ospid: 62168) received an instance eviction notification from instance 1 [2]
Mon Jun 07 01:29:26 2021
Received an instance abort message from instance 1
Mon Jun 07 01:29:26 2021
Received an instance abort message from instance 1
Mon Jun 07 01:29:26 2021
Please check instance 1 alert and LMON trace files for detail.
Mon Jun 07 01:29:26 2021
Please check instance 1 alert and LMON trace files for detail.
Mon Jun 07 01:29:26 2021
LMS0 (ospid: 62192): terminating the instance due to error 481
Mon Jun 07 01:29:26 2021
System state dump requested by (instance=2, osid=62192 (LMS0)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/cdb/cdb2/trace/cdb2_diag_62123_20210607012926.trc
Mon Jun 07 01:29:26 2021
ORA-1092 : opitsk aborting process
Mon Jun 07 01:29:27 2021
License high water mark = 1251
Mon Jun 07 01:29:31 2021
Instance terminated by LMS0, pid = 62192
Mon Jun 07 01:29:31 2021
USER (ospid: 85262): terminating the instance
Mon Jun 07 01:29:31 2021
Instance terminated by USER, pid = 85262
Mon Jun 07 01:29:33 2021
Starting ORACLE instance (normal) (OS id: 85397)
Mon Jun 07 01:29:33 2021
CLI notifier numLatches:37 maxDescs:3986
Mon Jun 07 01:29:33 2021
**********************************************************************
Mon Jun 07 01:29:33 2021
Dump of system resources acquired for SHARED GLOBAL AREA (SGA) Mon Jun 07 01:29:33 2021Per process system memlock (soft) limit = UNLIMITED
Mon Jun 07 01:29:33 2021Expected per process system memlock (soft) limit to lockSHARED GLOBAL AREA (SGA) into memory: 128G
Mon Jun 07 01:29:33 2021Available system pagesizes:4K, 2048K
Mon Jun 07 01:29:33 2021Supported system pagesize(s):
Mon Jun 07 01:29:33 2021PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
Mon Jun 07 01:29:33 20212048K            67200           65538           65538        NONE
Mon Jun 07 01:29:33 2021Reason for not supporting certain system pagesizes:
Mon Jun 07 01:29:33 20214K - Large pagesizes only
Mon Jun 07 01:29:33 2021
**********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 24

查看更多日志，发现在关闭之前，出现了较多的ORA-04031错误。

Mon Jun 07 01:18:14 2021
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_ora_14795.trc  (incident=691727) (PDBNAME=CDB$ROOT):
ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1069","kglseshtTable")
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_lmd0_10613.trc  (incident=707486) (PDBNAME=CDB$ROOT):
ORA-04031: unable to allocate 8504 bytes of shared memory ("shared pool","unknown object","sga heap(5,0)","ges big msg pool")
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jun 07 01:18:16 2021
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_ora_14832.trc  (incident=691688) (PDBNAME=CDB$ROOT):
ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^320","kglseshtTable")
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jun 07 01:18:16 2021
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_ora_14837.trc  (incident=691735) (PDBNAME=CDB$ROOT):
ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1234","kglseshtTable")
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jun 07 01:18:18 2021
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_ora_14912.trc  (incident=691871) (PDBNAME=CDB$ROOT):
ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^159","kglseshtTable")
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_lmd0_10613.trc  (incident=707487) (PDBNAME=CDB$ROOT):
ORA-04031: unable to allocate 8504 bytes of shared memory ("shared pool","unknown object","sga heap(5,0)","ges big msg pool")
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
DDE: Problem Key 'ORA 4031' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
Mon Jun 07 01:18:33 2021
Errors in file /u01/app/oracle/diag/rdbms/cdb/cdb1/trace/cdb1_m000_15479.trc:
ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^2147","kglseshtTable")

AWR报告信息分析

拉取了宕机前半小时的AWR报告，从AWR报告中得知，shared pool中ges enqueues和ges resource dynamic分别达到了18G和16G。
Oracle官方参考文献

根据Oracle官方文档记载：ORA-04031 Errors Occurring with High “ges resource dynamic” & “ges enqueues” Memory Usage In The Shared Pool (Doc ID 2063751.1)，该问题是Oracle的BUG，该BUG出现于Oracle RAC 12.1.0.1至12.1.0.2，此问题在Oracle RAC 12.2中被修复，当前数据库版本为Oracle RAC 12.1.0.2。
处理操作
根据Oracle官方参考文献，解决该问题有两种方式：临时解决和永久解决。
a) 临时解决方案
修改数据库参数，_GES_DIRECT_FREE_RES_TYPE=“CTARAHDXBB”，该参数是静态参数，需要重启数据库生效，可采取2个节点轮流修改的方法，减少业务停机时间，推荐采用此方案。
b) 永久解决方案
安装数据库补丁，补丁号Patch:21260431，需要安排业务停机时间安装补丁。

ges resource dynamic和ges enqueues占用较高导致实例终止问题处理相关推荐

ges resource dynamic 和 ges enqueues较高导致数据库宕机
数据库版本: Oracle RAC 12.1.0.2 数据库架构 ODA 客户反应业务系统连不上数据库,登上数据库检查数据库状态,发现所有PDB均为MOUNT状态. [oracle@node0 ~]$ ...
Windows10 关于系统中断CPU占用过高导致电脑变卡的解决办法
Windows10 关于系统中断CPU占用过高导致电脑变卡的解决办法最近一段时间笔记本一直很卡,不管打开几个程序,任务管理器中总会有CPU占用80%以上,这一度让我抓狂,开始网上搜教程,然后开始 ...
ubuntu 内存占用过高导致卡死解决办法
前言如果本文存在问题,请指出,谢谢! 文章目录前言一.具体表现二.原因三.解决方案四.其他优化总结一.具体表现例如下图注:图示版本为ubuntu18.04,现在我用的是ubuntu ...
win10 dcom服务器进程占用cpu,win10DCOM进程及资源管理CPU占用过高导致主机卡顿
您好, 了解到您进程占用的问题. 请问您是在什么时候遇到这个问题的,在出现这个问题之前,您在系统下进行过何种操作?(比如:更新,升级设备驱动,修改系统配置,使用三方优化类软件进行系统清理等)目前,您可 ...
关闭Microsoft Compatibility Telemetry服务解决VScode CPU内存占用过高导致电脑卡顿（实测有效）
解决方法: 关闭Microsoft Compatibility Telemetry服务. Microsoft Compatibility Telemetry是微软下的一个监测数据收集服务,如果加入Mi ...
svchost占用内存过高_「电脑常用技巧」电脑卡的原因（占用过高怎么办）？
大部分人在使用电脑时候,对于卡的定义都没什么概念.因为所谓的电脑卡实际上是有很多种原因导致的关于常见的数据占用过高导致电脑卡顿的原因和解决技巧. 检测电脑数据使用的管理器(任务管理器)打开方式:ct ...
无盘服务器内存问题,无盘服务器内存占用过高
无盘服务器内存占用过高内容精选换一换 Windows操作系统云服务器蓝屏,如图1所示.使用了来源不明的第三方软件.CPU占用过高导致.因为误操作或者病毒引起的系统文件.注册表损坏.操作系统在蓝屏的 ...
内存占用过高，缓存不释放导致死机处理方案
故障现象: 1.某分行部署的某台服务器内存占用过高,导致死机: 2.代码层面检查暂未发现问题,服务器硬重启持续一段时间后(3-5天)再次占满. 发现问题: 赶往现场后进行检查,当时是一切正常的,今有D ...
oracle lms进程内存,【案例】Oracle ges resource消耗内存高报错ORA-04031 MOS解决办法...
天萃荷净 Oracle研究中心案例分析:运维DBA反映Oracle数据库10.2.0.4.12每间隔一段时间就必须重启,运行一断时间报ORA-04031错误oracle ges res cache l ...

ges resource dynamic和ges enqueues占用较高导致实例终止问题处理

ges resource dynamic和ges enqueues占用较高导致实例终止问题处理相关推荐

最新文章

热门文章