Oracle v12.2 Bug 27163928触发ORA-4031 导致实例crash.

不啰嗦，具体分析过程如下：

1. alert log

2018-02-22T16:00:23.021370+08:00
Errors in file /u01/app/oracle/diag/rdbms/masaike/masaike1/trace/masaike1_m001_50771.trc (incident=5427952):
ORA-04031: unable to allocate 4224 bytes of shared memory ("shared pool","SELECT dbin.instance_number,...","TCHK^c22c7d28","kkotbp : kkoipt")
Incident details in: /u01/app/oracle/diag/rdbms/masaike/masaike1/incident/incdir_5427952/masaike1_m001_50771_i5427952.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

22日最早的4031触发在下午4点，这个时间段开始没有大量4031出现。

2018-02-22T20:05:00.915809+08:00
Errors in file /u01/app/oracle/diag/rdbms/masaike/masaike1/trace/masaike1_ora_45285.trc (incident=5433666):
ORA-04031: unable to allocate 4048 bytes of shared memory ("shared pool","declare
v_cursor integer;
...","PL/SQL MPCODE","pdy3M63_Finish_Page_Table: Copy data seg")
Incident details in: /u01/app/oracle/diag/rdbms/masaike/masaike1/incident/incdir_5433666/masaike1_ora_45285_i5433666.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
......
***********************************************************************
Fatal NI connect error 12170.

VERSION INFORMATION:
TNS for Linux: Version 12.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.2.0.1.0 - Production
Time: 22-FEB-2018 20:44:05
Tracing not turned on.
Tns error struct:
ns main err code: 12535

TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=190.123.178.68)(PORT=58152))
2018-02-22T20:44:05.727171+08:00
***********************************************************************
......

20:00左右开始出现业务进程触发4031，之后大量的连接报错，说明4031已经影响到数据库的整体性能，碎片严重。

2018-02-22T20:59:37.884506+08:00
Restarting dead background process MMON
Starting background process MMON
2018-02-22T20:59:37.904907+08:00
MMON started with pid=95, OS id=82438
2018-02-22T20:59:43.561240+08:00
opidrv aborting process CL24 ospid (83880) as a result of ORA-4031
2018-02-22T20:59:43.576138+08:00
opidrv aborting process CL25 ospid (83892) as a result of ORA-4031
2018-02-22T20:59:43.766862+08:00
opidrv aborting process CL36 ospid (83939) as a result of ORA-4031
2018-02-22T20:59:43.818709+08:00
opidrv aborting process CL39 ospid (83976) as a result of ORA-4031
2018-02-22T20:59:43.971824+08:00
opidrv aborting process CL45 ospid (84015) as a result of ORA-4031
2018-02-22T20:59:43.971862+08:00
opidrv aborting process CL46 ospid (84025) as a result of ORA-4031
2018-02-22T20:59:44.022049+08:00
opidrv aborting process CL51 ospid (84035) as a result of ORA-4031
2018-02-22T20:59:44.126307+08:00
opidrv aborting process CL57 ospid (84060) as a result of ORA-4031
2018-02-22T20:59:44.142147+08:00
opidrv aborting process CL58 ospid (84062) as a result of ORA-4031
2018-02-22T20:59:44.226535+08:00
opidrv aborting process CL63 ospid (84072) as a result of ORA-4031
2018-02-22T20:59:44.525697+08:00
opidrv aborting process CL81 ospid (84130) as a result of ORA-4031
2018-02-22T20:59:44.547475+08:00
opidrv aborting process CL82 ospid (84132) as a result of ORA-4031
2018-02-22T20:59:44.561992+08:00
opidrv aborting process CL83 ospid (84134) as a result of ORA-4031
2018-02-22T20:59:44.778414+08:00
opidrv aborting process CL96 ospid (84160) as a result of ORA-4031
2018-02-22T20:59:44.794834+08:00
opidrv aborting process CL97 ospid (84162) as a result of ORA-4031
2018-02-22T20:59:44.860335+08:00
opidrv aborting process CL01 ospid (84173) as a result of ORA-4031
2018-02-22T20:59:44.961698+08:00
opidrv aborting process CL07 ospid (84197) as a result of ORA-4031
2018-02-22T20:59:45.043893+08:00
opidrv aborting process CL12 ospid (84207) as a result of ORA-4031
2018-02-22T20:59:45.666646+08:00
opidrv aborting process CL48 ospid (84293) as a result of ORA-4031
2018-02-22T20:59:48.388508+08:00
Instance Critical Process (pid: 183, ospid: 84072, CL63) died unexpectedly
PMON (ospid: 56733): terminating the instance due to error 12752
2018-02-22T20:59:48.509139+08:00
System state dump requested by (instance=1, osid=56733 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/masaike/masaike1/trace/masaike1_diag_56811_20180222205948.trc
2018-02-22T20:59:49.128835+08:00
ORA-1092 : opitsk aborting process
2018-02-22T20:59:50.319126+08:00
License high water mark = 4496
2018-02-22T20:59:54.771874+08:00
Instance terminated by PMON, pid = 56733
2018-02-22T20:59:54.773210+08:00
Warning: 2 processes are still attach to shmid 1409038:
(size: 143360 bytes, creator pid: 53461, last attach/detach pid: 56813)
2018-02-22T20:59:55.333231+08:00
USER (ospid: 85323): terminating the instance
2018-02-22T20:59:55.338825+08:00
Instance terminated by USER, pid = 85323

直接原因是由于CL63进程died导致实例重启，CLnn代表Cleanup Helper Processes，被包含在PMON group中。这个特性是在12.2中引入。
具体可以参考https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cncpt/process-architecture.html#GUID-B5CA9579-53DB-442C-A85F-F21FD334833A下面这个部分15.4.1.1 Process Monitor Process (PMON) Group
15.4.1.1.3 Cleanup Helper Processes (CLnn)
CLMN delegates cleanup work to the CLnn helper processes.
The CLnn processes assist in the cleanup of terminated processes and sessions. The number of helper processes is proportional to the amount of cleanup work to be done and the current efficiency of cleanup.

A cleanup process can become blocked, which prevents it from proceeding to clean up other processes. Also, if multiple processes require cleanup, then cleanup time can be significant. For these reasons, Oracle Database can use multiple helper processes in parallel to perform cleanup, thus alleviating slow performance.

2. 重启前的4031 trace

- alert
2018-02-22T20:44:00.552799+08:00
Errors in file /u01/app/oracle/diag/rdbms/masaike/masaike1/trace/masaike1_ora_28848.trc (incident=5415624):
ORA-04031: unable to allocate 728 bytes of shared memory ("shared pool","unknown object","KGLH0^9d1660bd","kkscs")
2018-02-22T20:44:00.553363+08:00
Errors in file /u01/app/oracle/diag/rdbms/masaike/masaike1/trace/masaike1_ora_21328.trc (incident=5412384):
ORA-04031: unable to allocate 728 bytes of shared memory ("shared pool","DATABASE","KGLH0^ee7e02e2","kkscs")
ORA-04031: unable to allocate 40 bytes of shared memory ("shared pool","unknown object","KGLH0^9d1660bd","kglHeapInitialize:temp")
ORA-04031: unable to allocate 728 bytes of shared memory ("shared pool","unknown object","KGLH0^9d1660bd","kkscs")
Incident details in: /u01/app/oracle/diag/rdbms/masaike/masaike1/incident/incdir_5415624/masaike1_ora_28848_i5415624.trc
Incident details in: /u01/app/oracle/diag/rdbms/masaike/masaike1/incident/incdir_5412384/masaike1_ora_21328_i5412384.trc

- masaike1_ora_28848_i5415624.trc
*** 2018-02-22T20:44:00.554936+08:00
*** SESSION ID:(23158.41458) 2018-02-22T20:44:00.554950+08:00
*** CLIENT ID:() 2018-02-22T20:44:00.554957+08:00
*** SERVICE NAME:(masaike) 2018-02-22T20:44:00.554963+08:00
*** MODULE NAME:(oracle@masaike) 2018-02-22T20:44:00.554971+08:00
*** ACTION NAME:() 2018-02-22T20:44:00.554978+08:00
*** CLIENT DRIVER:() 2018-02-22T20:44:00.554989+08:00
==============================================
TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 1
----------------------------------------------
"gc index split transactio " 1652 MB 23%
"gcs resources " 731 MB 10%
"free memory " 451 MB 6%
"gcs shadows " 450 MB 6%
"ges resource permanent " 416 MB 6%
"ges resource dynamic " 345 MB 5%
"ges enqueues " 332 MB 5%
"db_block_hash_buckets " 236 MB 3%
"gc read wait freelist " 187 MB 3%
"Checkpoint queue " 168 MB 2%
-----------------------------------------
free memory 451 MB
memory alloc. 6653 MB
Sub total 7104 MB
==============================================
TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 1
----------------------------------------------
"free memory " 2394 MB
"ges resource dynamic " 2005 MB
"gc index split transactio " 1652 MB
"KGLH0 " 1063 MB
"gcs resources " 731 MB
"SQLA " 704 MB
"gcs shadows " 450 MB
"SQLP " 449 MB
"ges resource permanent " 416 MB
"ges enqueues " 332 MB
==============================================
TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 2
----------------------------------------------
"gc index split transactio " 1678 MB 25%
"gcs resources " 731 MB 11%
"gcs shadows " 450 MB 7%
"free memory " 432 MB 6%
"ges resource permanent " 416 MB 6%
"ges resource dynamic " 345 MB 5%
"ges enqueues " 337 MB 5%
"db_block_hash_buckets " 237 MB 4%
"gc read wait freelist " 187 MB 3%
"Checkpoint queue " 168 MB 3%
-----------------------------------------
free memory 432 MB
memory alloc. 6288 MB
Sub total 6720 MB
==============================================
TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 2
----------------------------------------------
"free memory " 2173 MB
"ges resource dynamic " 2004 MB
"gc index split transactio " 1678 MB
"KGLH0 " 1076 MB
"gcs resources " 731 MB
"SQLA " 731 MB
"gcs shadows " 450 MB
"SQLP " 445 MB
"ges resource permanent " 416 MB
"ges enqueues " 337 MB
==============================================
TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 3
----------------------------------------------
"gc index split transactio " 1653 MB 25%
"gcs resources " 731 MB 11%
"gcs shadows " 450 MB 7%
"free memory " 436 MB 7%
"ges resource permanent " 416 MB 6%
"ges resource dynamic " 345 MB 5%
"ges enqueues " 338 MB 5%
"db_block_hash_buckets " 238 MB 4%
"Checkpoint queue " 171 MB 3%
"event statistics per sess " 153 MB 2%
-----------------------------------------
free memory 436 MB
memory alloc. 6220 MB
Sub total 6656 MB
==============================================
TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 3
----------------------------------------------
"free memory " 2073 MB
"ges resource dynamic " 2004 MB
"gc index split transactio " 1653 MB
"KGLH0 " 1084 MB
"SQLA " 758 MB
"gcs resources " 731 MB
"SQLP " 450 MB
"gcs shadows " 450 MB
"ges resource permanent " 416 MB
"ges enqueues " 338 MB
TOTALS ---------------------------------------
Total free memory 1319 MB
Total memory alloc. 19 GB
Grand total 20 GB
==============================================

==============================================
TOP 20 MEMORY USES ACROSS SGA HEAP 1 - 3
----------------------------------------------
"gc index split transactio " 4983 MB 24%
"gcs resources " 2194 MB 11%
"gcs shadows " 1350 MB 7%
"free memory " 1319 MB 6%
"ges resource permanent " 1249 MB 6%
"ges resource dynamic " 1035 MB 5%
"ges enqueues " 1006 MB 5%
"db_block_hash_buckets " 712 MB 3%
"Checkpoint queue " 508 MB 2%
"gc read wait freelist " 467 MB 2%
"event statistics per sess " 458 MB 2%
"gc name table " 320 MB 2%
"gcs resv res hash bucket " 320 MB 2%
"dbktb: trace buffer " 313 MB 2%
"ksunfy : SSO free list " 277 MB 1%
"Multiblock Index SO " 235 MB 1%
"object queue hash buckets " 208 MB 1%
"kglsim object batch " 178 MB 1%
"KKSSP " 156 MB 1%
"ges resource hash table " 150 MB 1%
TOTALS ---------------------------------------
Total free memory 1319 MB
Total memory alloc. 19 GB
Grand total 20 GB
==============================================
==============================================
TOP 20 MAXIMUM MEMORY USES ACROSS SGA HEAP 1 - 3
----------------------------------------------
"free memory " 13 GB
"ges resource dynamic " 12 GB
"gc index split transactio " 10 GB
"KGLH0 " 6447 MB
"gcs resources " 4387 MB
"SQLA " 4387 MB
"gcs shadows " 2700 MB
"SQLP " 2688 MB
"ges resource permanent " 2497 MB
"ges enqueues " 2012 MB
"KQR X PO " 1789 MB
"KGLHD " 1642 MB
"db_block_hash_buckets " 1424 MB
"Checkpoint queue " 1016 MB
"gc read wait freelist " 934 MB
"event statistics per sess " 915 MB
"ges big msg " 858 MB
"KGLDA " 730 MB
"gc name table " 640 MB
"gcs resv res hash bucket " 640 MB

可以发现3个SUB POOL中gc index split transactio的compent的当前大小都有些异常，和没有发生4031的实例2对比，不在一个量级。

3. 重启时间前后的AWR对比，因为21:00左右重启，所以没有20-22的数据

19:00 - 20:00

Pool Name Begin MB End MB % Diff
shared Checkpoint queue 507.86 507.86 0.00
shared Multiblock Index SO 235.27 235.27 0.00
shared db_block_hash_buckets 712.00 712.00 0.00
shared dbktb: trace buffer 312.50 312.50 0.00
shared event statistics per sess 457.70 457.70 0.00
shared free memory 1,315.90 1,317.82 0.15
shared gc index split transactio 4,975.39 4,978.76 0.07
shared gc name table 320.00 320.00 0.00
shared gc read wait freelist 466.92 466.92 0.00
shared gcs resources 2,193.61 2,193.61 0.00
shared gcs resv res hash bucket 320.00 320.00 0.00
shared gcs shadows 1,349.92 1,349.92 0.00
shared ges enqueues 1,005.92 1,005.92 0.00
shared ges resource dynamic 1,035.37 1,035.37 0.00
shared ges resource permanent 1,248.53 1,248.53 0.00
shared ksunfy : SSO free list 276.78 276.78 0.00
shared object queue hash buckets 208.00 208.00 0.00
shared row cache 0.02 0.02 0.00

22:00 - 23:00

Pool Name Begin MB End MB % Diff
shared Checkpoint queue 507.86 507.86 0.00
shared KGLH0 368.86 679.57 84.23
shared Multiblock Index SO 235.27 235.27 0.00
shared SQLA 536.84 1,009.83 88.11
shared db_block_hash_buckets 712.00 712.00 0.00
shared dbktb: trace buffer 312.50 312.50 0.00
shared event statistics per sess 457.70 457.70 0.00
shared free memory 7,680.55 6,747.76 -12.14
shared gc name table 320.00 320.00 0.00
shared gc read wait freelist 466.92 466.92 0.00
shared gcs resources 2,193.61 2,193.61 0.00
shared gcs resv res hash bucket 320.00 320.00 0.00
shared gcs shadows 1,349.92 1,349.92 0.00
shared ges enqueues 630.18 630.18 0.00
shared ges resource dynamic 209.68 209.68 0.00

对比来看，重启之后gc index split transaction没有统计在内。并且20:00的End value为4,978.76MB，而前面分析的大约在20:44左右的4031 trace中gc index split transaction组件在4983 MB，说明一直在缓慢增长。

4. 和节点2 21:00 - 22:00的进行对比

Pool Name Begin MB End MB % Diff
shared Checkpoint queue 507.86 507.86 0.00
shared KGLH0 765.41
shared KKSSP 312.16 213.46 -31.62
shared Multiblock Index SO 235.27 235.27 0.00
shared SQLA 318.24 920.27 189.17
shared db_block_hash_buckets 712.00 712.00 0.00
shared dbktb: trace buffer 312.50 312.50 0.00
shared event statistics per sess 457.70 457.70 0.00
shared free memory 3,406.10 2,259.91 -33.65
shared gc name table 320.00 320.00 0.00
shared gc read wait freelist 466.92 466.92 0.00
shared gcs resources 2,193.61 2,193.61 0.00
shared gcs resv res hash bucket 320.00 320.00 0.00
shared gcs shadows 1,349.92 1,349.92 0.00
shared ges enqueues 1,438.79 1,438.79 0.00
shared ges resource dynamic 2,868.59 2,868.59 0.00
shared ges resource permanent 1,248.53 1,248.53 0.00
shared ksunfy : SSO free list 276.78 276.78 0.00
shared object queue hash buckets 208.00 208.00 0.00
shared parameter table block 256.43 -100.00
shared row cache 0.02 0.02 0.00

节点2的ges resource dynamic在2868MB，同样不低，但是gc index split transaction这个组件不在统计内

5. 那么碎片的原因可能是ges resource dynamic还是gc index split transaction造成?

根据如下trace进行分析

2017-12-20T00:14:03.146906+08:00
Errors in file /u01/app/oracle/diag/rdbms/masaike/masaike1/trace/masaike1_p01g_66601.trc (incident=5410528):
ORA-04031: unable to allocate 13840 bytes of shared memory ("shared pool","unknown object","sga heap(3,0)","ges resource dynamic")
Incident details in: /u01/app/oracle/diag/rdbms/masaike/masaike1/incident/incdir_5410528/masaike1_p01g_66601_i5410528.trc

Dump file /u01/app/oracle/diag/rdbms/masaike/masaike1/incident/incdir_5410528/masaike1_p01g_66601_i5410528.trc

=============================================
TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 1
----------------------------------------------
"ges resource dynamic " 2005 MB 28%
"gcs resources " 731 MB 10%
"free memory " 601 MB 8%
"gcs shadows " 450 MB 6%
"ges resource permanent " 416 MB 6%
"ges enqueues " 283 MB 4%
"db_block_hash_buckets " 236 MB 3%
"gc read wait freelist " 187 MB 3%
"Checkpoint queue " 168 MB 2%
"event statistics per sess " 153 MB 2%
-----------------------------------------
free memory 601 MB
memory alloc. 6503 MB
Sub total 7104 MB
==============================================
TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 1
----------------------------------------------
"free memory " 2394 MB
"ges resource dynamic " 2005 MB
"KGLH0 " 1063 MB
"gcs resources " 731 MB
"SQLA " 527 MB
"gcs shadows " 450 MB
"SQLP " 449 MB
"ges resource permanent " 416 MB
"ges enqueues " 283 MB
"KGLHD " 272 MB
==============================================
TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 2
----------------------------------------------
"ges resource dynamic " 2004 MB 30%
"gcs resources " 731 MB 11%
"free memory " 646 MB 10%
"gcs shadows " 450 MB 7%
"ges resource permanent " 416 MB 6%
"ges enqueues " 288 MB 4%
"db_block_hash_buckets " 237 MB 4%
"gc read wait freelist " 187 MB 3%
"Checkpoint queue " 168 MB 3%
"event statistics per sess " 153 MB 2%
-----------------------------------------
free memory 646 MB
memory alloc. 6074 MB
Sub total 6720 MB
==============================================
TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 2
----------------------------------------------
"free memory " 2173 MB
"ges resource dynamic " 2004 MB
"KGLH0 " 1076 MB
"gcs resources " 731 MB
"SQLA " 568 MB
"gcs shadows " 450 MB
"SQLP " 445 MB
"ges resource permanent " 416 MB
"KQR X PO " 330 MB
"ges enqueues " 288 MB
==============================================
TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 3
----------------------------------------------
"ges resource dynamic " 2004 MB 30%
"gcs resources " 731 MB 11%
"free memory " 718 MB 11%
"gcs shadows " 450 MB 7%
"ges resource permanent " 416 MB 6%
"ges enqueues " 288 MB 4%
"db_block_hash_buckets " 238 MB 4%
"Checkpoint queue " 171 MB 3%
"event statistics per sess " 153 MB 2%
"gcs resv res hash bucket " 107 MB 2%
-----------------------------------------
free memory 718 MB
memory alloc. 5938 MB
Sub total 6656 MB
==============================================
TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 3
----------------------------------------------
"ges resource dynamic " 2004 MB
"free memory " 1430 MB
"KGLH0 " 1084 MB
"gcs resources " 731 MB
"SQLA " 606 MB
"SQLP " 450 MB
"gcs shadows " 450 MB
"ges resource permanent " 416 MB
"KQR X PO " 321 MB
"ges enqueues " 288 MB
TOTALS ---------------------------------------
Total free memory 1964 MB
Total memory alloc. 18 GB
Grand total 20 GB
==============================================
==============================================
TOP 20 MEMORY USES ACROSS SGA HEAP 1 - 3
----------------------------------------------
"ges resource dynamic " 6013 MB 29%
"gcs resources " 2194 MB 11%
"free memory " 1964 MB 10%
"gcs shadows " 1350 MB 7%
"ges resource permanent " 1249 MB 6%
"ges enqueues " 859 MB 4%
"db_block_hash_buckets " 712 MB 3%
"Checkpoint queue " 508 MB 2%
"gc read wait freelist " 467 MB 2%
"event statistics per sess " 458 MB 2%
"gc name table " 320 MB 2%
"gcs resv res hash bucket " 320 MB 2%
"dbktb: trace buffer " 313 MB 2%
"ksunfy : SSO free list " 277 MB 1%
"Multiblock Index SO " 235 MB 1%
"object queue hash buckets " 208 MB 1%
"ges resource hash table " 150 MB 1%
"object queue header free " 150 MB 1%
"keomg: entry list " 144 MB 1%
"kghrcx batch " 131 MB 1%
TOTALS ---------------------------------------
Total free memory 1964 MB
Total memory alloc. 18 GB
Grand total 20 GB
==============================================
==============================================
TOP 20 MAXIMUM MEMORY USES ACROSS SGA HEAP 1 - 3
----------------------------------------------
"ges resource dynamic " 12 GB
"free memory " 12 GB
"KGLH0 " 6447 MB
"gcs resources " 4387 MB
"SQLA " 3400 MB
"gcs shadows " 2700 MB
"SQLP " 2688 MB
"ges resource permanent " 2497 MB
"KQR X PO " 1789 MB
"ges enqueues " 1719 MB
"KGLHD " 1642 MB
"db_block_hash_buckets " 1424 MB
"Checkpoint queue " 1016 MB
"gc read wait freelist " 934 MB
"event statistics per sess " 915 MB
"ges big msg " 858 MB
"KGLDA " 730 MB
"gc name table " 640 MB
"gcs resv res hash bucket " 640 MB
"dbktb: trace buffer " 625 MB
==============================================

可以发现在12-20日时，ges resource dynamic已经到达了6G，历史峰值上也到达过12G。但是gc index split transaction并没有出现在TOP中。
且该组件大小从历史上来看也被释放过，所以02-22日的主要原因还是在gc index split transaction组件的异常使用上。

6. gc index split transaction的作用

参考ORA-4031 Error Due to Increasing SGA Heap Chunks for 'gc index split transaction' (Doc ID 2339919.1)、Bug 27163928 - ORA-4031 error due to leakage of 'gc index split transaction' permanent memory chunks (Doc ID 27163928.8)

BUG中说明，gc index split transaction是在12cR2的新特性fast index split wait带来的内存组件。关于这个特性未找到明确的说明，BUG中有描述：

The feature is implemented in KCL layer for global cache of RAC and it's used for a process to wait for index splits on remote or local.
Basically a array buffer is allocated for a session while it's initialized on connection.

说明这个特性和RAC环境下的索引分裂（本地或者远程）以及user logon有关，并且从BUG中研发的重现来看，通过set _gc_fast_index_split_wait=0并不能解决这个问题

# inst 1 19:00 - 20:00
user logons cumulative 23,142 6.45 0.05

# inst 2 21:00 - 22:00
user logons cumulative 3,849 1.07 0.01

从awr中user logons cumulative的统计数据来看，实例1的logon也远高于实例2（只取有负载的两个实例，通过ASHDUMP观察active session主要集中在实例1和2），从这方面来看和BUG是匹配的

7. 如何解决gc index split transaction组件增长过高的问题

参考ORA-4031 Error Due to Increasing SGA Heap Chunks for 'gc index split transaction' (Doc ID 2339919.1)，这部分也是perm类型的组件，因此无法通过flush shared pool来释放，

1）应用one-off patch，需要OPatch信息来确认是否存在可用patch。
2）客户端程序使用率连接池，避免出现大量的短连接登陆操作。
3）从应用角度，可以从跨节点并发Insert造成索引分裂的角度来减缓这个组件的增长。

Segments by Global Cache Buffer Busy

Owner Tablespace Name Object Name Subobject Name Obj. Type Obj# Dataobj# GC Buffer Busy % of Capture
SYS SYSTEM USER$ TABLE 22 10 562 82.40
MASAIKE MASAIKE_DATA IDX_MASAIKE_SERIAL_NO INDEX 488853 488853 67 9.82
MASAIKE MASAIKE_DATA IDX_MASAIKE_VALID_TIME INDEX 488852 488852 43 6.30
MASAIKE ECHNWAP_DATA TDMASAIKE201802 TABLE 538927 538927 4 0.59
MASAIKE MASAIKE_IDX IDX_MASAIKE_201802 INDEX 540204 540204 2 0.29

Cluster Wait Time (s) Executions %Total Elapsed Time(s) %Clu %CPU %IO SQL Id SQL Module SQL Text
9.49 260,265 14.90 69.60 13.64 36.72 46.55 51r2pujd84cj7 JDBC Thin Client select a1_...
9.36 524,784 14.69 139.61 6.71 30.39 27.50 63udjnjwwdzh2 JDBC Thin Client insert into t01 (MSGI...
5.81 582,832 9.12 75.02 7.74 21.75 1.62 7khyz5337f312 JDBC Thin Client insert into t01 (MSG...
4.26 136,939 6.69 57.85 7.37 33.77 38.42 99ft2rzjkm40g JDBC Thin Client insert into t02 (UP_T...
3.67 15,173 5.76 8.59 42.69 47.86 0.02 g050ygw7q9qzc JDBC Thin Client insert into t03 (SERI...

8. 通过应用补丁27163928,gc index split transactio组件未在继续增长，4031问题得到解决。

9. 附录sql核查gc index split transactio组件增长情况

SQL>
SQL> Select to_char(dh.begin_interval_time, 'yyyy-mm-dd hh24:mi:ss') begin_interval_time,
2 dh.instance_number,
3 dhs.name,
4 Sum(dhs.bytes) / 1024 / 1024 mb
5 From dba_hist_snapshot dh, dba_hist_sgastat dhs
6 Where dh.snap_id = dhs.snap_id
7 And dh.instance_number = dhs.instance_number
8 And dh.con_id = dhs.con_id
9 And dhs.name = 'gc index split transactio' and dh.instance_number=1
10 Group By to_char(dh.begin_interval_time, 'yyyy-mm-dd hh24:mi:ss'),
11 dh.instance_number,
12 dhs.name
13 Order By to_char(dh.begin_interval_time, 'yyyy-mm-dd hh24:mi:ss'), dh.instance_number;
BEGIN_INTERVAL_TIME INSTANCE_NUMBER NAME MB
------------------- --------------- ---------------------------------------------------------------- ----------
2018-04-02 08:00:19 1 gc index split transactio 3138.89177
2018-04-02 09:00:04 1 gc index split transactio 3142.47413
2018-04-02 10:00:05 1 gc index split transactio 3145.96922
2018-04-02 11:00:16 1 gc index split transactio 3149.31023
2018-04-02 12:00:03 1 gc index split transactio 3152.62398
2018-04-02 13:00:15 1 gc index split transactio 3156.11864
......
2018-04-23 10:00:11 1 gc index split transactio 4785.7306
2018-04-23 11:00:22 1 gc index split transactio 4789.03942
2018-04-23 12:00:33 1 gc index split transactio 4792.28026
2018-04-23 13:00:04 1 gc index split transactio 4795.5872
2018-04-23 14:00:04 1 gc index split transactio 4798.89805
2018-04-23 15:00:06 1 gc index split transactio 4802.22398
2018-04-23 16:00:17 1 gc index split transactio 4805.54382
2018-04-23 17:00:28 1 gc index split transactio 4808.76595
2018-04-23 18:00:00 1 gc index split transactio 4812.09579
2018-04-23 19:00:04 1 gc index split transactio 4815.41288
2018-04-23 20:00:06 1 gc index split transactio 4818.72141

518 rows selected.

这里可以发现应用打补丁前gc index split transactio 组件一直在缓慢增长

SQL>
SQL> Select inst_id, Name, Sum(bytes) / 1024 / 1024 mb
2 From gv$sgastat
3 Where Name = 'gc index split transactio' and INST_ID=1
4 Group By inst_id, Name;

INST_ID NAME MB
---------- -------------------------- ----------
1 gc index split transactio 25.9837646

SQL>

应用补丁后，gc index split transactio组件稳定在25，未在增长。

Oracle v12.2 Bug 27163928触发ORA-4031 导致实例crash.相关推荐

Oracle ora-15070,查询字段过多触发了Oracle的BUG？【ORA-01465: 无效的十六进制数字】...
当我使用以下INSERT INTO SELECT语法操作数据库时,报ORA-01465的错误,然后当我削减插入字段和查询字段,该错误消失了. 所以我怀疑这是Oracle的一个bug. 如果有比较了解O ...
Oracle 12c因bug导致ORA-04031问题处理过程 | 云和恩墨技术通讯精选
各位亲爱的用户/读者朋友们: 为了及时共享行业案例,通告共性问题,达成知识共享和提前预防,我们整理和编辑了<云和恩墨技术通讯>(8月刊),通过对过去一段时间的知识回顾和故障归纳,以期提供有 ...
Oracle 11g 的bug？： aix 上，expdp 11.2.0.1 导出，impdp 11.2.0.3 导入，Interval 分区的【Interval】分区属性成了【N】...
如题: Oracle 11g 的bug?: aix 上,expdp 11.2.0.1 导出,impdp 11.2.0.3 导入,Interval 分区的 [Interval] 分区属性成了[N] 谨记 ...
oracle批量插入报错,[数据库]Mybatis 批量插入数据关于Oracle 批量插入报错：ORA
[数据库]Mybatis 批量插入数据关于Oracle 批量插入报错:ORA 0 2020-08-19 08:00:06 Mybatis 批量插入数据关于Oracle 批量插入报错:ORA-009 ...
oracle触发器触发事件,oracle 触发器的种类和触发事件
oracle触发器的种类和触发事件,DML触发器,DDL事件触发器,替代触发器,查看触发器触发器的种类和触发事件触发器必须由事件才能触发.触发器的触发事件分可为3类,分别是DML事件.DDL事件和 ...
oracle11g ora 29927,【案例】Oracle内存泄漏进行10046跟踪分析07445导致数据库宕机
天萃荷净在一次ORA-7445导致oracle数据库down掉故障分析中,发现sql因某种原因导致大量的sql area中很多内存泄露,最终导致数据库down掉.通过实验找出类此奇怪SQL. SEL ...
oracle实例的概念组成,oracle体系结构的两个基本概念:数据库和实例
您可能感兴趣的话题: oracle 核心提示:要了解oracle体系结构必须先了解两个基本的概念: 数据库和实例. 要了解oracle体系结构必须先了解两个基本的概念: 数据库和实例. 一: 数据库 ...
oracle报609,案例:Oracle报错ORA-609 TNS-12537 TNS-12547 连接超时导致
天萃荷净生产数据库alert日志中出现错误信息ORA-609 TNS-12537 TNS-12547,分析原因为数据库inbound connect timeout默认为60导致 1.数据库aler ...
Oracle公有云上的ADG配置（单实例）
概述本文描述在OCI上搭建的标准ADG上的配置,以促进对概念的理解. 此ADG配置中,主备数据库均为单实例.主机名分别为db01和db02.数据库名为CDB.主库的DB_UNIQUE_NAME为ch ...

Oracle v12.2 Bug 27163928触发ORA-4031 导致实例crash.

Oracle v12.2 Bug 27163928触发ORA-4031 导致实例crash.相关推荐

最新文章

热门文章