【案例】Oracle服务器diag进程占据了12g的磁盘空间分析解决办法

时间:2016-11-13 20:10   来源:Oracle研究中心   作者:网络   点击:

次

天萃荷净

Oracle研究中心案例分析：巡检时发现某套rac的awr居然没有生成，而且手工创建快照也没反应。发现某个节点的/oracle 一共20g，oracle本身才8G左右，另外的12g不知道被什么东西占据了，ls -ltr居然无法看见，经主机工程师检查，发现是diag进程占据了12g的磁盘空间。

在操作之前，我做了一个systemstate dump，对该dump我简单的分析了一下，发现了如下有用信息：

System global information:

processes: base c0000000ad649960, size 500, cleanup c0000000ad68c020

allocation: free sessions c0000000ad78d620, free calls 0000000000000000

control alloc errors: 0 (process), 0 (session), 0 (call)

PMON latch cleanup depth: 0

seconds since PMON's last scan for dead processes: 19  ###### 有19个死进程

###### 我们首先来搜索diag进程：######

PROCESS 3:

—————————————-

SO: c0000000ad68c810, type: 2, owner: 0000000000000000, flag: INIT/-/-/0x00

(process) Oracle pid=3, calls cur/top: c0000000adab8358/c0000000adab8358, flag: (2) SYSTEM

int error: 0, call error: 0, sess error: 0, txn error 0

(post info) last post received: 0 0 24

last post received-location: ksasnd

last process to post me: c0000000ad6956f0 235 0

last post sent: 0 0 48

last post sent-location: ksoreq_reply

last process posted by me: c0000000ad782910 1 0

(latch info) wait_event=0 bits=0

Process Group: DEFAULT, pseudo proc: c0000000ad784f20

O/S info: user: oracle, term: UNKNOWN, ospid: 27628

OSD pid info: Unix process pid: 27628, image: oracle@uamdb2 (DIAG)

###### 然后搜索c0000000ad68c810 字符串，发现如下内容：######

SO: c0000000ab2bac00, type: 19, owner: c0000000ad68c810, flag: INIT/-/-/0x00

GES MSG BUFFERS: st=emp chunk=0x0000000000000000 hdr=0x0000000000000000 lnk=0x0000000000000000 flags=0x0 inc=0

outq=0 sndq=0 opid=0 prmb=0x0

mbg[i]=(0 0) mbg[b]=(0 0) mbg[r]=(0 0)

fmq[i]=(0 0) fmq[b]=(0 0) fmq[r]=(0 0)

mop[s]=0 mop[q]=0 pendq=0 zmbq=0

nonksxp_recvs=0

————process 0xc0000000ab2bac00——————–

proc version      : 0

Local node        : 1

pid               : 27628

lkp_node          : 1

svr_mode          : 0

proc state        : KJP_UNFREEZE

Last drm hb acked : 0

Total accesses    : 5

Imm.  accesses    : 0

Locks on ASTQ     : 0

Locks Pending AST : 0

Granted locks     : 0

AST_Q:

PENDING_Q:

GRANTED_Q:

—————————————-

SO: c0000000ada72528, type: 4, owner: c0000000ad68c810, flag: INIT/-/-/0x00

(session) sid: 554 trans: 0000000000000000, creator: c0000000ad68c810, flag: (51) USR/- BSY/-/-/-/-/-

DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000

txn branch: 0000000000000000

oct: 0, prv: 0, sql: 0000000000000000, psql: 0000000000000000, user: 0/SYS

service name: SYS$BACKGROUND

waiting for 'DIAG idle wait' blocking sess=0x0000000000000000 seq=331 wait_time=0 seconds since wait started=62193

component=1, where=1, wait time(millisec)=c8

Dumping Session Wait History

for 'DIAG idle wait' count=1 wait_time=195274

component=1, where=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=195286

component=1, where=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=214760

component=1, where=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=195267

component=1, where=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=195282

component=1, wherhttp://www.oracleplus.nete=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=195282

component=1, where=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=195281

component=1, where=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=195278

component=1, where=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=195280

component=1, where=1, wait time(millisec)=c8

for 'DIAG idle wait' count=1 wait_time=195270

component=1, where=1, wait time(millisec)=c8

temporary object counter: 0

—————————————-

UOL used : 0 locks(used=0, free=0)

KGX Atomic Operation Log c0000000acb7a5a0

Mutex 0000000000000000(0, 0) idn 0 oper NONE

Library Cache uid 554 efd 0 whr 0 slp 0

KGX Atomic Operation Log c0000000acb7a5e8

Mutex 0000000000000000(0, 0) idn 0 oper NONE

Library Cache uid 554 efd 0 whr 0 slp 0

KGX Atomic Operation Log c0000000acb7a630

Mutex 0000000000000000(0, 0) idn 0 oper NONE

Library Cache uid 554 efd 0 whr 0 slp 0

—————————————-

SO: c0000000a1a7fa18, type: 41, owner: c0000000ada72528, flag: INIT/-/-/0x00

(dummy) nxc=0, nlb=0

—————————————-

SO: c0000000adab8358, type: 3, owner: c0000000ad68c810, flag: INIT/-/-/0x00

(call) sess: cur c0000000ada72528, rec 0, usr c0000000ada72528; depth: 0

—————————————-

SO: c0000000ab2adec8, type: 16, owner: c0000000ad68c810, flag: INIT/-/-/0x00

(osp req holder)

###### 然后来看mmon进程：######

SO: c0000000add59f60, type: 11, owner: c0000000ad692f40, flag: INIT/-/-/0x00

(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad692f40,

event: 1, last message event: 1381841,

last message waited event: 1381841, messages read: 1376494

channel: (c0000000add81260) MMON remote action broadcast channel

scope: 27, event: 1381841, last mesage event: 1381841,

publishers/subscribers: 2/2,

messages published: 1376494

—————————————-

SO: c0000000ada60ee0, type: 4, owner: c0000000ad692f40, flag: INIT/-/-/0x00

(session) sid: 541 trans: 0000000000000000, creator: c0000000ad692f40, flag: (51) USR/- BSY/-/-/-/-/-

DID: 0002-0010-00000003, short-term DID: 0002-0010-00000004

txn branch: 0000000000000000

oct: 0, prv: 0, sql: c0000000ac7f3760, psql: 0000000000000000, user: 0/SYS

service name: SYS$BACKGROUND

waiting for 'rdbms ipc message' blocking sess=0x0000000000000000 seq=49622 wait_time=0 seconds since wait started=2

timeout=12c, =0, =0

Dumping Session Wait History

for 'rdbms ipc message' count=1 wait_time=1503849

timeout=99, =0, =0

for 'rdbms ipc message' count=1 wait_time=1435494

timeout=12c, =0, =0

for 'rdbms ipc message' count=1 wait_time=2939422

timeout=12c, =0, =0

for 'rdbms ipc message' count=1 wait_time=517538

timeout=34, =0, =0

for 'rdbms ipc message' count=1 wait_time=2421822

timeout=12c, =0, =0

for 'rdbms ipc message' count=1 wait_time=2470581

timeout=fc, =0, =0

for 'rdbms ipc message' count=1 wait_time=1

timeout=fc, =0, =0

for 'rdbms ipc message' count=1 wait_time=468661

timeout=12c, =0, =0

for 'rdbms ipc message' count=1 wait_time=2933950

timeout=12c, =0, =0

for 'os thread startup' count=1 wait_time=41005

=0, =0, =0

另外看了几个其他进程的信息，发现都是在等待 rdbms ipc message。

记得当时还做了一个oradebug hanganalyze 3，发现lck0进程居然阻塞了451个objects。

由于当时的产生的trace我忘记取下来了, 所以就只能用这个trace中去寻找信息了。

###### 搜索lck字符串：######

PROCESS 20:

—————————————-

SO: c0000000ad694f00, type: 2, owner: 0000000000000000, flag: INIT/-/-/0x00

(process) Oracle pid=20, calls cur/top: c0000000adabb810/c0000000adabb810, flag: (6) SYSTEM

int error: 0, call error: 0, sess error: 0, txn error 0

(post info) last post received: 0 0 33

last post received-location: ksrpublish

last process to post me: c0000000ad694f00 1 6

last post sent: 0 0 33

last post sent-location: ksrpublish

last process posted by me: c0000000ad694f00 1 6

(latch info) wait_event=0 bits=0

Process Group: DEFAULT, pseudo proc: c0000000ad784f20

O/S info: user: oracle, term: UNKNOWN, ospid: 27684

OSD pid info: Unix process pid: 27684, image: oracle@uamdb2 (LCK0)

###### 然后搜索字符串c0000000ad694f00 ######

SO: c0000000adbb9fd0, type: 5, owner: c0000000ad694f00, flag: INIT/-/-/0x00

(enqueue) XR-00000000-00000000 DID: 0000-0014-00000003

lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x3

res: 0xc0000000adcbf510, mode: NULL, lock_flag: 0x0

own: 0xc0000000ada5b940, sess: 0xc0000000ada5b940, proc: 0xc0000000ad694f00, prv: 0xc0000000adcbf520

slk: 0xc0000000ab40db98

这段内容是enqueue state object信息，发现该进程持有一个enqueue。

从这里的 res_flag: 0x3 可以判断：这个enqueue是global类型的，而且正在被持有，还没被free。

继续往下搜索，发现了如下类似的大量信息：

SO: c0000000add5e1e8, type: 11, owner: c0000000ad694f00, flag: INIT/-/-/0x00

(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad694f00,

event: 3, last message event: 3,

last message waited event: 3, messages read: 0

channel: (c0000000add81108) kea interrupt signal channel

scope: 5, event: 3, last mesage event: 0,

publishers/subscribers: 1/2,

messages published: 0

—————————————-

SO: c0000000add5e0d8, type: 11, owner: c0000000ad694f00, flag: INIT/-/-/0x00

(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad694f00,

event: 3, last message event: 3326313,

last message waited event: 3326313, messages read: 3324562

channel: (c0000000add7ee18) kxfp control signal channel

scope: 5, event: 3326313, last mesage event: 3326313,

publishers/subscribers: 4/2,

messages published: 3324562

这里的type值为11，代表的意思是：Broadcast Handle(即广播消息句柄)。难怪做oradebug的时候，发现这个lck进程阻塞了451个objects，而且等待居然都是rdbms ipc message。

继续往下搜索，发现了如下关键信息：

—————————————-

SO: c0000000ada5b940, type: 4, owner: c0000000ad694f00, flag: INIT/-/-/0x00

(session) sid: 537 trans: 0000000000000000, creator: c0000000ad694f00, flag: (51) USR/- BSY/-/-/-/-/-

DID: 0000-0014-00000003, short-term DID: 0000-0000-00000000

txn branch: 0000000000000000

oct: 0, prv: 0, sql: 0000000000000000, psql: 0000000000000000, user: 0/SYS

service name: SYS$BACKGROUND

waiting for 'rdbms ipc message' blocking sess=0x0000000000000000 seq=43421 wait_time=0 seconds since wait started=1

timeout=12c, =0, =0

Dumping Session Wait History

for 'ksxr poll remote instances' count=1 wait_time=7

=0, =0, =0

for 'ksxr poll remote instances' count=1 wait_time=7

=0, =0, =0

for 'rdbms ipc message' count=1 wait_time=1906819

timeout=c2, =0, =0

for 'ksxr poll remote instances' count=1 wait_time=9

=0, =0, =0

for 'rdbms ipc message' count=1 wait_time=12

timeout=c2, =0, =0

for 'rdbms ipc message' count=1 wait_time=1041870

timeout=12c, =0, =0

for 'ksxr poll remote instances' count=1 wait_time=7

=0, =0, =0

for 'ksxr poll remote instances' count=1 wait_time=10

=0, =0, =0

for 'rdbms ipc message' count=1 wait_time=807692

timeout=52, =0, =0

for 'rdbms ipc message' count=1 wait_time=27

timeout=52, =0, =0

temporary object counter: 0

—————————————-

UOL used : 0 locks(used=0, free=0)

KGX Atomic Operation Log c0000000aca2e6e8

Mutex 0000000000000000(0, 0) idn 0 oper NONE

Cursor Pin uid 537 efd 3 whr 11 slp 0

KGX Atomic Operation Log c0000000aca2e730

Mutex 0000000000000000(0, 0) idn 0 oper NONE

Library Cache uid 537 efd 0 whr 0 slp 0

KGX Atomic Operation Log c0000000aca2e778

Mutex 0000000000000000(0, 0) idn 0 oper NONE

Library Cache uid 537 efd 0 whr 0 slp 0

—————————————-

SO: c0000000a1a7fe78, type: 41, owner: c0000000ada5b940, flag: INIT/-/-/0x00

(dummy) nxc=0, nlb=0

—————————————-

SO: c0000000add5ab28, type: 11, owner: c0000000ad694f00, flag: INIT/-/-/0x00

(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad694f00,

event: 20, last message event: 26,

last message waited event: 26, messages read: 1

channel: (c0000000add7a2d8) system events broadcast channel

scope: 2, event: 204545, last mesage event: 26,

publishers/subscribers: 0/75,

messages published: 1

—————————————-

SO: c0000000ab2ba708, type: 19, owner: c0000000ad694f00, flag: INIT/-/-/0x00

GES MSG BUFFERS: st=emp chunk=0x0000000000000000 hdr=0x0000000000000000 lnk=0x0000000000000000 flags=0x0 inc=36

outq=0 sndq=0 opid=20 prmb=0x0

mbg[i]=(3277535 648561) mbg[b]=(0 0) mbg[r]=(0 0)

fmq[i]=(4 0) fmq[b]=(0 0) fmq[r]=(0 0)

mop[s]=648561 mop[q]=3277535 pendq=0 zmbq=0

nonksxp_recvs=0

————process 0xc0000000ab2ba708——————–

proc version      : 0

Local node        : 1

pid               : 27684

lkp_node          : 1

svr_mode          : 0

proc state        : KJP_NORMAL

Last drm hb acked : 0

Total accesses    : 221605743

Imm.  accesses    : 196696635

Locks on ASTQ     : 0

Locks Pending AST : 113

Granted locks     : 56081

AST_Q:

PENDING_Q:

lp 0xc0000000acb505c0 gl KJUSERNL rl KJUSERPR rp 0xc000000094eeda88 [0x2345][0xffffffff],[IV]

master 1 pid 27684 bast 0 rseq 1 mseq 0 history 0x9565a

convert opt KJUSERGETVALUE KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK

lp 0xc00000007a220300 gl KJUSERNL rl KJUSERPR rp 0xc000000094eeda88 [0x2345][0xffffffff],[IV]

master 1 pid 27684 bast 0 rseq 1 mseq 0 history 0x565a565a

convert opt KJUSERGETVALUE KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK

lp 0xc000000074ac4c38 gl KJUSERNL rl KJUSERPR rp 0xc000000094eeda88 [0x2345][0xffffffff],[IV]

master 1 pid 27684 bast 0 rseq 1 mseq 0 history 0x565a565a

convert opt KJUSERGETVALUE KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK

lp 0xc0000000748ca940 gl KJUSERNL rl KJUSERPR rp 0xc000000094eeda88 [0x2345][0xffffffff],[IV]

master 1 pid 27684 bast 0 rseq 1 mseq 0 history 0x565a565a

这里的rp 0xc000000094eeda88 我猜应该是enqueue等待的资源名，于是搜索resouce 0xc000000094eeda88字符串，发现了如下信息：

———-enqueue 0xc00000007a3cb2e0————————

lock version     : 1

Owner node       : 1

grant_level      : KJUSERNL

req_level        : KJUSERPR

bast_level       : KJUSERNL

notify_func      : 0x0000000000000000

resp             : 0xc000000094eeda88

procp            : 0xc0000000ab2ba708

pid              : 27684

proc version     : 0

oprocp           : 0x0000000000000000

opid             : 0

group lock owner : 0x0000000000000000

xid              : 0000-0000-00000000

dd_time          : 0.0 secs

dd_count         : 0

timeout          : 0.0 secs

On_timer_q       : N

On_dd_q          : N

lock_state       : CONVERTING

Open Options     :  KJUSERPROCESS_OWNED KJUSERCLIENTLOCK

Convert options  : KJUSERGETVALUE KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK

History          : 0x565a565a

Msg_Seq          : 0x0

res_seq          : 1

valblk           : 0x000000020001fcc00000000000000000 .

———-resource 0xc000000094eeda88———————-

resname       : [0x2345][0xffffffff],[IV]

Local node    : 1

dir_node      : 1

master_node   : 1

hv idx        : 35

hv last r.inc : 6

current inc   : 36

hv status     : 0

hv master     : 1

open options  :

grant_bits    : KJUSERNL KJUSERPR

grant mode    : KJUSERNL  KJUSERCR  KJUSERCW  KJUSERPR  KJUSERPW  KJUSEREX

count         : 114         0         0         333         0         0

val_state     : KJUSERVS_VALUE

valblk        : 0x000000020001fcc00000000000000000 .

access_node   : 1

vbreq_state   : 0

state         : x8

resp          : 0xc000000094eeda88

On Scan_q     : N

Total accesses: 4524902

Imm.  accesses: 1948489

Granted_locks : 333

Cvting_locks  : 114

总的来说，给人的感觉似乎lck是罪魁祸首，其实不竟然。

用awk脚本格式化了这个systemstate dump后，得到下面的结果：

System State 1

~~~~~~~~~~~~~~~~

1:

2:  waiting for 'pmon timer'            wait

3:  waiting for 'DIAG idle wait'        wait

4:  waiting for 'rdbms ipc message'     wait

5:  waiting for 'rdbms ipc message'     wait

6:  waiting for 'ges remote message'    wait

7:  waiting for 'gcs remote message'    wait

8:  waiting for 'gcs remote message'    wait

9:  waiting for 'rdbms ipc message'     wait

10: waiting for 'rdbms ipc message'     wait

11: waiting for 'rdbms ipc message'     wait

12: waiting for 'rdbms ipc message'     wait

13: waiting for 'smon timer'            wait

14: waiting for 'rdbms ipc message'     wait

15: waiting for 'rdbms ipc message'     wait

16: waiting for 'rdbms ipc message'     wait

17: waiting for 'rdbms ipc message'     wait

18:

19:

20: waiting for 'rdbms ipc message'     wait

21: waiting for 'SQL*Net message from client' wait

22: waiting for 'rdbms ipc message'     wait

23: waiting for 'rdbms ipc message'     wait

24: waiting for 'SQL*Net message from client' wait

25: waiting for 'Streams AQ: qmn coordinator idle wait' wait

26: waiting for 'SQL*Net message from client' wait

27: waiting for 'SQL*Net message from client' wait

Cmd: PL/SQL Execute

28: for 'Streams AQ: waiting for time management or cleanup tasks' wait

29:

30: waiting for 'SQL*Net message from client' wait

31: waiting for 'SQL*Net message from client' wait

32: waiting for 'SQL*Net message from client' wait

33: waiting for 'SQL*Net message from client' wait

34: waiting for 'SQL*Net message from client' wait

35: waiting for 'SQL*Net message from client' wait

36: waiting for 'SQL*Net message from client' wait

37: waiting for 'SQL*Net message from client' wait

38: waiting for 'SQL*Net message from client' wait

39: waiting for 'SQL*Net message from client' wait

Cmd: Insert

40: waiting for 'SQL*Net message from client' wait

41: waiting for 'SQL*Net message from client' wait

42: last wait for 'SQL*Net message from client'

43: waiting for 'SQL*Net message from client' wait

45: waiting for 'Streams AQ: qmn slave idle wait' wait

46: waiting for 'SQL*Net message from client' wait

Cmd: Insert

47: waiting for 'SQL*Net message from client' wait

48: waiting for 'SQL*Net message from client' wait

49: waiting for 'SQL*Net message from client' wait

50: waiting for 'SQL*Net message from client' wait

52: waiting for 'SQL*Net message from client' wait

53: waiting for 'SQL*Net message from client' wait

54: waiting for 'SQL*Net message from client' wait

55: waiting for 'SQL*Net message from client' wait

56: waiting for 'SQL*Net message from client' wait

58: waiting for 'SQL*Net message from client' wait

59: waiting for 'SQL*Net message from client' wait

60: waiting for 'SQL*Net message from client' wait

61: waiting for 'SQL*Net message from client' wait

65: waiting for 'SQL*Net message from client' wait

66: waiting for 'SQL*Net message from client' wait

90: waiting for 'SQL*Net message from client' wait

91: waiting for 'SQL*Net message from client' wait

92: waiting for 'SQL*Net message from client' wait

93: waiting for 'SQL*Net message from client' wait

94: waiting for 'SQL*Net message from client' wait

95: waiting for 'SQL*Net message from client' wait

96: waiting for 'SQL*Net message from client' wait

106:waiting for 'SQL*Net message from client' wait

107:waiting for 'SQL*Net message from client' wait

108:waiting for 'SQL*Net message from client' wait

109:waiting for 'SQL*Net message from client' wait

110:waiting for 'SQL*Net message from client' wait

111:waiting for 'SQL*Net message from client' wait

112:waiting for 'SQL*Net message from client' wait

113:waiting for 'SQL*Net message from client' wait

114:waiting for 'SQL*Net message from client' wait

115:waiting for 'SQL*Net message from client' wait

116:waiting for 'SQL*Net message from client' wait

117:waiting for 'SQL*Net message from client' wait

120:waiting for 'SQL*Net message from client' wait

NO BLOCKING PROCESSES FOUND

881983 Lines Processed.

从分析结果来看，断定pmon进程才是真正的罪魁祸首，搜索pmon关键字:

PROCESS 2:

—————————————-

SO: c0000000ad68c020, type: 2, owner: 0000000000000000, flag: INIT/-/-/0x00

(process) Oracle pid=2, calls cur/top: c0000000adab8098/c0000000adab8098, flag: (e) SYSTEM

int error: 0, call error: 0, sess error: 0, txn error 0

(post info) last post received: 0 0 48

last post received-location: ksoreq_reply

last process to post me: c0000000ad692750 4 2

last post sent: 0 0 24

last post sent-location: ksasnd

last process posted by me: c0000000ad68d000 1 6

(latch info) wait_event=0 bits=0

Process Group: DEFAULT, pseudo proc: c0000000ad784f20

O/S info: user: oracle, term: UNKNOWN, ospid: 27626

OSD pid info: Unix process pid: 27626, image: oracle@uamdb2 (PMON)

###### 省略部分信息 ######

—————————————-

SO: c0000000add586a8, type: 11, owner: c0000000ad68c020, flag: INIT/-/-/0x00

(broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: c0000000ad68c020,

event: 1, last message event: 1,

last message waited event: 1, messages read: 0

channel: (c0000000add7e4b0) scumnt mount lock

scope: 1, event: 15, last mesage event: 0,

publishers/subscribers: 0/14,

messages published: 0

—————————————-

SO: c0000000ada73a90, type: 4, owner: c0000000ad68c020, flag: INIT/-/-/0x00

(session) sid: 555 trans: 0000000000000000, creator: c0000000ad68c020, flag: (51) USR/- BSY/-/-/-/-/-

DID: 0002-0002-00000004, short-term DID: 0000-0000-00000000

txn branch: 0000000000000000

oct: 0, prv: 0, sql: 0000000000000000, psql: 0000000000000000, user: 0/SYS

service name: SYS$BACKGROUND

waiting for 'pmon timer' blocking sess=0x0000000000000000 seq=50 wait_time=0 seconds since wait started=75789

duration=12c, =0, =0

Dumping Session Wait History

for 'pmon timer' count=1 wait_time=2929152

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2929462

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2929351

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2929175

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2928083

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2929290

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2929509

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2929262

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2929283

duration=12c, =0, =0

for 'pmon timer' count=1 wait_time=2929525

duration=12c, =0, =0

temporary object counter: 0

2929152/3600/24=33天左右。换句话说，pmon进程在1个月前就出现异常了，同事也说5月26号的时候重启过，

后来似乎就有问题了，掐指一算，时间似乎刚刚对上。

最后我在中午申请了一点停机时间，我们进行了处理，首先我kill了diag进程，12g的磁盘空间得到了释放，

最开始我断定是lck进程导致的，于是kill了该进程，结果直接导致该节点重启了，不过还好的是，23s就重启完成了。

--------------------------------------ORACLE-DBA----------------------------------------

最权威、专业的Oracle案例资源汇总之【案例】Oracle服务器diag进程占据了12g的磁盘空间分析解决办法