帮朋友分析日志的过程中发现在两个节点上多次出现类此错误

节点1日志分析

Mon Mar 5 08:19:20 2012

Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc:

ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

Mon Mar 5 08:19:21 2012

Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc:

ORA-07445: exception encountered: core dump [kghalf()+961] [SIGSEGV] [Invalid permissions for mapped object] [0x000000000] [] []

ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

Mon Mar 5 08:19:22 2012

Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc:

ORA-07445: exception encountered: core dump [kghalf()+464] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

ORA-07445: exception encountered: core dump [kghalf()+961] [SIGSEGV] [Invalid permissions for mapped object] [0x000000000] [] []

ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

对应trace文件

/u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc

Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options

JServer Release 9.2.0.8.0 - Production

ORACLE_HOME = /u01/app/oracle/product/9.2.0

System name:HP-UX

Node name:HDDB1

Release:B.11.23

Version:U

Machine:ia64

Instance name: hddms1

Redo thread mounted by this instance: 1

Oracle process number: 42

Unix process pid: 18556, image: oracle@HDDB1 (TNS V1-V3)

*** 2012-03-05 08:19:20.675

*** SESSION ID:(100.40701) 2012-03-05 08:19:20.602

Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x9fffffffbf580000, PC: [0x400000000182a9f0, __milli_memcpy()+2448]

r1: 6000000000524900 r20: 8 br5: 0

r2: 0 r21: 9fffffffbf4c21ec br6: 40000000018ad000

r3: 37 r22: 8 br7: e000000120001400

r4: 0 r23: c00000005e3bac98 ip: 400000000182a9f0

r5: 60000000005311d8 r24: 9fffffffbf4c2268 iipa: 0

r6: 40000004 r25: 9fffffffbf4c2268 cfm: 2a54295428

r7: 9fffffffffff1ba8 r26: 9fffffffbf580000 um: 1a

r8: 9fffffffbf4c2268 r27: 9fffffffbf57ff00 rsc: 1f

r9: 8000 r28: 9fffffffbf580008 bsp: 9fffffffbf802150

r10: 9fffffffbf4ba260 r29: 9fffffffbf57ff08 bspstore: 9fffffffbf802150

r11: 0 r30: 0 rnat: 0

r12: 9ffffffffffe5240 r31: 0 ccv: 20000000

r13: 9fffffffbf5a1420 NaTs: 0 unat: 0

r14: 9fffffffbf4c23e8 PRs: ffffffffffff4837 fpsr: 9804c8274433f

r15: 9fffffffbf580028 br0: 400000000202a7e0 pfs: c000000000001b3e

r16: 3fffffffffffdff br1: 0 lc: 3ffffffffffce89

r17: 9fffffffbf4c2258 br2: 0 ec: 5

r18: 0 br3: 0 isr: 9fffffffbf802150

r19: 80 br4: 0 ifa: 0

Reason code: 0008

*** 2012-03-05 08:19:20.735

ksedmp: internal or fatal error

ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

Current SQL statement for this session:

update pmis.PM_BZ_DC_DWRFHSJ set ZGFDFH=:v_result where PARINST_ID=:v_pid and T_ID>:v_mdtbegin and T_ID<=:v_mdtend

----- PL/SQL Call Stack -----

object line object

handle number name

c00000008975ebc0 49 procedure PMIS.PM_DC_SJJGGC_EXECINF

c0000000971ef530 2 anonymous block

----- Call Stack Trace -----

calling call entry argument values in hex

location type point (? means dubious value)

-------------------- -------- -------------------- ----------------------------

ksedmp()+528 call ksedst() 000000001 ?

C050000000000996 ?

4000000002A53E60 ?

ssexhd()+1184 call ksedmp() 000000003 ?

C0500000000010A9 ?

40000000025E1420 ?

FFFFFFFFFFFF9075 ?

9FFFFFFFFFFDF2E0 ?

6000000000514428 ?

6000000000514420 ?

60000000005178E0 ?

call ssexhd() 400000000086EFB0 ?

000000015 ? 000000007 ?

C050000000000085 ?

__milli_memcpy()+24 call

48

$cold_updgrh()+1088 call __milli_memcpy() 000000000 ? 000000000 ?

000000000 ? 000000000 ?

9FFFFFFFBF564348 ?

000000000 ? 000000000 ?

000000000 ?

upduaw()+224 call $cold_updgrh() C0000000A2665550 ?

9FFFFFFFBF558FA0 ?

C000000000000C1E ?

4000000001916600 ?

000000000 ? 00000E137 ?

kdusru()+720 call upduaw() C0000000A2665550 ?

C000000000005BBF ?

4000000001912AC0 ?

00000E135 ?

9FFFFFFFFFFE52B0 ?

C0000000A2665660 ?

9FFFFFFFFFFE63F0 ?

9FFFFFFFBF5590D8 ?

kauupd()+544 call kdusru() 9FFFFFFFBF4C5D24 ?

000000000 ?

9FFFFFFFBF558FA0 ?

000000000 ?

C00000008B5D1108 ?

C00000008B5D1138 ?

000000018 ?

9FFFFFFFBF5591E0 ?

updrow()+3024 call kauupd() 9FFFFFFFFFFE65D0 ?

000000000 ?

6000000000530010 ?

000000000 ?

C00000009202CF38 ?

00000010B ? 00001AA7A ?

C000000086F2B778 ?

qerupFetch()+608 call updrow() 9FFFFFFFBF4C89F0 ?

C0000000A2665550 ?

C000000000000C1D ?

40000000018F5BF0 ?

000000000 ?

updaul()+1008 call qerupFetch() C0000000A0DB24D0 ?

000000000 ?

C0000000A2665550 ?

000007FFF ?

updThreePhaseExe()+ call updaul() C000000091A57FA0 ?

432 000000006 ? 000000000 ?

C0000000000019BB ?

40000000019DF9B0 ?

000008E23 ?

60000000001D30F0 ?

0000000BD ?

updexe()+624 call updThreePhaseExe() C000000091A57FA0 ?

000000000 ?

9FFFFFFFBF558FA0 ?

9FFFFFFFFFFF1AF0 ?

C000000000000FA6 ?

40000000019DFCA0 ?

000000000 ?

opiexe()+6944 call updexe() C000000091A57FA0 ?

9FFFFFFFBF4C8A00 ?

C000000000002858 ?

4000000001857620 ?

000008E25 ?

9FFFFFFFFFFF1AD0 ?

9FFFFFFFBF558FA0 ?

9FFFFFFFBF4C88D4 ?

opipls()+3488 call opiexe() 000000004 ?

9FFFFFFFFFFF2F50 ?

9FFFFFFFFFFF1D30 ?

C000000000002C60 ?

40000000019852D0 ?

000000000 ? 000007123 ?

9FFFFFFFFFFF1B40 ?

opiodr()+3088 call opipls() 9FFFFFFFFFFF2F50 ?

9FFFFFFFFFFF2F58 ?

000000004 ?

C00000000000214A ?

40000000018747B0 ?

00000E123 ?

9FFFFFFFFFFF2E70 ?

9FFFFFFFFFFF2DE0 ?

rpidrus()+304 call opiodr() 9FFFFFFFFFFF3810 ?

9FFFFFFFFFFF37E0 ?

9FFFFFFFFFFF36F8 ?

9FFFFFFFFFFF36F4 ?

skgmstack()+288 call rpidrus() 9FFFFFFFFFFF4DA0 ?

C000000000000716 ?

4000000001862FE0 ?

9FFFFFFFFFFF47F0 ?

600000000004F3C0 ?

9FFFFFFFFFFF4DB8 ?

6000000000524900 ?

9FFFFFFFFFFF4DA0 ?

rpidru()+256 call skgmstack() 9FFFFFFFFFFF4DC0 ?

600000000004EDF0 ?

00000F618 ?

400000000129E400 ?

9FFFFFFFFFFF4DA0 ?

rpiswu2()+784 call rpidru() 00000057B ?

C000000000001329 ?

4000000001870CE0 ?

rpidrv()+2096 call rpiswu2() C00000007D7B3708 ?

9FFFFFFFFFFF5510 ?

6000000000515CE8 ?

60000000000502E8 ?

9FFFFFFFFFFF5520 ?

000000205 ?

6000000000530010 ?

6000000000531090 ?

psddr0()+256 call rpidrv() 000000000 ? 000000066 ?

9FFFFFFFFFFF6630 ?

00000003A ?

psdnal()+672 call psddr0() 000000000 ? 000000066 ?

9FFFFFFFFFFF6630 ?

000000030 ?

pevm_EXIM()+400 call psdnal() 9FFFFFFFFFFF84A0 ?

9FFFFFFFFFFF8440 ?

6000000000531A04 ?

600000000064BAA0 ?

C000000090880CC8 ?

600000000063EA80 ?

pfrrun()+37120 call pevm_EXIM() 9FFFFFFFBF567AD8 ?

C000000090880CC8 ?

C000000000001532 ?

400000000173DAD0 ?

000006269 ?

4000000002190B40 ?

9FFFFFFFBF4BA068 ?

9FFFFFFFBF5C08A0 ?

peicnt()+544 call pfrrun() 9FFFFFFFBF567CDC ?

C000000000000D1D ?

4000000001676FE0 ?

kkxexe()+832 call peicnt() 9FFFFFFFFFFF84A0 ?

9FFFFFFFBF567AD8 ?

C000000000000818 ?

40000000021A3A80 ?

00000FEAB ?

9FFFFFFFFFFF7EB0 ?

9FFFFFFFBF567CDC ?

000000000 ?

opiexe()+12592 call kkxexe() 9FFFFFFFFFFF8430 ?

C000000000002858 ?

4000000001858C30 ?

0000080AB ?

9FFFFFFFFFFF8420 ?

9FFFFFFFBF4F3A6C ?

9FFFFFFFBF4F3A68 ?

9FFFFFFFFFFF84A0 ?

opiall0()+3456 call opiexe() 000000004 ?

9FFFFFFFFFFF9A80 ?

9FFFFFFFFFFF8830 ?

C0000000000024D1 ?

40000000017A3350 ?

000000000 ? 00000C82B ?

9FFFFFFFFFFF8640 ?

kpoal8()+2272 call opiall0() 000000002 ?

9FFFFFFFFFFF9A10 ?

9FFFFFFFFFFFA2B4 ?

000000000 ?

9FFFFFFFFFFF9954 ?

9FFFFFFFFFFFA2BC ?

000000000 ?

FFFFFFFFFFFFDFFF ?

opiodr()+3088 call kpoal8() 9FFFFFFFFFFFA0B0 ?

000000000 ? 000000000 ?

C00000000000214A ?

40000000018747B0 ?

00000F82F ?

9FFFFFFFFFFFA130 ?

9FFFFFFFFFFFA1D0 ?

ttcpip()+1888 call opiodr() 9FFFFFFFFFFFA460 ?

9FFFFFFFFFFFA430 ?

9FFFFFFFFFFFA348 ?

9FFFFFFFFFFFA344 ?

C0000000000018B9 ?

400000000172CB70 ?

000000000 ? 00000C0AB ?

opitsk()+1920 call ttcpip() 6000000000052C40 ?

000000002 ?

9FFFFFFFFFFFB950 ?

6000000000273AA0 ?

9FFFFFFFFFFFBAB0 ?

9FFFFFFFFFFFB8C4 ?

9FFFFFFFBF75A660 ?

4000000000D2C7AA ?

opiino()+2656 call opitsk() 000000000 ? 000000000 ?

C000000000000D1F ?

400000000236CE90 ?

000000000 ?

opiodr()+3088 call opiino() 60000000005DEC18 ?

6000000000531170 ?

9FFFFFFFFFFFF4C0 ?

C00000000000214A ?

40000000018747B0 ?

00000E825 ?

9FFFFFFFFFFFD7C0 ?

9FFFFFFFFFFFF4D0 ?

opidrv()+1088 call opiodr() 9FFFFFFFFFFFDF10 ?

9FFFFFFFFFFFDEE0 ?

9FFFFFFFFFFFDDF8 ?

9FFFFFFFFFFFDDF4 ?

sou2o()+48 call opidrv() 9FFFFFFFFFFFEF60 ?

000000004 ?

9FFFFFFFFFFFF4C0 ?

main()+352 call sou2o() 9FFFFFFFFFFFF4E0 ?

00000003C ? 000000004 ?

9FFFFFFFFFFFF4C0 ?

main_opd_entry()+80 call main() 000000000 ?

9FFFFFFFFFFFF9B0 ?

C000000000000004 ?

C00000000002FA60 ?

--------------------- Binary Stack Dump ---------------------

观察节点2，发现类此错误不同之处有

Wed Mar 7 08:19:11 2012

Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc:

ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

Wed Mar 7 08:19:12 2012

Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc:

ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], []

ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

Wed Mar 7 08:19:13 2012

Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc:

ORA-07445: exception encountered: core dump [kgidmp()+1568] [SIGSEGV] [Address not mapped to object] [0xB38F00000000D9] [] []

ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], []

ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

Wed Mar 7 08:19:14 2012

Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc:

ORA-07445: exception encountered: core dump [kghalf()+993] [SIGSEGV] [Address not mapped to object] [0xC0000000A8B01DA0] [] []

ORA-07445: exception encountered: core dump [kgidmp()+1568] [SIGSEGV] [Address not mapped to object] [0xB38F00000000D9] [] []

ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], []

ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

对应trace文件不同之处

*** 2012-03-07 08:19:11.582

ksedmp: internal or fatal error

ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

No current SQL statement being executed.

----- PL/SQL Call Stack -----

object line object

handle number name

c000000090c4a818 1356 procedure PMIS.PM_DC_SJJGGC_QDX

c0000000a25512c0 43 procedure PMIS.PM_DC_SJJGGC_EXECINF

c00000009a8306e8 2 anonymous block

通过这两个节点的错误记录，都包含PMIS.PM_DC_SJJGGC_EXECINF和anonymous，所以初步怀疑是某个程序在这个时间定运行某个任务(包含PMIS.PM_DC_SJJGGC_EXECINF过程)导致。

继续观察trace日志发现两个节点都是类此情况相同的程序，相同的主机

SO: c00000007d7b3708, type: 4, owner: c00000007d6d0190, flag: INIT/-/-/0x00

(session) trans: c000000082974470, creator: c00000007d6d0190, flag: (100041) USR/- BSY/-/-/-/-/-

DID: 0001-002A-0000693D, short-term DID: 0000-0000-00000000

txn branch: 0000000000000000

oct: 6, prv: 0, sql: c0000000a229ab38, psql: c000000084e6b510, user: 517/PMIS

O/S info: user: Administrator, term: DMIS_SWITCH2, ospid: 5600:3020, machine: WORKGROUP\DMIS_SWITCH2

program: PMTDCtrl.exe

application name: PMTDCtrl.exe, hash value=0

last wait for 'enqueue' blocking sess=0x0 seq=32849 wait_time=127

name|mode=54540004, id1=1, id2=10

temporary object counter: 0

现在更加怀疑是PMTDCtrl.exe运行某过程导致数据库出现这些错误

错误总结

查看MOS发现该问题是一个bug导致(Bug 6166690:ORA-7445 [MILLI_MEMCPY] WHEN REF CURSOR AS OUT PARAMETER)

错误原因:It is caused by a Dynamic SQL used to obtain a REF CURSOR from a procedure (that opens the cursor).

处理建议:通过个bug，我们可以分析PMIS.PM_DC_SJJGGC_EXECINF过程，看看是否符合bug描述,如果符合建议改写过程或者打上补丁(Patch 2709343)

补丁:p2709343_92080_HPUX-IA64

oracle 000000000A2F6630,ORA-7445[__milli_memcpy]分析相关推荐

  1. 看懂oracle trc文件,oracle 的 debug 工具 和 分析 trc 文件工具 ass.awk

    oracle的dump工具和分析trc文件工具ass.awk 之前介绍了oracle的一个hang住时的一个分析工具hanganalyze,这里再引入另外的一个工具,分析系统进程和单个进程的工具 1. ...

  2. ORACLE全部ORA错误收集整理

    有时数据库出现问题,不是每次都有网络可查,所以把所有的ora系列的错误整理出来, 在最没有办法的时候,需要自己来解决,有了这些根据,问题会好办的.虽说对于数据库方面, DBA很强大,他们在遇到错误时, ...

  3. java程序造成ora-7445_Problem Key 'ORA 7445 [pevm_icd_call_common]'

    新装一套19c rac,使用了10月份补丁.其中一个节点的告警日志一直报以下错,cdump把oracle_home目录快撑满了. DDE: Problem Key 'ORA 7445 [pevm_ic ...

  4. python查询oracle数据库_python针对Oracle常见查询操作实例分析

    本文实例讲述了python针对Oracle常见查询操作.分享给大家供大家参考,具体如下: 1.子查询(难): 当进行查询的时候,发现需要的数据信息不明确,需要先通过另一个查询得到, 此查询称为子查询: ...

  5. oracle性能优化之awr分析

    oracle性能优化之awr分析 作者:bingjava 最近某证券公司系统在业务期间系统运行缓慢,初步排查怀疑是数据库存在性能问题,因此导出了oracle的awr报告进行分析,在此进行记录. 导致系 ...

  6. 崔华 oracle简历,2013数据库大会:崔华-基于Oracle的SQL优化案例分析

    2013数据库大会:崔华-基于Oracle的SQL优化案例分析 崔华的新书即将出版,其数据库大会上的演讲也非常精彩,他的新书十分值得期待. 2013年中国数据库技术大会第二天的"Oracle ...

  7. oracle tnsnames.ora文件用法说明

    oracle tnsnames.ora文件用法说明 CreationTime--2018年8月10日08点32分 Author:Marydon 1.用途 oracle客户端所需要的一个文件,通过该文件 ...

  8. Oracle JOB异常中断原因分析

    链接:http://blog.itpub.net/28602568/viewspace-1731805/ 标题: Oracle JOB异常中断原因分析 作者:lōττéry©版权所有[文章允许转载,但 ...

  9. 12505 java_Java jdbc Oracle error: ORA 12505

    Java jdbc Oracle error: ORA 12505 更改oracle Listener 配置:D:\Oracle\moon\product\11.1.0\db_1\NETWORK\AD ...

  10. Oracle常见ORA错误及解决办法

    Oracle常见ORA错误及解决办法 问题1 1.1 错误现象: EXP-00008: ORACLE error 1013 encountered ORA-01013: user requested ...

最新文章

  1. JDK8中好用的日期处理-LocalDate类-LocalTime-LocalDateTIme，mysql解决时区相差13小时的问题，日期格式器DateTimeFormatter
  2. POJ-3067 Japan---树状数组逆序对变形
  3. 搭建MySQL高可用架构MHA
  4. .NET Oject And Json
  5. spring相关记录
  6. 基于java网上商城源码设计
  7. Flash cs4 for mac 序列号。
  8. python的统计库_Python-Scipy库-卡方分布统计量计算
  9. 原始对偶方法——转载
  10. 为什么我的 WordPress 网站被封了？
  11. 锂电池基本原理解析：充电及放电机制
  12. 卫星勘测洪水数据网站
  13. 在微型计算机中1mb等于多少字节,1MB等于多少字节？
  14. 华为鸿蒙OS5摄概念机,华为P50Pro概念图：首发鸿蒙OS，后置5摄能让iPhone12甘拜下风吗...
  15. JAVA如何利用迅雷接口解析磁力,关于神秘代码(磁力链接)如何正确使用
  16. 数据仓库之数据质量监控
  17. 用UDP实现简易的聊天室
  18. 怎样在Mac或Windows上的Parallels中删除虚拟机？
  19. Matplotlib实现单画布绘制多个子图
  20. Kubernetes中Sidecar生命周期管理

热门文章

  1. 「GoTeam 招聘时间」星图比特 后端开发工程师（上海）
  2. 如何求数组中绝对值最小的数？
  3. 面试经历（华为，瑞晟，大华，海康，虹软，顺丰）
  4. linux 两个子进程,父进程创建两个子进程,在收到中断信号后杀死两个子进程
  5. 【用例】研究生招生报名管理系统分析
  6. 如何用VR改变驾驶陋习?
  7. 汇编中的dup和equ
  8. python3 ldap 查询用户，使用管理重置用户密码
  9. windows开启Hyper-V虚拟化功能并创建虚拟机
  10. 为什么现在流行resultful,webservice无人问津?