这个吧。

Bug No. 5656453

Filed 11-NOV-2006 Updated 13-AUG-2008

Product Oracle Server - Enterprise Edition Product Version  9.2.0.7

Platform HP-UX PA-RISC (64-bit) Platform Version No Data

Database Version 9.2.0.7 Affects Platforms  Generic

Severity  Severe Loss of Service Status Development to Q/A

Base Bug N/A Fixed in Product Version 10.1.0.99

Problem statement:

ORA-29740 ON A 2-NODE CLUSTER WITH "IPC SEND TIMEOUT"

--------------------------------------------------------------------------------

*** 11/11/06 02:08 am *** TAR: ---- 16833194.600 . PROBLEM: -------- ORA-29740, instance eviction, on a 2-node cluster. . From alert.log: -------------- .     Sat Nov 11 01:53:09 2006     IPC Send timeout detected. Sender ospid 4693     Sat Nov 11 01:53:38 2006     IPC Send timeout detected. Sender ospid 2593     Sat Nov 11 01:54:24 2006     IPC Send timeout detected. Sender ospid 4949     Sat Nov 11 01:54:48 2006     IPC Send timeout detected. Sender ospid 5055     Sat Nov 11 01:54:55 2006     IPC Send timeout detected. Sender ospid 5057     Sat Nov 11 01:55:15 2006     IPC Send timeout detected. Sender ospid 5276     Sat Nov 11 01:56:48 2006     Errors in file /oracle/admin/CTSPRD2/bdump/ctsprd2_lmon_19720.trc:     ORA-29740: evicted by member 1, group incarnation 5     Sat Nov 11 01:56:56 2006     Errors in file /oracle/admin/CTSPRD2/bdump/ctsprd2_lmon_19720.trc:     ORA-29740: evicted by member 1, group incarnation 5     Sat Nov 11 01:56:56 2006     LMON: terminating instance due to error 29740     Sat Nov 11 01:56:56 2006 . From Lmon trace file: -------------------- .  Reconfiguration complete     *** 2006-11-11 01:45:02.363     kjxgrcomerr: Communications reconfig: instance 0 (4,4)     Submitting asynchronized dump request [2]     kjxgrrcfgchk: Initiating reconfig, reason 3     *** 2006-11-11 01:45:07.492     kjxgmrcfg: Reconfiguration started, reason 3                    kjxgmcs: Setting state to 4 1.     *** 2006-11-11 01:45:07.503     Obtained RR update lock for sequence 4, RR seq 4     *** 2006-11-11 01:46:48.256     kjxgrrecp2: Waiting for split-brain resolution, upd 1, seq 5          Stack trace :          ksedmp ksddoa ksdpcg ksdpec ksfpec kgesev       ksesec2 kjxgrdtrt  kjxgrrecp2 kjxgrpropvote     kjxgrrcfgchk kjxggpoll       kjfmact kjfdact kjrchc kjfcln ksbrdp opirip        opidrv sou2o main     $START$ . . One-off patch for Bug 4047167 is already applied. . Now, they started the cluster back and both nodes are up. . OS watcher is not installed.  Now we got it installed. . DIAGNOSTIC ANALYSIS: -------------------- . WORKAROUND: ----------- . RELATED BUGS: ------------- bug 4047167 . REPRODUCIBILITY: ---------------- . TEST CASE: ---------- . STACK TRACE: ------------     ksedmp ksddoa ksdpcg ksdpec ksfpec kgesev       ksesec2 kjxgrdtrt  kjxgrrecp2 kjxgrpropvote     kjxgrrcfgchk kjxggpoll       kjfmact kjfdact kjrchc kjfcln ksbrdp opirip        opidrv sou2o main     $START$ . SUPPORTING INFORMATION: ----------------------- . 24 HOUR CONTACT INFORMATION FOR P1 BUGS: ---------------------------------------- . DIAL-IN INFORMATION: -------------------- . IMPACT DATE: ------------ . *** 11/11/06 02:09 am *** (CHG: Sta->16) *** 11/11/06 02:24 am *** *** 11/11/06 03:17 am *** (CHG: Sta->10) *** 11/11/06 03:17 am *** *** 11/11/06 03:26 am *** *** 11/11/06 04:33 am *** *** 11/11/06 04:45 am *** *** 11/11/06 04:48 am *** *** 11/11/06 08:30 am *** *** 11/11/06 09:23 am *** *** 11/11/06 09:49 am *** (CHG: Sta->16) *** 11/11/06 09:49 am *** *** 11/11/06 10:27 am *** *** 11/11/06 10:28 am *** (CHG: Asg->NEW OWNER) *** 11/11/06 11:24 am *** (CHG: Sta->10) *** 11/11/06 11:24 am *** *** 11/11/06 11:55 am *** *** 11/11/06 02:14 pm *** (CHG: Sta->16) *** 11/11/06 02:14 pm *** *** 11/11/06 03:58 pm *** *** 11/11/06 03:58 pm *** (CHG: Sta->10) *** 11/11/06 05:53 pm *** *** 11/11/06 05:54 pm *** *** 11/12/06 09:26 am *** (CHG: Sta->16) *** 11/12/06 09:26 am *** *** 11/12/06 11:21 am *** (CHG: Sta->10) *** 11/12/06 11:21 am *** *** 11/12/06 02:55 pm *** (CHG: Sta->16) *** 11/12/06 02:55 pm *** *** 11/12/06 03:12 pm *** *** 11/12/06 03:21 pm *** (CHG: Sta->10) *** 11/12/06 03:21 pm *** *** 11/12/06 03:45 pm *** *** 11/12/06 04:36 pm *** (CHG: Sta->16) *** 11/12/06 04:36 pm *** *** 11/12/06 06:41 pm *** *** 11/12/06 07:09 pm *** (CHG: Sta->10) *** 11/12/06 07:09 pm *** *** 11/13/06 03:19 am *** *** 11/13/06 03:20 am *** *** 11/13/06 03:20 am *** (CHG: Sta->16) *** 11/13/06 04:45 am *** (CHG: Sta->10) *** 11/13/06 04:45 am *** *** 11/13/06 05:33 am *** *** 11/13/06 06:03 am *** *** 11/13/06 06:47 am *** (CHG: Pri->2) *** 11/13/06 06:47 am *** *** 11/13/06 06:56 am *** *** 11/13/06 06:56 am *** (CHG: Sta->16) *** 11/13/06 07:01 am *** (CHG: Sta->10) *** 11/13/06 07:51 am *** *** 11/13/06 08:03 am *** *** 11/13/06 08:04 am *** *** 11/13/06 10:03 am *** *** 11/13/06 12:48 pm *** (CHG: Sta->16) *** 11/13/06 12:48 pm *** *** 11/13/06 12:52 pm *** *** 11/13/06 01:23 pm *** (CHG: Sta->10) *** 11/13/06 01:23 pm *** *** 11/13/06 01:24 pm *** *** 11/13/06 01:36 pm *** *** 11/13/06 05:00 pm *** (CHG: Sta->16) *** 11/13/06 05:00 pm *** *** 11/13/06 05:58 pm *** (CHG: Sta->10) *** 11/13/06 05:58 pm *** *** 11/13/06 06:30 pm *** *** 11/13/06 06:30 pm *** (CHG: Sta->16) *** 11/13/06 06:52 pm *** *** 11/14/06 06:37 am *** (CHG: Sta->10) *** 11/14/06 06:37 am *** *** 11/14/06 07:28 am *** *** 11/14/06 07:28 am *** (CHG: Sta->16) *** 11/14/06 08:57 am *** (CHG: Sta->10) *** 11/14/06 09:05 am *** *** 11/14/06 11:16 am *** *** 11/14/06 04:07 pm *** . *** 11/14/06 04:12 pm *** (CHG: Sta->16) *** 11/14/06 06:25 pm *** *** 11/15/06 06:15 am *** *** 11/15/06 06:15 am *** (CHG: Sta->10) *** 11/15/06 08:50 am *** *** 11/15/06 01:35 pm *** *** 11/15/06 03:16 pm *** (CHG: Sta->16) *** 11/15/06 03:16 pm *** *** 11/16/06 03:38 am *** *** 11/16/06 03:38 am *** *** 11/16/06 05:32 am *** (CHG: Sta->10) *** 11/16/06 05:33 am *** *** 11/16/06 01:28 pm *** (CHG: Sta->16) *** 11/16/06 01:28 pm *** *** 11/16/06 01:54 pm *** (CHG: Sta->10) *** 11/16/06 01:54 pm *** *** 11/17/06 08:26 pm *** *** 11/17/06 08:45 pm *** *** 11/17/06 08:45 pm *** (CHG: Sta->16) *** 11/17/06 08:51 pm *** *** 11/18/06 05:04 am *** (CHG: Sta->10) *** 11/18/06 05:04 am *** *** 11/18/06 05:56 am *** *** 11/18/06 06:01 am *** *** 11/18/06 06:02 am *** (CHG: Sta->16) *** 11/18/06 06:11 am *** *** 11/18/06 06:12 am *** (CHG: Sta->10) *** 11/18/06 06:13 am *** *** 11/18/06 06:21 am *** *** 11/19/06 10:20 am *** *** 11/19/06 10:22 am *** (CHG: Sta->16) *** 11/20/06 06:34 am *** (CHG: Sta->10) *** 11/20/06 06:34 am *** *** 11/20/06 10:20 am *** (CHG: Sta->16) *** 11/20/06 10:20 am *** *** 11/20/06 10:30 am *** *** 11/20/06 11:02 am *** (CHG: Sta->10) *** 11/20/06 11:02 am *** *** 11/20/06 04:24 pm *** (CHG: Sta->16) *** 11/20/06 04:24 pm *** *** 11/20/06 05:15 pm *** *** 11/20/06 05:46 pm *** *** 11/20/06 05:46 pm *** (CHG: Sta->11) *** 11/20/06 05:47 pm *** (CHG: Asg->NEW OWNER) *** 11/20/06 05:47 pm *** *** 11/21/06 10:18 am *** *** 11/21/06 10:18 am *** (CHG: Confirmed Flag->Y) *** 11/21/06 10:18 am *** (CHG: Asg->NEW OWNER) *** 11/21/06 10:18 am *** (CHG: Asg->NEW OWNER) *** 11/24/06 03:56 am *** (CHG: Asg->NEW OWNER) *** 11/24/06 03:56 am *** *** 11/24/06 05:10 am *** (CHG: DevPri->2) *** 11/24/06 05:10 am *** (CHG: Sta->30) *** 11/24/06 05:10 am *** *** 11/24/06 06:27 am *** (CHG: Sta->11) *** 11/24/06 06:27 am *** *** 11/24/06 11:08 am *** *** 11/30/06 06:17 am *** *** 12/01/06 08:42 am *** *** 12/01/06 08:54 am *** *** 12/01/06 12:27 pm *** ESCALATED *** 12/01/06 12:28 pm *** *** 12/05/06 12:29 pm *** *** 12/05/06 01:08 pm *** *** 12/13/06 06:15 am *** *** 12/13/06 06:48 am *** (CHG: Sta->30) *** 12/13/06 06:48 am *** *** 12/13/06 06:01 pm *** *** 12/14/06 05:50 am *** (CHG: Sta->11) *** 12/14/06 05:50 am *** *** 12/14/06 06:19 am *** *** 12/14/06 06:20 am *** (CHG: Sta->30) *** 12/14/06 06:31 am *** *** 12/14/06 06:38 am *** (CHG: Sta->11) *** 12/14/06 06:38 am *** *** 12/14/06 07:09 am *** *** 12/14/06 07:21 am *** *** 12/19/06 05:00 am *** *** 12/28/06 12:10 pm *** *** 12/29/06 03:09 pm *** *** 01/17/07 12:06 pm *** *** 01/22/07 02:16 pm *** *** 01/24/07 02:51 pm *** (CHG: Sta->30) *** 01/24/07 02:51 pm *** *** 02/13/07 02:26 pm *** (CHG: Sta->11) *** 02/13/07 02:26 pm *** *** 02/20/07 02:54 pm *** *** 02/20/07 03:14 pm *** (CHG: Pri->1) *** 02/20/07 03:18 pm *** (CHG: Pri->2) *** 02/20/07 03:36 pm *** (CHG: Sta->30) *** 02/20/07 03:36 pm *** *** 02/20/07 03:49 pm *** *** 02/20/07 03:59 pm *** *** 02/20/07 08:51 pm *** *** 02/20/07 09:01 pm *** *** 02/20/07 09:31 pm *** *** 02/20/07 10:38 pm *** *** 02/20/07 11:12 pm *** *** 02/20/07 11:14 pm *** (CHG: Sta->11) *** 02/21/07 02:55 am *** *** 02/21/07 08:48 am *** (CHG: Sta->30) *** 02/21/07 08:48 am *** *** 02/21/07 08:52 am *** *** 02/21/07 10:01 am *** (CHG: Sta->11) *** 02/21/07 10:01 am *** *** 02/21/07 02:39 pm *** *** 02/21/07 02:40 pm *** *** 02/21/07 02:59 pm *** (CHG: Pri->1) *** 02/21/07 03:03 pm *** (CHG: Pri->2) *** 02/21/07 03:03 pm *** ESCALATION -> CLOSED *** 02/21/07 03:14 pm *** *** 02/23/07 08:44 am *** *** 02/23/07 10:10 am *** (CHG: Sta->30) *** 02/23/07 10:10 am *** *** 02/23/07 10:19 am *** *** 02/23/07 11:07 am *** (CHG: Sta->11) *** 02/23/07 11:07 am *** *** 02/23/07 11:52 am *** *** 02/23/07 11:52 am *** (CHG: Sta->30) *** 02/23/07 07:01 pm *** ESCALATION -> REOPENED *** 02/26/07 10:04 am *** *** 02/26/07 10:13 am *** *** 02/26/07 10:32 am *** *** 02/26/07 10:36 am *** *** 02/26/07 10:51 am *** *** 02/26/07 12:32 pm *** (CHG: Sta->11) *** 02/26/07 12:32 pm *** *** 02/26/07 01:15 pm *** (CHG: Sta->30) *** 02/26/07 01:16 pm *** *** 02/26/07 03:22 pm *** *** 02/26/07 03:27 pm *** (CHG: Sta->11) *** 02/26/07 05:11 pm *** *** 02/27/07 08:42 am *** (CHG: Sta->30) *** 02/27/07 08:42 am *** *** 02/27/07 08:43 am *** (CHG: Sta->11) *** 02/27/07 10:53 pm *** (CHG: Asg->NEW OWNER) *** 02/27/07 10:53 pm *** *** 02/28/07 04:14 pm *** *** 02/28/07 04:18 pm *** *** 02/28/07 07:16 pm *** *** 03/01/07 01:00 pm *** *** 03/01/07 01:48 pm *** *** 03/01/07 03:37 pm *** *** 03/01/07 03:50 pm *** *** 03/02/07 06:09 am *** *** 03/05/07 08:36 am *** *** 03/05/07 10:41 am *** *** 03/05/07 12:37 pm *** *** 03/05/07 06:11 pm *** *** 03/05/07 06:12 pm *** *** 03/05/07 06:17 pm *** *** 03/06/07 12:00 am *** *** 03/06/07 09:12 am *** *** 03/06/07 09:12 am *** *** 03/06/07 09:13 am *** *** 03/06/07 09:13 am *** *** 03/06/07 09:14 am *** *** 03/06/07 09:14 am *** *** 03/06/07 09:14 am *** *** 03/06/07 09:15 am *** *** 03/06/07 09:16 am *** *** 03/06/07 09:16 am *** *** 03/06/07 09:16 am *** *** 03/06/07 09:17 am *** *** 03/06/07 09:19 am *** (CHG: Sta->30) *** 03/06/07 09:19 am *** *** 03/07/07 01:02 pm *** (CHG: Sta->11) *** 03/07/07 01:02 pm *** *** 03/08/07 01:01 am *** *** 03/12/07 01:14 am *** *** 03/19/07 07:05 pm *** *** 03/22/07 07:58 am *** *** 03/22/07 07:58 am *** *** 03/22/07 07:58 am *** *** 03/22/07 07:59 am *** *** 03/22/07 07:59 am *** *** 03/22/07 07:59 am *** *** 03/22/07 08:00 am *** *** 03/22/07 08:00 am *** *** 03/22/07 08:01 am *** (CHG: Fixed->10.1.0.99) *** 03/22/07 08:01 am *** (CHG: Sta->80) *** 03/22/07 08:01 am *** Rediscovery Information:     If your problem fulfills all of the conditions below, it's highly   likely to be a duplicate of this problem. . 1. Using Oracle UDP IPC library. 2. A dedicated server process or a shared server process reported    'IPC Send timeout'. 3. SKGXPCTX dump of the server process shows send request(s) with    'CNT=2' or 'CNT=3'. . Workaround: None.   Release Notes: ]] Oracle reported 'IPC Send timeout' though there is no network ]] failure. *** 03/22/07 01:34 pm *** *** 03/22/07 02:03 pm *** *** 03/22/07 02:03 pm *** *** 03/30/07 08:58 pm *** *** 03/31/07 12:48 am *** *** 04/01/07 04:57 pm *** ESCALATION -> CLOSED *** 04/02/07 12:39 am *** *** 04/04/07 05:13 am *** *** 04/04/07 05:15 am *** *** 04/13/07 04:24 am *** *** 04/13/07 04:24 am *** *** 04/15/07 08:05 pm *** *** 04/18/07 01:01 am *** *** 04/20/07 04:38 pm *** *** 04/23/07 04:44 am *** *** 04/23/07 04:46 am *** *** 05/03/07 01:04 am *** *** 05/03/07 01:05 am *** *** 05/07/07 11:27 pm *** *** 05/08/07 12:16 am *** *** 05/10/07 12:21 am *** *** 06/04/07 12:14 am *** *** 06/04/07 12:15 am *** *** 06/06/07 08:19 am *** *** 06/11/07 11:18 pm *** *** 06/11/07 11:20 pm *** *** 06/22/07 04:36 pm *** *** 06/22/07 04:41 pm *** *** 06/24/07 12:22 am *** *** 06/24/07 12:24 am *** *** 06/28/07 03:45 am *** *** 06/28/07 07:00 am *** *** 06/28/07 07:02 am *** *** 07/05/07 06:01 pm *** *** 07/06/07 04:23 am *** *** 07/09/07 12:20 am *** *** 07/09/07 12:20 am *** *** 08/24/07 09:20 am *** *** 08/27/07 03:18 pm *** *** 09/06/07 08:06 pm *** *** 09/06/07 11:40 pm *** *** 09/06/07 11:40 pm *** *** 12/20/07 12:56 am *** *** 12/20/07 03:10 am *** *** 12/20/07 03:10 am *** *** 01/03/08 07:09 pm *** *** 01/03/08 07:09 pm *** *** 01/10/08 01:20 am *** *** 01/14/08 05:57 pm *** *** 01/21/08 12:43 am *** *** 01/21/08 12:47 am *** *** 01/23/08 01:08 am *** *** 01/23/08 03:26 am *** *** 03/14/08 04:03 am *** *** 03/19/08 02:42 am *** *** 03/19/08 02:42 am *** *** 03/27/08 04:36 am *** *** 04/01/08 12:50 am *** *** 04/04/08 02:14 am *** *** 04/04/08 02:14 am *** *** 04/21/08 03:44 pm *** *** 04/22/08 01:18 am *** *** 04/22/08 01:18 am *** *** 04/22/08 03:24 pm *** *** 04/22/08 10:54 pm *** *** 04/22/08 10:54 pm *** *** 07/18/08 11:53 am *** *** 07/18/08 12:04 pm ***  bug 3588448 bug 4047167 bug 4672266 bug 3623774 bug 4627236 bug 4158957  bug 4396718 bug 4204954 *** 08/12/08 11:29 pm *** *** 08/12/08 11:34 pm *** *** 08/13/08 02:16 am *** *** 08/13/08 07:16 pm *** *** 08/13/08 07:16 pm ***