[suse-oracle] Kernel NULL pointer dereference

Kind Folks,

This is a two node Oracle RAC 11.1.0.7 on SuSE 10 SP3.

One of the node went down earlier today. No clear leads in the database/CRS
logs. However we were able to find the following in the system log
(/var/log/messages).

The stack trace (underlined) mentions about an Oracle process. Has anyone
seen this before?

Aug  1 05:38:28 ustrlxpd15 kernel: Unable to handle kernel NULL pointer
dereference at 0000000000000000 RIP:
Aug  1 05:38:28 ustrlxpd15 kernel: <0000000000000000>{_stext+2146431208}
Aug  1 05:38:28 ustrlxpd15 kernel: PGD c9c209067 PUD e0eba0067 PMD 0
Aug  1 05:38:28 ustrlxpd15 kernel: Oops: 0010 [3] SMP
Aug  1 05:38:28 ustrlxpd15 kernel: last sysfs
file: /devices/pci0000:00/0000:00:00.0/irq
Aug  1 05:38:28 ustrlxpd15 kernel: CPU 22
Aug  1 05:38:28 ustrlxpd15 kernel: Modules linked in: af_packet softdog
mptctl hpilo raw ipv6 dock oracleasm button battery ac apparmor loop
emcpvlumd emcpxcrypt emcpdm emcpgpx emcpmpx emcp usbhid dm_mod uhci_hcd
ehci_hcd usbcore be2net sg reiserfs mptsas scsi_transport_sas mptfc mptspi
mptscsih scsi_transport_spi mptbase edd fan thermal processor cciss qla2xxx
firmware_class scsi_transport_fc sd_mod scsi_mod
Aug  1 05:38:28 ustrlxpd15 kernel: Pid: 26294, comm: oracle Tainted: P
U 2.6.16.60-0.77.1-smp #1
Aug  1 05:38:28 ustrlxpd15 kernel: RIP: 0010:[<0000000000000000>]
<0000000000000000>{_stext+2146431208}
Aug  1 05:38:28 ustrlxpd15 kernel: RSP: 0018:ffff8100d65f3c10  EFLAGS:
00010087
Aug  1 05:38:28 ustrlxpd15 kernel: RAX: 00000000ef65e530 RBX:
0000000000000001 RCX: 0000000000000000
Aug  1 05:38:28 ustrlxpd15 kernel: RDX: 0000000000000000 RSI:
0000000000000001 RDI: ffff8107ef65e530
Aug  1 05:38:28 ustrlxpd15 kernel: RBP: ffff8100d65f3c48 R08:
ffff8107ef65e548 R09: ffff8100d65f3b18
Aug  1 05:38:28 ustrlxpd15 kernel: R10: ffff81080c1a6780 R11:
0000000000000070 R12: ffff8107b4fcbe18
Aug  1 05:38:28 ustrlxpd15 kernel: R13: ffff8109208f3bb0 R14:
0000000000000000 R15: 0000000000000000
Aug  1 05:38:28 ustrlxpd15 kernel: FS:  00002b77f7437e60(0000)
GS:ffff810fefa5eb40(0000) knlGS:0000000000000000
Aug  1 05:38:28 ustrlxpd15 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Aug  1 05:38:28 ustrlxpd15 kernel: CR2: 0000000000000000 CR3:
0000000b9c2c3000 CR4: 00000000000006e0
Aug  1 05:38:28 ustrlxpd15 kernel: Process oracle (pid: 26294, threadinfo
ffff8100d65f2000, task ffff8107b1f97080)
Aug  1 05:38:28 ustrlxpd15 kernel: Stack: ffffffff8012b9e7 ef65e53000000001
0000000000000000 0000000000000001
Aug  1 05:38:28 ustrlxpd15 kernel:        ffff8109208f3bb0 0000000000000001
0000000000000292 ffff8100d65f3c88
Aug  1 05:38:28 ustrlxpd15 kernel:        ffffffff8012ca07 ffff8107ef44b2f0
Aug  1 05:38:28 ustrlxpd15 kernel: Call Trace: <ffffffff8012b9e7>
{__wake_up_common+64}
Aug  1 05:38:28 ustrlxpd15 kernel:        <ffffffff8012ca07>{__wake_up+56}
<ffffffff8028c5e3>{sock_def_readable+52}
Aug  1 05:38:28 ustrlxpd15 kernel:        <ffffffff802ee1ae>
{unix_stream_sendmsg+672} <ffffffff8028824e>{do_sock_write+201}
Aug  1 05:38:28 ustrlxpd15 kernel:        <ffffffff802891a8>{sock_aio_write
+72} <ffffffff8018738d>{do_sync_write+199}
Aug  1 05:38:28 ustrlxpd15 kernel:        <ffffffff80149032>
{autoremove_wake_function+0} <ffffffff8019a011>{do_sys_poll+814}
Aug  1 05:38:28 ustrlxpd15 kernel:        <ffffffff80187cd9>{vfs_write+234}
<ffffffff8018828f>{sys_write+69}
Aug  1 05:38:28 ustrlxpd15 kernel:        <ffffffff8010ae36>{system_call
+126}
Aug  1 05:38:28 ustrlxpd15 kernel:
Aug  1 05:38:28 ustrlxpd15 kernel: Code:  Bad RIP value.
Aug  1 05:38:28 ustrlxpd15 kernel: RIP <0000000000000000>{_stext
+2146431208} RSP <ffff8100d65f3c10>
Aug  1 05:38:28 ustrlxpd15 kernel: CR2: 0000000000000000

Regards,
Sri

Sriram Varadhan
Oracle DBA

----------------------------------------------------------------------
This transmission, including attachments, is intended for the person or company named and may contain confidential and/or legally privileged information. Unauthorized disclosure, copying or use of this information may be unlawful and is prohibited. If you are not the intended recipient, please delete the original and all copies of this transmission and notify the sender.
_______________________________________________
suse-oracle mailing list
[email protected]
http://listx.novell.com/mailman/listinfo/suse-oracle