Kernel NULL pointer dereference when forwarding packets
George Ioannidis <[email protected]> Mon, 13 May 2013 04:28:50 +0200
| Newsgroups | gmane.network.routing.click |
|---|---|
| Message-ID | <[email protected]> |
Hello,
I am trying to run a simple Click kernel module which captures all
packets from interface *gbe0* and forwards them to the interface *gbe1*;
these are 10 Gigabit Intel Network cards.
My issue is that I get a NULL pointer dereference when I forward
packets, even with a very simple configuration.
Setup of *host A*:
*Click version*: 2.1 (latest from github), kernel module
*Configuration*: FromDevice(gbe0, PROMISC true) -> Queue(10000) ->
ToDevice(gbe1);
*kernel*: 3.2.0-39-generic x86_64 GNU/Linux.
*Distro*: Ubuntu 12.04.2 LTS
*Ethernet card*: [detailed in the *attachment*]
84:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit
SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Kernel driver in use: ixgbe
Kernel modules: ixgbe
*Test*: another machine, host B, is connected directly to host A.
Userspace click also runs on *host B* with the following configuration:
RandomSource( LENGTH 1514 ) -> ToDevice(gbe0);
So, the simple packet flow should be the following: Host_B [gbe0] ->
Host_A [gbe0] -> Host_A [gbe1]
*Kernel panic* produced at host A, 10-20 seconds after the experiment
starts: [full log in the *attachment*]
[ 1907.823186] BUG: unable to handle kernel NULL pointer dereference
at 0000000000000040
[ 1907.870136] IP: [<ffffffffa00f8b84>]
ixgbe_xmit_frame_ring+0x54/0x390 [ixgbe]
For 10-20 seconds, the experiment runs and packets are indeed forwarded
from gbe0 -> gbe1.
Do you have any ideas about what the problem could be?
Best Regards,
George
_______________________________________________
click mailing list
[email protected]
https://amsterdam.lcs.mit.edu/mailman/listinfo/click
click-panic.txt
(text/plain, 5 KB)
[ 1849.722810] click: starting router thread pid 2584 (ffff88081233ea00) [ 1907.823186] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040 [ 1907.870136] IP: [<ffffffffa00f8b84>] ixgbe_xmit_frame_ring+0x54/0x390 [ixgbe] [ 1907.912863] PGD 0 [ 1907.924955] Oops: 0000 [#1] SMP [ 1907.944384] CPU 1 [ 1907.955342] Modules linked in: click(O) proclikefs(O) binfmt_misc ext2 nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc sb_edac edac_core joydev wmi ipmi_si mac_hid ipmi_devintf ipmi_msghandler mei(C) lp parport ioatdma isci ixgbe libsas usbhid igb hid scsi_transport_sas mdio dca [ 1908.107134] [ 1908.116052] Pid: 2584, comm: kclick Tainted: G C O 3.2.0-39-generic #62-Ubuntu Intel Corporation S2600GZ/S2600GZ [ 1908.181229] RIP: 0010:[<ffffffffa00f8b84>] [<ffffffffa00f8b84>] ixgbe_xmit_frame_ring+0x54/0x390 [ixgbe] [ 1908.238485] RSP: 0018:ffff88081e69bd60 EFLAGS: 00010202 [ 1908.270231] RAX: 00000000000045e9 RBX: ffff8804162ea500 RCX: ffff880416313000 [ 1908.312876] RDX: 0000000000000000 RSI: ffff8808219a0780 RDI: 0000000000000000 [ 1908.355515] RBP: ffff88081e69bdb0 R08: 0000000000000001 R09: 0000000000010000 [ 1908.398154] R10: ffff88081ba40000 R11: ffff880422b2b090 R12: 0000000000000000 [ 1908.440799] R13: ffff8808219a0000 R14: ffff8808219a0780 R15: 000000000000dd86 [ 1908.483443] FS: 00007fc4b54f7700(0000) GS:ffff88042f620000(0000) knlGS:0000000000000000 [ 1908.531815] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1908.566160] CR2: 0000000000000040 CR3: 0000000001c05000 CR4: 00000000000406e0 [ 1908.608799] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1908.651444] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1908.694081] Process kclick (pid: 2584, threadinfo ffff88081e69a000, task ffff880822362e00) [ 1908.743489] Stack: [ 1908.755512] ffff88081e69bd70 ffffffff81313dbe ffff88081e69bda0 00ff88081e69be00 [ 1908.799976] 0000000000000000 ffff88081f872200 ffff8804162ea500 ffff8808219a0000 [ 1908.844446] 00000000fffffa52 ffff880822384000 ffff88081e69bdc0 ffffffffa00f8ee4 [ 1908.888922] Call Trace: [ 1908.903554] [<ffffffff81313dbe>] ? rb_erase+0xce/0x150 [ 1908.934779] [<ffffffffa00f8ee4>] ixgbe_xmit_frame+0x24/0x30 [ixgbe] [ 1908.972852] [<ffffffffa02fbec5>] _ZN8ToDevice12queue_packetEP6PacketP12netdev_queue+0x115/0x270 [click] [ 1909.029584] [<ffffffffa030c8c6>] ? _ZN13FullNoteQueue4pushEiP6Packet+0x16/0x330 [click] [ 1909.078008] [<ffffffffa02fc0ba>] _ZN8ToDevice8run_taskEP4Task+0x9a/0x2d0 [click] [ 1909.122794] [<ffffffffa02c2904>] _ZN12RouterThread6driverEv+0x374/0x5d0 [click] [ 1909.167079] [<ffffffffa0322c2c>] _ZL11click_schedPv+0x13c/0x240 [click] [ 1909.207146] [<ffffffff810571c5>] ? finish_task_switch+0xc5/0xf0 [ 1909.243058] [<ffffffff816680f4>] kernel_thread_helper+0x4/0x10 [ 1909.278487] [<ffffffffa0322af0>] ? _Z19click_cleanup_schedv+0x190/0x190 [click] [ 1909.322709] [<ffffffff816680f0>] ? gs_change+0x13/0x13 [ 1909.353932] Code: 00 00 49 89 d4 8b 97 d0 00 00 00 c6 45 cf 00 49 89 f6 48 89 fb 44 0f b7 7f 7e 05 ff 3f 00 00 2b 47 6c bf 00 00 00 00 0f b7 14 11 <41> 0f b7 4c 24 40 c1 e8 0e 8d 74 10 03 41 0f b7 54 24 42 41 0f [ 1909.470743] RIP [<ffffffffa00f8b84>] ixgbe_xmit_frame_ring+0x54/0x390 [ixgbe] [ 1909.513992] RSP <ffff88081e69bd60> [ 1909.534835] CR2: 0000000000000040 [ 1909.555032] ---[ end trace 5b2e0b234d34d82c ]--- [ 1909.583858] Kernel panic - not syncing: Fatal exception in interrupt [ 1909.621903] Pid: 2584, comm: kclick Tainted: G D C O 3.2.0-39-generic #62-Ubuntu [ 1909.669293] Call Trace: [ 1909.683982] [<ffffffff81644f38>] panic+0x91/0x1a4 [ 1909.712678] [<ffffffff8165eb5a>] oops_end+0xea/0xf0 [ 1909.742402] [<ffffffff81643de4>] no_context+0x150/0x15d [ 1909.774210] [<ffffffff81643fba>] __bad_area_nosemaphore+0x1c9/0x1e8 [ 1909.812241] [<ffffffff8130d466>] ? cpumask_next_and+0x36/0x50 [ 1909.847171] [<ffffffff8105b231>] ? find_busiest_group+0x171/0xbb0 [ 1909.884163] [<ffffffff81643fec>] bad_area_nosemaphore+0x13/0x15 [ 1909.920124] [<ffffffff81661766>] do_page_fault+0x426/0x520 [ 1909.953483] [<ffffffff81328a79>] ? swiotlb_map_page+0x59/0xf0 [ 1909.988406] [<ffffffff8165e0f5>] page_fault+0x25/0x30 [ 1910.019171] [<ffffffffa00f8b84>] ? ixgbe_xmit_frame_ring+0x54/0x390 [ixgbe] [ 1910.061350] [<ffffffff81313dbe>] ? rb_erase+0xce/0x150 [ 1910.092632] [<ffffffffa00f8ee4>] ixgbe_xmit_frame+0x24/0x30 [ixgbe] [ 1910.130717] [<ffffffffa02fbec5>] _ZN8ToDevice12queue_packetEP6PacketP12netdev_queue+0x115/0x270 [click] [ 1910.187504] [<ffffffffa030c8c6>] ? _ZN13FullNoteQueue4pushEiP6Packet+0x16/0x330 [click] [ 1910.235996] [<ffffffffa02fc0ba>] _ZN8ToDevice8run_taskEP4Task+0x9a/0x2d0 [click] [ 1910.280847] [<ffffffffa02c2904>] _ZN12RouterThread6driverEv+0x374/0x5d0 [click] [ 1910.325176] [<ffffffffa0322c2c>] _ZL11click_schedPv+0x13c/0x240 [click] [ 1910.365285] [<ffffffff810571c5>] ? finish_task_switch+0xc5/0xf0 [ 1910.401249] [<ffffffff816680f4>] kernel_thread_helper+0x4/0x10 [ 1910.438112] [<ffffffffa0322af0>] ? _Z19click_cleanup_schedv+0x190/0x190 [click] [ 1910.482401] [<ffffffff816680f0>] ? gs_change+0x13/0x13
ethernet-lspci-vvv.txt
(text/plain, 4.2 KB)
84:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 58
Region 0: Memory at 3c00ffc80000 (64-bit, prefetchable) [size=512K]
Region 2: I/O ports at c020 [size=32]
Region 4: Memory at 3c00ffd10000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 4s to 13s, TimeoutDis-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq+ ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 90-e2-12-8e-31-d2-17-42
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000ec300000 (64-bit, non-prefetchable)
Region 3: Memory at 00000000ec200000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe