Re: BUG: unable to handle kernel paging request at 00007ffd7ce07800
Doug Renton <[email protected]> Wed, 23 Nov 2016 14:39:16 -0500
| Newsgroups | gmane.linux.real-time.rtai |
|---|---|
| Message-ID | <[email protected]> |
Thanks, I will try that, Just to be sure, CPU's refer to cores, not actual CPU's This is ONE physical CPU with 12 cores. I am 99% sure I am correct on this, as I have tried this on a two CPU Xeon system with 16 cores total. Thanks, Doug On 11/23/2016 9:20 AM, Paolo Mantegazza wrote: > I've seen the same problem in a few instances, where it was due to the > need of warming up large caches. > In your case there are a few things I find strange (unclear to me at > least): > - I do not see why there should be any overrun for an ovl max of 26094 > nanos, within a period of 100000 nanos; > - a likely unharmful page fault > - (not so strange) it is possible that RTAI cannot see more than 8 CPUs > correctly; you should try and see what happens with a number of cpus < > 11, e.g. up to 8 in step of 2. > Paolo > > On 11/23/2016 08:09 PM, Doug Renton wrote: >> I should add I'm running rtai-5.0-test1.tar.bz2 >> Have there been changes not yet included in this? Should I try the CVS >> version? >> >> On 11/23/2016 1:38 PM, Doug Renton wrote: >>> Hi All, >>> I'm trying to setup my nre computer from "Question about >>> CPU/Motherboard" >>> Running the latency test has some startup overruns, but seems ok after >>> that. I've run it for hours, and it is OK. dmesg shows a number of >>> errors. I've tried lots of various compile options, and this error >>> stays. >>> Also the calibrate test absolutely fails. The second dmesg is from >>> that. >>> Any ideas? >>> Thanks, >>> Doug >>> >>> >>> * >>> * >>> * Type ^C to stop this application. >>> * >>> * >>> RTAI Testsuite - KERNEL space latency test (output data in nanoseconds) >>> >>> *** latency verification tool with RTAI own real time kernel tasks *** >>> *** period = 100000 (ns), avrgtime = 1 (s) *** >>> >>> RTH| lat min| ovl min| lat avg| lat max| ovl max| >>> overruns >>> RTD| 510| 510| 1205| 15184| 15184| 2576 >>> RTD| 205| 205| 1213| 22673| 22673| 2576 >>> RTD| 258| 205| 1198| 20373| 22673| 2576 >>> RTD| 510| 205| 1196| 14014| 22673| 2576 >>> RTD| 255| 205| 1197| 15051| 22673| 2576 >>> RTD| 306| 205| 1195| 8734| 22673| 2576 >>> RTD| 510| 205| 1201| 14104| 22673| 2576 >>> RTD| 512| 205| 1209| 21560| 22673| 2576 >>> RTD| 362| 205| 1195| 23104| 23104| 2576 >>> RTD| 357| 205| 1201| 15228| 23104| 2576 >>> RTD| 221| 205| 1204| 26094| 26094| 2576 >>> ^CRTD| 221| 205| 1204| 26094| 26094| 2576 >>> >>> >>> >>> linux-36hs:/usr/realtime # dmesg -C >>> linux-36hs:/usr/realtime # dmesg >>> [ 842.342715] I-pipe: head domain RTAI registered. >>> [ 842.342719] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0. >>> [ 842.342720] SYSINFO - # CPUs: 12, TIMER NAME: 'lapic', TIMER IRQ: >>> 4353, TIMER FREQ: 6246289, CLOCK NAME: 'tsc', CLOCK FREQ: 3398323000, >>> CPU FREQ: 3398323000, LINUX TIMER IRQ: 4353. >>> [ 842.353665] RTAI[malloc]: global heap size = 4194304 bytes, <BSD>. >>> [ 842.365013] , kstacks pool size = 524288 bytes. >>> [ 842.365016] RTAI[sched]: hard timer type/freq = lapic/6246289(Hz); >>> timing: oneshot; linear timed lists. >>> [ 842.365018] RTAI[sched]: Linux timer freq = 1000 (Hz), TimeBase freq >>> = 3398323000 hz. >>> [ 842.365019] RTAI[sched]: timer setup = 160 ns, resched latency = 0 >>> ns. >>> [ 842.365476] BUG: unable to handle kernel paging request at >>> 00007ffd7ce07800 >>> [ 842.365986] IP: [<ffffffffa04e8631>] handle_lxrt_request+0x131/0x7d0 >>> [rtai_sched] >>> [ 842.367090] PGD 7c85d0067 PUD 7e9a7d067 PMD 7e6c4a067 PTE >>> 80000007e3613867 >>> [ 842.368221] Oops: 0001 [#4] PREEMPT SMP >>> [ 842.369361] Modules linked in: rtai_sched(O+) rtai_malloc(PO) >>> rtai_hal(O) fuse af_packet snd_hda_codec_hdmi snd_hda_codec_realtek >>> snd_hda_codec_generic xfs libcrc32c x86_pkg_temp_thermal coretemp kvm >>> iTCO_wdt iTCO_vendor_support mxm_wmi snd_hda_intel snd_hda_controller >>> snd_hda_codec snd_hwdep crct10dif_pclmul crc32_pclmul >>> ghash_clmulni_intel snd_pcm aesni_intel aes_x86_64 lrw gf128mul >>> glue_helper ablk_helper cryptd snd_timer serio_raw e1000e snd mei_me mei >>> xhci_pci i2c_i801 ptp xhci_hcd pps_core lpc_ich shpchp soundcore >>> mfd_core dm_mod btrfs xor raid6_pq nouveau crc32c_intel video >>> i2c_algo_bit drm_kms_helper ttm drm wmi sg [last unloaded: rtai_hal] >>> [ 842.373165] CPU: 3 PID: 2632 Comm: calibrate Tainted: P D O >>> 3.18.22-STRADM #9 >>> [ 842.374469] Hardware name: ASUS All Series/X99-A II, BIOS 0801 >>> 06/28/2016 >>> [ 842.375776] task: ffff8807e75982d0 ti: ffff8807c8c84000 task.ti: >>> ffff8807c8c84000 >>> [ 842.377073] RIP: 0010:[<ffffffffa04e8631>] [<ffffffffa04e8631>] >>> handle_lxrt_request+0x131/0x7d0 [rtai_sched] >>> [ 842.378397] RSP: 0018:ffff8807c8c87e78 EFLAGS: 00010206 >>> [ 842.379730] RAX: ffff8807c8c84000 RBX: 00007ffd7ce07800 RCX: >>> 0000000000000407 >>> [ 842.381076] RDX: 0000000000000000 RSI: 00007ffd7ce07800 RDI: >>> 0000000000407818 >>> [ 842.382429] RBP: ffff8807c8c87ec8 R08: 00007ffd7ce08f69 R09: >>> 00007ffd00000001 >>> [ 842.383766] R10: 00007ffd7ce077c8 R11: 0000000000000202 R12: >>> 0000000000000000 >>> [ 842.385102] R13: 0000000000407818 R14: 0000000000000000 R15: >>> 0000000000000000 >>> [ 842.386432] FS: 00007f49073ca700(0000) GS:ffff88081f2c0000(0000) >>> knlGS:0000000000000000 >>> [ 842.387762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 842.389087] CR2: 00007ffd7ce07800 CR3: 00000007f5679000 CR4: >>> 00000000003407e0 >>> [ 842.390421] DR0: 0000000000000000 DR1: 0000000000000000 DR2: >>> 0000000000000000 >>> [ 842.391759] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: >>> 0000000000000400 >>> [ 842.393081] I-pipe domain Linux >>> [ 842.394389] Stack: >>> [ 842.395694] 00000000000002a6 ffff880000000000 0000000000000000 >>> ffff8807e71a6260 >>> [ 842.396992] 00000000e71a6000 ffff8807e71a6248 00007ffd7ce07800 >>> 0000000000000000 >>> [ 842.398272] 0000000000407818 0000000000000000 ffff8807c8c87f38 >>> ffffffffa04e8ec0 >>> [ 842.399537] Call Trace: >>> [ 842.400804] [<ffffffffa04e8ec0>] rtai_lxrt_invoke+0x1f0/0x900 >>> [rtai_sched] >>> [ 842.402074] [<ffffffff810ea996>] ? __audit_syscall_exit+0x236/0x2e0 >>> [ 842.403334] [<ffffffffa04e72b5>] lxrt_intercept_syscall+0xc5/0x190 >>> [rtai_sched] >>> [ 842.404596] [<ffffffff81602a18>] ? >>> int_check_syscall_exit_work+0x34/0x3d >>> [ 842.405852] [<ffffffff811005d9>] __ipipe_notify_syscall+0x19/0x20 >>> [ 842.407065] [<ffffffff8160268e>] pipeline_syscall+0xa/0x17 >>> [ 842.408238] Code: 89 04 24 31 c0 41 ff 52 08 48 83 c4 30 5b 41 5c 41 >>> 5d 41 5e 5d c3 48 c7 c7 b8 16 4f a0 e8 e8 c3 2d 00 48 c7 c0 da ff ff ff >>> eb 96 <4c> 8b 2e 8d 91 18 fc ff ff 83 fa 20 4c 89 e8 0f 87 85 00 00 00 >>> [ 842.409455] RIP [<ffffffffa04e8631>] handle_lxrt_request+0x131/0x7d0 >>> [rtai_sched] >>> [ 842.410647] RSP <ffff8807c8c87e78> >>> [ 842.411800] CR2: 00007ffd7ce07800 >>> [ 842.420356] ---[ end trace a536dac0772dd8ef ]--- >>> [ 842.420410] USERMODE CHECK: ERROR. >>> [ 842.420412] USERMODE CHECK PROVIDED (ns): KernelLatency 0, >>> UserLatency 0. >>> [ 842.420413] FINAL CALIBRATION SUMMARY (ns): KernelLatency 0, >>> UserLatency 0. >>> [ 850.797255] >>> >>> CPU USE SUMMARY >>> [ 850.797259] # 0 -> 83615 >>> [ 850.797260] # 1 -> 0 >>> [ 850.797261] # 2 -> 0 >>> [ 850.797262] # 3 -> 0 >>> [ 850.797262] # 4 -> 0 >>> [ 850.797263] # 5 -> 0 >>> [ 850.797264] # 6 -> 0 >>> [ 850.797264] # 7 -> 0 >>> [ 850.797265] # 8 -> 0 >>> [ 850.797266] # 9 -> 0 >>> [ 850.797266] # 10 -> 0 >>> [ 850.797267] # 11 -> 0 >>> [ 850.797268] END OF CPU USE SUMMARY >>> >>> [ 851.287459] RTAI[sched]: unloaded (forced hard/soft/hard transitions: >>> traps 0, syscalls 0). >>> [ 851.293033] RTAI[malloc]: unloaded. >>> [ 851.298420] I-pipe: head domain RTAI unregistered. >>> [ 851.298430] RTAI[hal]: unmounted. >>> >>> >>> >>> >>> linux-36hs:/usr/realtime/calibration # ./run >>> * >>> * >>> * Type ^C to stop this application. >>> * >>> * >>> /usr/realtime/bin/rtai-load: line 185: 2873 Killed $suflag $* >>> linux-36hs:/usr/realtime/calibration # >>> >>> >>> linux-36hs:/usr/realtime # dmesg >>> [ 1114.495362] I-pipe: head domain RTAI registered. >>> [ 1114.495365] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0. >>> [ 1114.495367] SYSINFO - # CPUs: 12, TIMER NAME: 'lapic', TIMER IRQ: >>> 4353, TIMER FREQ: 6246289, CLOCK NAME: 'tsc', CLOCK FREQ: 3398323000, >>> CPU FREQ: 3398323000, LINUX TIMER IRQ: 4353. >>> [ 1114.509743] RTAI[malloc]: global heap size = 4194304 bytes, <BSD>. >>> [ 1114.520995] , kstacks pool size = 524288 bytes. >>> [ 1114.520998] RTAI[sched]: hard timer type/freq = lapic/6246289(Hz); >>> timing: oneshot; linear timed lists. >>> [ 1114.520999] RTAI[sched]: Linux timer freq = 1000 (Hz), TimeBase freq >>> = 3398323000 hz. >>> [ 1114.521000] RTAI[sched]: timer setup = 160 ns, resched latency = 0 >>> ns. >>> [ 1114.521417] BUG: unable to handle kernel paging request at >>> 00007ffd243033a0 >>> [ 1114.521925] IP: [<ffffffffa04e8631>] handle_lxrt_request+0x131/0x7d0 >>> [rtai_sched] >>> [ 1114.523053] PGD 7e6a93067 PUD 7c8c9b067 PMD 7e7964067 PTE >>> 80000007cec47867 >>> [ 1114.524195] Oops: 0001 [#7] PREEMPT SMP >>> [ 1114.525342] Modules linked in: rtai_sched(O+) rtai_malloc(PO) >>> rtai_hal(O) fuse af_packet snd_hda_codec_hdmi snd_hda_codec_realtek >>> snd_hda_codec_generic xfs libcrc32c x86_pkg_temp_thermal coretemp kvm >>> iTCO_wdt iTCO_vendor_support mxm_wmi snd_hda_intel snd_hda_controller >>> snd_hda_codec snd_hwdep crct10dif_pclmul crc32_pclmul >>> ghash_clmulni_intel snd_pcm aesni_intel aes_x86_64 lrw gf128mul >>> glue_helper ablk_helper cryptd snd_timer serio_raw e1000e snd mei_me mei >>> xhci_pci i2c_i801 ptp xhci_hcd pps_core lpc_ich shpchp soundcore >>> mfd_core dm_mod btrfs xor raid6_pq nouveau crc32c_intel video >>> i2c_algo_bit drm_kms_helper ttm drm wmi sg [last unloaded: rtai_hal] >>> [ 1114.529176] CPU: 11 PID: 2981 Comm: calibrate Tainted: P D O >>> 3.18.22-STRADM #9 >>> [ 1114.530488] Hardware name: ASUS All Series/X99-A II, BIOS 0801 >>> 06/28/2016 >>> [ 1114.531800] task: ffff8807e795e350 ti: ffff8807e7424000 task.ti: >>> ffff8807e7424000 >>> [ 1114.533111] RIP: 0010:[<ffffffffa04e8631>] [<ffffffffa04e8631>] >>> handle_lxrt_request+0x131/0x7d0 [rtai_sched] >>> [ 1114.534442] RSP: 0018:ffff8807e7427e78 EFLAGS: 00010206 >>> [ 1114.535790] RAX: ffff8807e7424000 RBX: 00007ffd243033a0 RCX: >>> 0000000000000407 >>> [ 1114.537134] RDX: 0000000000000000 RSI: 00007ffd243033a0 RDI: >>> 0000000000407818 >>> [ 1114.538486] RBP: ffff8807e7427ec8 R08: 00007ffd24303f69 R09: >>> 00007ffd00000001 >>> [ 1114.539832] R10: 00007ffd24303368 R11: 0000000000000202 R12: >>> 0000000000000000 >>> [ 1114.541171] R13: 0000000000407818 R14: 0000000000000000 R15: >>> 0000000000000000 >>> [ 1114.542506] FS: 00007f14aad8a700(0000) GS:ffff88081f4c0000(0000) >>> knlGS:0000000000000000 >>> [ 1114.543836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 1114.545169] CR2: 00007ffd243033a0 CR3: 00000007c851c000 CR4: >>> 00000000003407e0 >>> [ 1114.546504] DR0: 0000000000000000 DR1: 0000000000000000 DR2: >>> 0000000000000000 >>> [ 1114.547839] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: >>> 0000000000000400 >>> [ 1114.549165] I-pipe domain Linux >>> [ 1114.550476] Stack: >>> [ 1114.551791] 00000000000002a6 ffff880700000000 0000000000000000 >>> ffff8807f3a92e60 >>> [ 1114.553086] 00000000f3a92c00 ffff8807f3a92e48 00007ffd243033a0 >>> 0000000000000000 >>> [ 1114.554362] 0000000000407818 0000000000000000 ffff8807e7427f38 >>> ffffffffa04e8ec0 >>> [ 1114.555631] Call Trace: >>> [ 1114.556900] [<ffffffffa04e8ec0>] rtai_lxrt_invoke+0x1f0/0x900 >>> [rtai_sched] >>> [ 1114.558181] [<ffffffff810ea996>] ? __audit_syscall_exit+0x236/0x2e0 >>> [ 1114.559442] [<ffffffffa04e72b5>] lxrt_intercept_syscall+0xc5/0x190 >>> [rtai_sched] >>> [ 1114.560707] [<ffffffff81602a18>] ? >>> int_check_syscall_exit_work+0x34/0x3d >>> [ 1114.561970] [<ffffffff811005d9>] __ipipe_notify_syscall+0x19/0x20 >>> [ 1114.563188] [<ffffffff8160268e>] pipeline_syscall+0xa/0x17 >>> [ 1114.564365] Code: 89 04 24 31 c0 41 ff 52 08 48 83 c4 30 5b 41 5c 41 >>> 5d 41 5e 5d c3 48 c7 c7 b8 16 4f a0 e8 e8 c3 2d 00 48 c7 c0 da ff ff ff >>> eb 96 <4c> 8b 2e 8d 91 18 fc ff ff 83 fa 20 4c 89 e8 0f 87 85 00 00 00 >>> [ 1114.565584] RIP [<ffffffffa04e8631>] handle_lxrt_request+0x131/0x7d0 >>> [rtai_sched] >>> [ 1114.566781] RSP <ffff8807e7427e78> >>> [ 1114.567935] CR2: 00007ffd243033a0 >>> [ 1114.569088] ---[ end trace a536dac0772dd8f2 ]--- >>> [ 1114.569171] USERMODE CHECK: ERROR. >>> [ 1114.569174] USERMODE CHECK PROVIDED (ns): KernelLatency 0, >>> UserLatency 0. >>> [ 1114.569175] FINAL CALIBRATION SUMMARY (ns): KernelLatency 0, >>> UserLatency 0. >>> [ 1114.579349] BUG: unable to handle kernel paging request at >>> 00007ffdc26b67e0 >>> [ 1114.579842] IP: [<ffffffffa04e8631>] handle_lxrt_request+0x131/0x7d0 >>> [rtai_sched] >>> [ 1114.580985] PGD 7cb615067 PUD 7eb676067 PMD 7cb63a067 PTE >>> 80000000d0f69867 >>> >>> >>> [ 1114.582127] Oops: 0001 [#8] PREEMPT SMP >>> [ 1114.583281] Modules linked in: rtai_sched(O) rtai_malloc(PO) >>> rtai_hal(O) fuse af_packet snd_hda_codec_hdmi snd_hda_codec_realtek >>> snd_hda_codec_generic xfs libcrc32c x86_pkg_temp_thermal coretemp kvm >>> iTCO_wdt iTCO_vendor_support mxm_wmi snd_hda_intel snd_hda_controller >>> snd_hda_codec snd_hwdep crct10dif_pclmul crc32_pclmul >>> ghash_clmulni_intel snd_pcm aesni_intel aes_x86_64 lrw gf128mul >>> glue_helper ablk_helper cryptd snd_timer serio_raw e1000e snd mei_me mei >>> xhci_pci i2c_i801 ptp xhci_hcd pps_core lpc_ich shpchp soundcore >>> mfd_core dm_mod btrfs xor raid6_pq nouveau crc32c_intel video >>> i2c_algo_bit drm_kms_helper ttm drm wmi sg [last unloaded: rtai_hal] >>> [ 1114.587108] CPU: 11 PID: 2994 Comm: calibration_hel Tainted: P D >>> O 3.18.22-STRADM #9 >>> [ 1114.588427] Hardware name: ASUS All Series/X99-A II, BIOS 0801 >>> 06/28/2016 >>> [ 1114.589739] task: ffff8807eb716650 ti: ffff8807e6ad0000 task.ti: >>> ffff8807e6ad0000 >>> [ 1114.591049] RIP: 0010:[<ffffffffa04e8631>] [<ffffffffa04e8631>] >>> handle_lxrt_request+0x131/0x7d0 [rtai_sched] >>> [ 1114.592381] RSP: 0018:ffff8807e6ad3e78 EFLAGS: 00010202 >>> [ 1114.593726] RAX: ffff8807e6ad0000 RBX: 00007ffdc26b67e0 RCX: >>> 00000000000003ea >>> [ 1114.595070] RDX: 0000000000000000 RSI: 00007ffdc26b67e0 RDI: >>> 00000000003ea828 >>> [ 1114.596425] RBP: ffff8807e6ad3ec8 R08: 000000000000000f R09: >>> 0000000000000000 >>> [ 1114.597770] R10: 00007ffdc26b6798 R11: 0000000000003202 R12: >>> 0000000000000000 >>> [ 1114.599112] R13: 00000000003ea828 R14: 0000000000000000 R15: >>> 0000000000000000 >>> [ 1114.600443] FS: 00007f5b26ace700(0000) GS:ffff88081f4c0000(0000) >>> knlGS:0000000000000000 >>> [ 1114.601778] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>> [ 1114.603109] CR2: 00007ffdc26b67e0 CR3: 00000007cb6ef000 CR4: >>> 00000000003407e0 >>> [ 1114.604446] DR0: 0000000000000000 DR1: 0000000000000000 DR2: >>> 0000000000000000 >>> [ 1114.605782] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: >>> 0000000000000400 >>> [ 1114.607104] I-pipe domain Linux >>> [ 1114.608420] Stack: >>> [ 1114.609727] 0000000000000000 00007ffdc2797000 ffff8807e6ad3e98 >>> ffff8807f5679e60 >>> [ 1114.611023] 00000000f5679c00 ffff8807f5679e48 00007ffdc26b67e0 >>> 0000000000000000 >>> [ 1114.612300] 00000000003ea828 0000000000000000 ffff8807e6ad3f38 >>> ffffffffa04e8ec0 >>> [ 1114.613572] Call Trace: >>> [ 1114.614841] [<ffffffffa04e8ec0>] rtai_lxrt_invoke+0x1f0/0x900 >>> [rtai_sched] >>> [ 1114.616122] [<ffffffff810ea996>] ? __audit_syscall_exit+0x236/0x2e0 >>> [ 1114.617389] [<ffffffffa04e72b5>] lxrt_intercept_syscall+0xc5/0x190 >>> [rtai_sched] >>> [ 1114.618660] [<ffffffff81602a18>] ? >>> int_check_syscall_exit_work+0x34/0x3d >>> [ 1114.619921] [<ffffffff811005d9>] __ipipe_notify_syscall+0x19/0x20 >>> [ 1114.621143] [<ffffffff8160268e>] pipeline_syscall+0xa/0x17 >>> [ 1114.622320] Code: 89 04 24 31 c0 41 ff 52 08 48 83 c4 30 5b 41 5c 41 >>> 5d 41 5e 5d c3 48 c7 c7 b8 16 4f a0 e8 e8 c3 2d 00 48 c7 c0 da ff ff ff >>> eb 96 <4c> 8b 2e 8d 91 18 fc ff ff 83 fa 20 4c 89 e8 0f 87 85 00 00 00 >>> [ 1114.623540] RIP [<ffffffffa04e8631>] handle_lxrt_request+0x131/0x7d0 >>> [rtai_sched] >>> [ 1114.624736] RSP <ffff8807e6ad3e78> >>> [ 1114.625895] CR2: 00007ffdc26b67e0 >>> [ 1114.627043] ---[ end trace a536dac0772dd8f3 ]--- >>> [ 1114.970918] RTAI[sched]: unloaded (forced hard/soft/hard transitions: >>> traps 0, syscalls 0). >>> [ 1114.979641] RTAI[malloc]: unloaded. >>> [ 1114.985289] I-pipe: head domain RTAI unregistered. >>> [ 1114.985299] RTAI[hal]: unmounted. >>> linux-36hs:/usr/realtime # >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Rtai mailing list >>> [email protected] >>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai >>> >> _______________________________________________ >> Rtai mailing list >> [email protected] >> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai > > > _______________________________________________ Rtai mailing list [email protected] https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai