Re: sched: problem with paging request on linux 3.16
PAUL-SOFTWARE <[email protected]> Wed, 10 May 2017 06:18:23 +0000
| Newsgroups | gmane.linux.real-time.rtai |
|---|---|
| Message-ID | <[email protected]> |
Hello Paolo, I patched sched.c and insmoded rtai_sched.ko with the parameters. The ops now is different: I-pipe: head domain RTAI registered. RTAI[hal]: mounted. ISOL_CPUS_MASK: 0. SYSINFO - # CPUs: 2, TIMER NAME: 'lapic', TIMER IRQ: 2305, TIMER FREQ: 99748999, CLOCK NAME: 'tsc', CLOCK FREQ: 3191981000, CPU FREQ: 3191981000, LINUX TIMER IRQ: 2305. RTAI[malloc]: global heap size = 2097152 bytes, <BSD>. , kstacks pool size = 524288 bytes. RTAI[sched]: hard timer type/freq = lapic/99748999(Hz); timing: ONESHOT; linear timed lists. RTAI[sched]: Linux timer freq = 1000 (Hz), TimeBase freq = 3191981000 hz. RTAI[sched]: timer setup = 66 ns, resched latency = 0 ns. RTAI[usi]: enabled. BUG: unable to handle kernel paging request at bf86091c IP: [<f878603d>] rtai_lxrt_invoke+0xaed/0x13d0 [rtai_sched] *pde = c2a4b067 Oops: 0001 [#1] PREEMPT SMP Modules linked in: rtai_sched(O+) rtai_hal(O) i915 drm_kms_helper drm i2c_algo_bit cfg80211 rfkill nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec coretemp snd_hwdep snd_pcsp kvm crc32_pclmul aesni_intel snd_pcm snd_timer snd aes_i586 xts lrw gf128mul ablk_helper cryptd soundcore mxm_wmi evdev serio_raw shpchp video thermal_sys tpm_tis tpm wmi button fuse parport_pc ppdev lp parport autofs4 ext4 crc16 mbcache jbd2 hid_generic usbhid sg sd_mod crc_t10dif crct10dif_generic crct10dif_common crc32c_intel r8169 8139cp mii xhci_hcd ahci libahci libata scsi_mod usbcore usb_common i2c_hid hid i2c_core CPU: 1 PID: 1291 Comm: calibrate Tainted: G O 3.16-rtai #15 Hardware name: System manufacturer System Product Name/H110M-C, BIOS 3402 04/06/2017 task: ea49b5c0 ti: f1a20000 task.ti: f1a20000 EIP: 0060:[<f878603d>] EFLAGS: 00010206 CPU: 1 EIP is at rtai_lxrt_invoke+0xaed/0x13d0 [rtai_sched] EAX: 0040780c EBX: 00000000 ECX: 00000407 EDX: bf86091c ESI: bf86091c EDI: 0040780c EBP: f1a21f84 ESP: f1a21f34 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 CR0: 8005003b CR2: bf86091c CR3: 33e85000 CR4: 00340790 I-pipe domain Linux Stack: 00000200 f77db840 c1095c3d 00000000 c16be840 c15cabc0 f1a21f98 00000001 fffffffe f1a21f64 c1095d8c 00000000 f1a21fac c11456a8 f1a21f98 eecf84c0 f3c27140 f1a21fb4 bf860878 b7583848 f1a21fa4 f8783fad 00000000 bf861f48 Call Trace: [<c1095c3d>] ? __call_rcu.constprop.58+0xcd/0x200 [<c1095d8c>] ? call_rcu+0x1c/0x20 [<c11456a8>] ? SyS_access+0x188/0x1e0 [<f8783fad>] ? lxrt_intercept_syscall+0xad/0x1b0 [rtai_sched] [<c10c65c4>] ? __ipipe_notify_syscall+0x14/0x30 [<c1457296>] ? pipeline_sysenter+0x7/0x1e Code: 10 8b 56 0c 89 54 24 0c 8b 56 08 89 54 24 08 8b 56 04 89 54 24 04 8b 16 89 14 24 ff 50 04 89 44 24 2c 89 54 24 30 e9 2f f6 ff ff <8b> 1a 8d 81 18 fc ff ff 83 f8 20 89 da 0f 87 9b 05 00 00 ff 24 EIP: [<f878603d>] rtai_lxrt_invoke+0xaed/0x13d0 [rtai_sched] SS:ESP 0068:f1a21f34 CR2: 00000000bf86091c ---[ end trace 019e450374b8e6f2 ]--- USERMODE CHECK: ERROR. USERMODE CHECK PROVIDED (ns): KernelLatency 0, UserLatency 0. FINAL CALIBRATION SUMMARY (ns): KernelLatency 3001, UserLatency 3001. Then I run the testsuite/kernel/latency: CPU USE SUMMARY # 0 -> 1075944 # 1 -> 0 # 2 -> 0 # 3 -> 0 # 4 -> 0 # 5 -> 0 # 6 -> 0 # 7 -> 0 END OF CPU USE SUMMARY Attached you find the output of the latency-run command. Thank you, Josef -----Ursprüngliche Nachricht----- Von: Paolo Mantegazza [mailto:[email protected]] Gesendet: Dienstag, 9. Mai 2017 19:08 An: PAUL-SOFTWARE; [email protected] Betreff: RE: [Rtai] sched: problem with paging request on linux 3.16 Sorry, I wanted to say: What you can still do is to comment out of sched.c the lines 2479 and 2480. With such a patch you should run the kernel latency test in SHOWROOM CVS, see what happens and report. Paolo. ________________________________________ From: Rtai [[email protected]] on behalf of Paolo Mantegazza [[email protected]] Sent: Tuesday, May 9, 2017 7:00 PM To: PAUL-SOFTWARE; [email protected] Subject: Re: [Rtai] sched: problem with paging request on linux 3.16 [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing] insmod ./rtai_sched.ko kernel_latency=3000 user_latency=3000, is the right way. Nonetheless I would have wished you got on it before releasing 5.0. I should not have missed a trivial commit I forgot. It should be OK in VULCANO CVS now. I have it clear that your trouble comes from the very first execution of whatever you'll run in user space. A thing I can reproduce if I keep the syscall auditing option in the kernel and have the related deamon installed. Yesterday night I tried for a cause the problem with no syscall auditing for a couple of hours without success. With what I asked you I wanted just to be sure and know either if the trouble comes when the existence of the calibration files is checked or when the actual calibration is carried out. What you can still do is to comment out use the insmod above and comment out of sched.c the lines 2479 and 2480, suing the above isnmod. With such a patch you should run the latency test in showroom, see what happens and report. Paolo ________________________________________ From: Rtai [[email protected]] on behalf of PAUL-SOFTWARE [[email protected]] Sent: Tuesday, May 9, 2017 5:41 PM To: [email protected] Subject: Re: [Rtai] sched: problem with paging request on linux 3.16 Hello Paolo, I use the new RTAI 5.0. When I do insmod ./rtai_sched.ko kernel_latency=3000 user_latency=3000 the parameters aren't known: rtai_sched: unknown parameter 'kernel_latency' ignored rtai_sched: unknown parameter 'user_latency' ignored And this doesn't help. Calibration is still done. Then I tried insmod ./rtai_sched.ko KernelLatency=3000 UserLatency=3000 This parameter are known but the calibration is still done and there is still the kernel ops: RTAI[hal]: mounted. ISOL_CPUS_MASK: 0. SYSINFO - # CPUs: 2, TIMER NAME: 'lapic', TIMER IRQ: 2305, TIMER FREQ: 99750000, CLOCK NAME: 'tsc', CLOCK FREQ: 3192012000, CPU FREQ: 3192012000, LINUX TIMER IRQ: 2305. RTAI[malloc]: global heap size = 2097152 bytes, <BSD>. , kstacks pool size = 524288 bytes. RTAI[sched]: hard timer type/freq = lapic/99750000(Hz); timing: ONESHOT; linear timed lists. RTAI[sched]: Linux timer freq = 1000 (Hz), TimeBase freq = 3192012000 hz. RTAI[sched]: timer setup = 66 ns, resched latency = 0 ns. RTAI[usi]: enabled. BUG: unable to handle kernel paging request at bf97c36c IP: [<f8795fed>] rtai_lxrt_invoke+0xaed/0x13d0 [rtai_sched] *pde = c2b0d067 Oops: 0001 [#1] PREEMPT SMP Modules linked in: rtai_sched(O+) rtai_hal(O) i915 drm_kms_helper drm i2c_algo_bit cfg80211 rfkill nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc snd_hda_codec_hdmi mxm_wmi video thermal_sys snd_hda_codec_realtek snd_hda_codec_generic coretemp serio_raw kvm evdev crc32_pclmul aesni_intel aes_i586 xts lrw gf128mul ablk_helper cryptd tpm_tis tpm shpchp snd_hda_intel snd_hda_controller snd_pcsp snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore button wmi fuse parport_pc ppdev lp parport autofs4 ext4 crc16 mbcache jbd2 hid_generic usbhid sg sd_mod crc_t10dif crct10dif_generic crct10dif_common crc32c_intel ahci libahci r8169 8139cp mii libata scsi_mod xhci_hcd usbcore usb_common i2c_hid hid i2c_core CPU: 1 PID: 1293 Comm: calibrate Tainted: G O 3.16-rtai #15 Hardware name: System manufacturer System Product Name/H110M-C, BIOS 3402 04/06/2017 task: f3c12a70 ti: f3d00000 task.ti: f3d00000 EIP: 0060:[<f8795fed>] EFLAGS: 00010206 CPU: 1 EIP is at rtai_lxrt_invoke+0xaed/0x13d0 [rtai_sched] EAX: 0040780c EBX: 00000000 ECX: 00000407 EDX: bf97c36c ESI: bf97c36c EDI: 0040780c EBP: f3d01f84 ESP: f3d01f34 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 CR0: 8005003b CR2: bf97c36c CR3: 2dd4b000 CR4: 00340790 I-pipe domain Linux Stack: 00000200 f77db840 c1095c3d 00000000 c16be840 c15cabc0 f3d01f98 00000001 fffffffe f3d01f64 c1095d8c 00000000 f3d01fac c11456a8 f3d01f98 f1f5b3c0 f6ea9240 f3d01fb4 bf97c2c8 b750e848 f3d01fa4 f8793f5d 00000000 bf97df48 Call Trace: [<c1095c3d>] ? __call_rcu.constprop.58+0xcd/0x200 [<c1095d8c>] ? call_rcu+0x1c/0x20 [<c11456a8>] ? SyS_access+0x188/0x1e0 [<f8793f5d>] ? lxrt_intercept_syscall+0xad/0x1b0 [rtai_sched] [<c10c65c4>] ? __ipipe_notify_syscall+0x14/0x30 [<c1457296>] ? pipeline_sysenter+0x7/0x1e Code: 10 8b 56 0c 89 54 24 0c 8b 56 08 89 54 24 08 8b 56 04 89 54 24 04 8b 16 89 14 24 ff 50 04 89 44 24 2c 89 54 24 30 e9 2f f6 ff ff <8b> 1a 8d 81 18 fc ff ff 83 f8 20 89 da 0f 87 9b 05 00 00 ff 24 EIP: [<f8795fed>] rtai_lxrt_invoke+0xaed/0x13d0 [rtai_sched] SS:ESP 0068:f3d01f34 CR2: 00000000bf97c36c ---[ end trace 4070b2700c0c5ca7 ]--- USERMODE CHECK: ERROR. USERMODE CHECK PROVIDED (ns): KernelLatency 940, UserLatency 940. FINAL CALIBRATION SUMMARY (ns): KernelLatency 940, UserLatency 940. For testing I switched off SMP at all. Same results. I have an Intel i3-6100T processor with 2 cores. Hyperthreading is disabled in the BIOS. This is cat /proc/interrupts CPU0 CPU1 0: 60 0 IO-APIC-edge timer 1: 269 0 IO-APIC-edge i8042 5: 0 0 IO-APIC-edge parport0 8: 1 0 IO-APIC-edge rtc0 9: 0 0 IO-APIC-fasteoi acpi 136: 146 0 PCI-MSI-edge xhci_hcd 137: 878 0 PCI-MSI-edge eth2 138: 14209 0 PCI-MSI-edge ahci 139: 427 0 PCI-MSI-edge snd_hda_intel NMI: 0 0 Non-maskable interrupts LOC: 18609 112842 Local timer interrupts SPU: 0 0 Spurious interrupts PMI: 0 0 Performance monitoring interrupts IWI: 0 0 IRQ work interrupts RTR: 0 0 APIC ICR read retries RES: 6648 6905 Rescheduling interrupts CAL: 360 6161 Function call interrupts TLB: 245 380 TLB shootdowns ERR: 0 MIS: 0 Josef Von: Paolo Mantegazza [mailto:[email protected]] Gesendet: Montag, 8. Mai 2017 17:31 An: PAUL-SOFTWARE; [email protected] Betreff: Re: [Rtai] sched: problem with paging request on linux 3.16 You could also try the test by inserting the needed modules manually, using "insmod ???/rtai_sched.ko kernel_latency= 3000 user_latency=3000", so to avoid the calibration. Paolo On 05/08/2017 12:59 PM, PAUL-SOFTWARE wrote: Hello Paolo, Using the calibration data from the running system doesn't help. I have tried RTAI from CVS and the new released RTAI 5.0 without success. Even when I use the same kernel settings you have posted a few days ago, the kernel ops with the second hardware still exists. The hal patch I use is identically to the one that was in VULCANO CVS. But the patch was removed. Do you know why? Unfortunately I can't use a newer kernel: The Igh Ethercat Master has only patched diver for network interfaces up to kernel 3.16. Josef Von: Paolo Mantegazza [mailto:[email protected]] Gesendet: Mittwoch, 3. Mai 2017 14:57 An: PAUL-SOFTWARE; [email protected]<mailto:[email protected]> Betreff: Re: [Rtai] sched: problem with paging request on linux 3.16 Have you configured your kernel for a UP setup? The latency calibration failed. you can try copying the file RTAIINSTALLDIR/calibration/latencies that succeeded in the new installation and see what happens. Moreover, I do not think that the patch you use is distributed with the test version you are using. Try with one in test2 tarball. All in all we are about to release the final 5.0, which is what is in VULCANO CVS. TRy it too. Paolo. On 05/03/2017 01:53 PM, PAUL-SOFTWARE wrote: Hello, I installed rtai-5.0-test2 on Debian Jessie with 32Bit kernel 3.16.7, using the hal-patch from magma (hal-linux-3.16.7-x86-5x.patch<https://github.com/ShabbyX/RTAI/blob/master/base/arch/x86/patches/hal-linux-3.16.7-x86-5x.patch>). Everything works fine on a board with Intel i3-4130 CPU. On a system with Intel i3-6110 CPU I get an error when I insert rtai_sched.ko. This is the kernel log when I start testsuite/kernel/latency: I-pipe: head domain RTAI registered. RTAI[hal]: mounted. ISOL_CPUS_MASK: 0. SYSINFO - # CPUs: 1, TIMER NAME: 'lapic', TIMER IRQ: 2305, TIMER FREQ: 99758000, CLOCK NAME: 'tsc', CLOCK FREQ: 3192285000, CPU FREQ: 3192285000, LINUX TIMER IRQ: 2305. RTAI[malloc]: global heap size = 2097152 bytes, <BSD>. , kstacks pool size = 524288 bytes. RTAI[sched]: hard timer type/freq = lapic/99758000(Hz); timing: oneshot; linear timed lists. RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq = 3192285000 hz.. RTAI[sched]: timer setup = 64 ns, resched latency = 0 ns. RTAI[usi]: enabled. BUG: unable to handle kernel paging request at bfebea1c IP: [<f86e2845>] rtai_lxrt_invoke+0xae5/0x13c0 [rtai_sched] *pde = 108de067 *pte = 10c15067 Oops: 0001 [#1] PREEMPT Modules linked in: rtai_sched(O+) rtai_hal(O) i915 drm_kms_helper drm i2c_algo_bit cfg80211 rfkill nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc snd_hda_codec_hdmi coretemp snd_hda_codec_realtek snd_hda_codec_generic kvm snd_hda_intel snd_hda_controller snd_hda_codec crc32_pclmul snd_hwdep aesni_intel snd_pcsp aes_i586 xts snd_pcm snd_timer lrw gf128mul snd soundcore ablk_helper cryptd mxm_wmi battery evdev serio_raw tpm_tis tpm shpchp wmi video button fuse parport_pc ppdev lp parport autofs4 ext4 crc16 mbcache jbd2 hid_generic usbhid sg sd_mod crc_t10dif crct10dif_generic crct10dif_common crc32c_intel ahci libahci xhci_hcd r8169 8139cp mii libata scsi_mod usbcore usb_common fan thermal_sys i2c_hid hid i2c_core CPU: 0 PID: 1346 Comm: calibrate Tainted: G O 3.16-rtai #8 Hardware name: System manufacturer System Product Name/H110M-C, BIOS 3402 04/06/2017 task: d08dc010 ti: d08f8000 task.ti: d08f8000 EIP: 0060:[<f86e2845>] EFLAGS: 00010206 CPU: 0 EIP is at rtai_lxrt_invoke+0xae5/0x13c0 [rtai_sched] EAX: 0040780c EBX: 00000000 ECX: 00000407 EDX: bfebea1c ESI: bfebea1c EDI: 0040780c EBP: d08f9f84 ESP: d08f9f34 DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 CR0: 8005003b CR2: bfebea1c CR3: 108dd000 CR4: 00340790 I-pipe domain Linux Stack: d08f9f58 c10863ee c15e4180 c1087745 d08f9f98 c15e4240 d08f9f98 00000001 fffffffe d08f9f64 c108789c 00000000 d08f9fac c1145b90 d08f9f98 ed94bc40 e19b3ac0 d08f9fb4 bfebe978 b74f6848 d08f9fa4 f86e09dd 00000000 bfebff48 Call Trace: [<c10863ee>] ? rcu_is_watching+0x1e/0x40 [<c1087745>] ? __call_rcu.constprop.60+0xa5/0x1e0 [<c108789c>] ? call_rcu+0x1c/0x20 [<c1145b90>] ? SyS_access+0x190/0x1f0 [<f86e09dd>] ? lxrt_intercept_syscall+0x8d/0x190 [rtai_sched] [<c10bdcd4>] ? __ipipe_notify_syscall+0x14/0x30 [<c1468115>] ? pipeline_sysenter+0x7/0x1e Code: 10 8b 56 0c 89 54 24 0c 8b 56 08 89 54 24 08 8b 56 04 89 54 24 04 8b 16 89 14 24 ff 50 04 89 44 24 2c 89 54 24 30 e9 37 f6 ff ff <8b> 1a 8d 81 18 fc ff ff 83 f8 20 89 da 0f 87 97 05 00 00 ff 24 EIP: [<f86e2845>] rtai_lxrt_invoke+0xae5/0x13c0 [rtai_sched] SS:ESP 0068:d08f9f34 CR2: 00000000bfebea1c ---[ end trace 426b8ee3f3f6e396 ]--- USERMODE CHECK: ERROR. USERMODE CHECK PROVIDED (ns): KernelLatency 0, UserLatency 0. FINAL CALIBRATION SUMMARY (ns): KernelLatency 1, UserLatency 1. Has anybody an idea what causes this ops and how I can avoid it? Best regards, Josef _______________________________________________ Rtai mailing list [email protected]<mailto:[email protected]> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai _______________________________________________ Rtai mailing list [email protected]<mailto:[email protected]> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai _______________________________________________ Rtai mailing list [email protected] https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai _______________________________________________ Rtai mailing list [email protected] https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
latency_run.txt
(text/plain, 8.6 KB)
* * * Type ^C to stop this application. * * RTAI Testsuite - KERNEL space latency test (output data in nanoseconds) *** latency verification tool with RTAI own real time kernel tasks *** *** period = 100000 (ns), avrgtime = 1 (s) *** RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns RTD| -2701| -2701| -2430| 5101| 5101| 0 RTD| -2639| -2701| -2436| -898| 5101| 0 RTD| -2667| -2701| -2440| -1219| 5101| 0 RTD| -2678| -2701| -2439| -1884| 5101| 0 RTD| -2659| -2701| -2441| -1968| 5101| 0 RTD| -2680| -2701| -2434| -78| 5101| 0 RTD| -2668| -2701| -2428| 111| 5101| 0 RTD| -2737| -2737| -2413| 1607| 5101| 0 RTD| -2708| -2737| -2438| -1467| 5101| 0 RTD| -2696| -2737| -2421| 5484| 5484| 0 RTD| -2684| -2737| -2436| 1348| 5484| 0 RTD| -2692| -2737| -2437| -979| 5484| 0 RTD| -2685| -2737| -2428| 1028| 5484| 0 RTD| -2654| -2737| -2439| -935| 5484| 0 RTD| -2674| -2737| -2439| -1736| 5484| 0 RTD| -2670| -2737| -2438| -1961| 5484| 0 RTD| -2668| -2737| -2440| -928| 5484| 0 RTD| -2671| -2737| -2438| 2243| 5484| 0 RTD| -2670| -2737| -2437| 2431| 5484| 0 RTD| -2665| -2737| -2420| -240| 5484| 0 RTD| -2702| -2737| -2440| -539| 5484| 0 RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns RTD| -2663| -2737| -2437| -906| 5484| 0 RTD| -2675| -2737| -2440| -761| 5484| 0 RTD| -2673| -2737| -2439| -2098| 5484| 0 RTD| -2668| -2737| -2441| -2050| 5484| 0 RTD| -2666| -2737| -2439| -1867| 5484| 0 RTD| -2667| -2737| -2439| -928| 5484| 0 RTD| -2650| -2737| -2438| -987| 5484| 0 RTD| -2674| -2737| -2440| -935| 5484| 0 RTD| -2661| -2737| -2438| -925| 5484| 0 RTD| -2670| -2737| -2440| -2014| 5484| 0 RTD| -2677| -2737| -2439| -1994| 5484| 0 RTD| -2684| -2737| -2440| -1880| 5484| 0 RTD| -2854| -2854| -2440| 1723| 5484| 0 RTD| -2675| -2854| -2440| -2013| 5484| 0 RTD| -2648| -2854| -2438| -900| 5484| 0 RTD| -2675| -2854| -2440| -2030| 5484| 0 RTD| -2657| -2854| -2440| -1870| 5484| 0 RTD| -2664| -2854| -2440| -899| 5484| 0 RTD| -2665| -2854| -2438| -901| 5484| 0 RTD| -2661| -2854| -2440| -1614| 5484| 0 RTD| -2639| -2854| -2437| -1839| 5484| 0 RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns RTD| -2697| -2854| -2440| -1758| 5484| 0 RTD| -2644| -2854| -2438| -926| 5484| 0 RTD| -2660| -2854| -2439| -1609| 5484| 0 RTD| -2667| -2854| -2439| -1613| 5484| 0 RTD| -2713| -2854| -2440| -899| 5484| 0 RTD| -2668| -2854| -2438| -904| 5484| 0 RTD| -2676| -2854| -2439| -907| 5484| 0 RTD| -2668| -2854| -2436| 1641| 5484| 0 RTD| -2672| -2854| -2439| -757| 5484| 0 RTD| -2664| -2854| -2438| -1961| 5484| 0 RTD| -2664| -2854| -2441| -897| 5484| 0 RTD| -2663| -2854| -2437| -1463| 5484| 0 RTD| -2661| -2854| -2441| 54| 5484| 0 RTD| -2674| -2854| -2438| 3705| 5484| 0 RTD| -2666| -2854| -2441| -1803| 5484| 0 RTD| -2699| -2854| -2437| -1466| 5484| 0 RTD| -2663| -2854| -2439| -1576| 5484| 0 RTD| -2697| -2854| -2439| 13| 5484| 0 RTD| -2664| -2854| -2440| -47| 5484| 0 RTD| -2659| -2854| -2438| -904| 5484| 0 RTD| -2674| -2854| -2438| -1619| 5484| 0 RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns RTD| -2713| -2854| -2437| 1583| 5484| 0 RTD| -2670| -2854| -2438| 1196| 5484| 0 RTD| -2662| -2854| -2437| -1034| 5484| 0 RTD| -2664| -2854| -2439| -317| 5484| 0 RTD| -2644| -2854| -2438| -924| 5484| 0 RTD| -2675| -2854| -2438| -942| 5484| 0 RTD| -2658| -2854| -2437| -935| 5484| 0 RTD| -2668| -2854| -2439| 1621| 5484| 0 RTD| -2670| -2854| -2437| -1210| 5484| 0 RTD| -2666| -2854| -2439| -860| 5484| 0 RTD| -2695| -2854| -2437| -868| 5484| 0 RTD| -2696| -2854| -2441| -1828| 5484| 0 RTD| -2679| -2854| -2437| -1450| 5484| 0 RTD| -2679| -2854| -2439| 930| 5484| 0 RTD| -2664| -2854| -2438| -891| 5484| 0 RTD| -2681| -2854| -2440| -1941| 5484| 0 RTD| -2669| -2854| -2438| -1857| 5484| 0 RTD| -2676| -2854| -2441| 4945| 5484| 0 RTD| -2696| -2854| -2432| 932| 5484| 0 RTD| -2678| -2854| -2426| 1403| 5484| 0 RTD| -2702| -2854| -2429| 5080| 5484| 0 RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns RTD| -2699| -2854| -2428| 1631| 5484| 0 RTD| -2672| -2854| -2435| -875| 5484| 0 RTD| -2674| -2854| -2436| 1091| 5484| 0 RTD| -2666| -2854| -2437| -1674| 5484| 0 RTD| -2663| -2854| -2439| -1795| 5484| 0 RTD| -2661| -2854| -2439| -1903| 5484| 0 RTD| -2667| -2854| -2440| -1788| 5484| 0 RTD| -2645| -2854| -2438| -280| 5484| 0 RTD| -2702| -2854| -2439| -895| 5484| 0 RTD| -2656| -2854| -2438| -554| 5484| 0 RTD| -2664| -2854| -2439| -1493| 5484| 0 RTD| -2664| -2854| -2437| -921| 5484| 0 RTD| -2663| -2854| -2439| -1694| 5484| 0 RTD| -2665| -2854| -2437| 2805| 5484| 0 RTD| -2699| -2854| -2439| 306| 5484| 0 RTD| -2651| -2854| -2438| -1883| 5484| 0 RTD| -2682| -2854| -2440| -909| 5484| 0 RTD| -2664| -2854| -2438| -572| 5484| 0 RTD| -2696| -2854| -2438| -173| 5484| 0 RTD| -2699| -2854| -2437| -1827| 5484| 0 RTD| -2705| -2854| -2434| -1709| 5484| 0 RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns RTD| -2676| -2854| -2394| 2884| 5484| 0 RTD| -2676| -2854| -2394| 2884| 5484| 0