Re: RTAI in debian9
Mark Fortescue <[email protected]> Thu, 01 Mar 2018 14:34:00 +0000
| Newsgroups | gmane.linux.real-time.rtai |
|---|---|
| Message-ID | <[email protected]> |
Hi Paul and Paolo, One thing I found, while trying to handle system lock up caused by RTAI = issues in interrupt handlers, was that enabling the Linux serial console = and using a separate computer to log the serial output, allowed me to = locate a number of issues including unwanted video driver interactions = with RTAI. The video system usually locks up well before the serial subsystem = during a kernel crash/panic thus you loose the console output that = explains what went wrong. Regards Mark. On 01/03/18 13:09, Paul Pfeiffer wrote: > Yes, all tests are run as root. > > > On 03/01/2018 12:55 PM, Paolo Mantegazza wrote: >> The failing kern latency test does not tell me anything. Are you >> making it all as root? >> >> Paolo. >> >> On 03/01/2018 11:23 AM, Paul Pfeiffer wrote: >>> Thanks for the explanation. >>> >>> I cannot run the test from the root directory of the RTAI >>> installation, because the run script has relative references to >>> executables. But I could run the preempt test directly in the preempt >>> directory, because there was already a .runinfo file. The results are >>> attached. >>> >>> However, starting the Kernel latency test, crashed the machine... >>> maybe this is a hint towards the problem? >>> >>> I digged a little bit into the source code and found the >>> call_usermodehelper function in base/sched/sched.c in the calibrate >>> function. Am I right that this is a call to the executable defined in >>> /base/arch/x86/calibration/calibrate.c? Then, it seems that the crash >>> happens in the else branch of the main function, where no latencies >>> file is present. Important calls seem to be rt_thread_init, >>> rt_make_hard_real_time and rt_make_soft_real_time. Any idea what >>> could go wrong? >>> >>> Thank you, Paul >>> >>> ___ >>> >>> # Running tests >>> >>> dynclamp@abbott:~/Documents$ cd /usr/realtime/testsuite/ >>> >>> ## User test >>> >>> dynclamp@abbott:/usr/realtime/testsuite$ cd user/latency >>> dynclamp@abbott:/usr/realtime/testsuite/user/latency$ sudo ./run >>> [sudo] password for dynclamp: >>> >>> RTAI Testsuite - USER space latency test (output data in nanoseconds) >>> >>> *** latency verification tool with real time hardened user space >>> processes/threads *** >>> *** period =3D 100000 (ns), avrgtime =3D 1 (s) *** >>> >>> RTAI Testsuite - USER latency (all data in nanoseconds) >>> 2018/03/1 10:33:45 >>> RTH| lat min| ovl min| lat avg| lat max| ovl max| >>> overruns >>> RTD| -230| -230| 19| 12676| 12676| 0 >>> RTD| -168| -230| 35| 14333| 14333| 0 >>> RTD| -169| -230| 14| 5108| 14333| 0 >>> RTD| -180| -230| 16| 7970| 14333| 0 >>> RTD| -263| -263| 14| 4522| 14333| 0 >>> RTD| -141| -263| 20| 4838| 14333| 0 >>> RTD| -161| -263| 16| 3436| 14333| 0 >>> RTD| -144| -263| 19| 2030| 14333| 0 >>> RTD| -70| -263| 17| 2639| 14333| 0 >>> RTD| -73| -263| 41| 2836| 14333| 0 >>> RTD| -136| -263| 34| 3806| 14333| 0 >>> RTD| -161| -263| 29| 5334| 14333| 0 >>> RTD| -142| -263| 41| 4396| 14333| 0 >>> RTD| -38| -263| 59| 4724| 14333| 0 >>> RTD| -222| -263| 42| 3530| 14333| 0 >>> RTD| -162| -263| 30| 3017| 14333| 0 >>> RTD| -170| -263| 18| 2422| 14333| 0 >>> RTD| -133| -263| 18| 3717| 14333| 0 >>> RTD| -240| -263| 28| 5888| 14333| 0 >>> RTD| -102| -263| 41| 4356| 14333| 0 >>> RTD| -102| -263| 41| 4356| 14333| 0 >>> ^C >>> >>> S =3D 98.696, EXECTIME =3D 0.0107068 >>> rmmod: ERROR: Module rtai_msg is not currently loaded >>> rmmod: ERROR: Module rtai_mbx is not currently loaded >>> rmmod: ERROR: Module rtai_sem is not currently loaded >>> rmmod: ERROR: Module rtai_sched is not currently loaded >>> rmmod: ERROR: Module rtai_hal is not currently loaded >>> dynclamp@abbott:/usr/realtime/testsuite/user/latency$ cd ../preempt/ >>> dynclamp@abbott:/usr/realtime/testsuite/user/preempt$ sudo ./run >>> * >>> * >>> * Type ^C to stop this application. >>> * >>> * >>> RTAI Testsuite - LXRT preempt (all data in nanoseconds) >>> RTH| lat min| lat avg| lat max| jit fast| jit slow >>> RTD| -401| -127| 2382| 5393| 1792 >>> RTD| -406| -133| 2382| 5393| 1792 >>> RTD| -406| -131| 2382| 5393| 2016 >>> RTD| -406| -129| 3134| 5393| 3302 >>> RTD| -406| -127| 3623| 5393| 3302 >>> RTD| -406| -133| 3623| 5393| 3302 >>> RTD| -406| -132| 3623| 5393| 3302 >>> RTD| -406| -134| 3623| 5393| 4961 >>> RTD| -406| -134| 5743| 5393| 4961 >>> RTD| -406| -132| 5743| 5393| 4961 >>> RTD| -406| -136| 5743| 5393| 4961 >>> RTD| -406| -134| 5743| 5393| 4961 >>> RTD| -408| -126| 5743| 5393| 4961 >>> RTD| -408| -133| 5743| 5393| 4961 >>> RTD| -408| -131| 6401| 6292| 4961 >>> RTD| -408| -127| 6701| 8385| 4961 >>> RTD| -408| -121| 6701| 8385| 4961 >>> ^CRTD| -408| -121| 6701| 8385| 4961 >>> >>> ## Kernel tests >>> >>> cd ../../kern/latency >>> >>> sudo ./run >>> >>> CRASH >>> >>> >>> On 02/28/2018 09:46 PM, Paolo Mantegazza wrote: >>>> The third number is the assumed default period used for the >>>> calibration, if it succeeded and you did not set it to a different >>>> one when configuring RTAI. >>>> Clearly, when set by hands, it has little meaning, albeit being >>>> needed when rtai_sched is insmoded. >>>> >>>> It is strange that, once you ran latency, the preempt test do not >>>> run. Recall that you are assumed to run those tests from the RTAI >>>> install dir. Be sure of doing it in such a way and let me know. >>>> >>>> Now that you ran rtai_sched using the latencies file, the >>>> call_userhelper support seems to be working also on your system. So >>>> I've even less clues than before on the why it did not calibrate. In >>>> fact the automatic calibration is nothing but the latency test in >>>> disguise, coupled to a trivial Kalman filter to estimate the >>>> expected latency on the fly. >>>> >>>> Thanks for your feedback, Paolo. >>>> >>>> _______________________________________ >>>> From: Paul Pfeiffer [[email protected]] >>>> Sent: Wednesday, February 28, 2018 7:14 PM >>>> To: Paolo Mantegazza; [email protected] >>>> Subject: Re: [Rtai] RTAI in debian9 >>>> >>>> Thanks for the feedback. >>>> >>>> I added the latencies file and now the rtai_sched module can be loaded >>>> without specifiying user or kernel latency. In the kernel output, I see >>>> that the kernel latency is now 440ns and the user latency 880ns. >>>> What is >>>> the meaning of the third number? >>>> >>>> I tried running the other tests, but they do not work, because of a >>>> missing .runinfo file. Do I have to create this? What should be in this >>>> file? >>>> >>>> We will use the RTAI patched kernel to perform recordings from neurons >>>> with real time feedback loops. I will update you, when we have tried >>>> the >>>> new RTAI patch within this application. >>>> >>>> Best, Paul >>>> >>>> >>>> On 02/27/2018 10:59 PM, Paolo Mantegazza wrote: >>>>> Your system seems OK. If you edit a file, named "latencies", with a >>>>> single line containing: >>>>> 440 880 10000 >>>>> and copy it into "rtai_installdir/calibration/latencies. >>>>> you'll have what the calibration should have done and there will be >>>>> non need to set those latencies at rtai_sched.ko insmod. >>>>> In any case, once you have done what above, may test both latency >>>>> and preempt, in user, kern and kthread directories of yout >>>>> rtai_installdir and then report what happened? >>>>> Do not test switches, It locks the cpu for a bit of time and may be >>>>> risky. I'll not be surprised of a muted system. >>>>> As for why it does not work on your system, without a diagnose >>>>> hands on, I've no clue. >>>>> Moreover I'll be glad to hear of some further checks with a >>>>> production code of yours, if any. >>>>> >>>>> Thanks, Paolo >>>>> >>>>> ________________________________________the >>>>> From: Rtai [[email protected]] on behalf of Paul Pfeiffer >>>>> [[email protected]] >>>>> Sent: Tuesday, February 27, 2018 4:41 PM >>>>> To: [email protected] >>>>> Subject: Re: [Rtai] RTAI in debian9 >>>>> >>>>> Hi, >>>>> >>>>> thanks for the nice discussion in this thread. I have a similar >>>>> issue to >>>>> the one described by josu, but on a real machine. >>>>> >>>>> Setup: Linux kernel 4.9.76 and RTAI 5.1 >>>>> >>>>> For patching the kernel and compilation of the kernel modules, I >>>>> followed the instructions from Jan Benda, here >>>>> http://relacs.sourceforge.net/plugins/rtaicomedi/ >>>>> >>>>> Insertion of the rtai_hal module works, but insertion of rtai_sched >>>>> kills the system. >>>>> >>>>> Like Paolo suggested, I also tried loading the rtai_sched with >>>>> kernel_latency and user_latency set to 1000. This works. Does this >>>>> mean >>>>> that the calibration is the problem? How could I solve this for my >>>>> setup? >>>>> >>>>> Also, I ran the suggested modified latency test. However, like josu, I >>>>> had to change the insertion of the rtai_sched module to >>>>> >>>>> ``` >>>>> >>>>> insmod $MODULES/rtai_sched.ko kernel_latency=3D1 user_latency=3D1 >>>>> >>>>> ``` >>>>> >>>>> otherwise the system would stop responding. >>>>> >>>>> Please find below the output of the test and the corresponding kernel >>>>> messages. >>>>> >>>>> Thank you, Paul >>>>> >>>>> ________________ >>>>> >>>>> $ sudo ./testsuite/user/latency/run >>>>> >>>>> RTAI Testsuite - USER space latency test (output data in nanoseconds) >>>>> >>>>> *** latency verification tool with real time hardened user space >>>>> processes/threads *** >>>>> *** period =3D 100000 (ns), avrgtime =3D 1 (s) *** >>>>> >>>>> RTAI Testsuite - USER latency (all data in nanoseconds) >>>>> 2018/02/27 16:18:13 >>>>> RTH| lat min| ovl min| lat avg| lat max| ovl max| >>>>> overruns >>>>> RTD| 726| 726| 887| 5216| 5216| 0 >>>>> RTD| 705| 705| 884| 2667| 5216| 0 >>>>> RTD| 545| 545| 884| 3267| 5216| 0 >>>>> RTD| 599| 545| 890| 8122| 8122| 0 >>>>> RTD| 711| 545| 884| 15637| 15637| 0 >>>>> RTD| 638| 545| 882| 6058| 15637| 0 >>>>> RTD| 722| 545| 877| 3233| 15637| 0 >>>>> RTD| 741| 545| 881| 2186| 15637| 0 >>>>> RTD| 733| 545| 874| 3213| 15637| 0 >>>>> RTD| 734| 545| 874| 2666| 15637| 0 >>>>> RTD| 718| 545| 874| 1823| 15637| 0 >>>>> RTD| 729| 545| 874| 2813| 15637| 0 >>>>> RTD| 709| 545| 883| 5861| 15637| 0 >>>>> RTD| 722| 545| 897| 12125| 15637| 0 >>>>> RTD| 721| 545| 895| 3858| 15637| 0 >>>>> RTD| 598| 545| 891| 2384| 15637| 0 >>>>> RTD| 701| 545| 877| 2205| 15637| 0 >>>>> RTD| 642| 545| 878| 3895| 15637| 0 >>>>> RTD| 590| 545| 880| 1844| 15637| 0 >>>>> RTD| 725| 545| 877| 3841| 15637| 0 >>>>> RTD| 549| 545| 884| 2105| 15637| 0 >>>>> 2018/02/27 16:18:35 >>>>> RTH| lat min| ovl min| lat avg| lat max| ovl max| >>>>> overruns >>>>> RTD| 708| 545| 880| 3730| 15637| 0 >>>>> RTD| 717| 545| 875| 3596| 15637| 0 >>>>> RTD| 721| 545| 878| 1470| 15637| 0 >>>>> ^CRTD| 721| 545| 878| 1470| >>>>> 15637| 0 >>>>> >>>>> >>> S =3D 98.696, EXECTIME =3D 0.0105475 >>>>> >>>>> >>>>> $sudo dmesg -T -w >>>>> >>>>> [Tue Feb 27 16:18:12 2018] I-pipe: head domain RTAI registered. >>>>> [Tue Feb 27 16:18:12 2018] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0. >>>>> [Tue Feb 27 16:18:12 2018] SYSINFO - # CPUs: 4, TIMER NAME: 'lapic', >>>>> TIMER IRQ: 4354, TIMER FREQ: 437999993, CLOCK NAME: 'tsc', CLOCK FREQ: >>>>> 3500000000, CPU FREQ: 3500000000, LINUX TIMER IRQ: 4354. >>>>> [Tue Feb 27 16:18:12 2018] >>>>> REMARK: RTAI WILL NOT ACCESS USER >>>>> SPACE ON >>>>> STACKS ARGS ITS WAY. >>>>> >>>>> [Tue Feb 27 16:18:12 2018] RTAI[malloc]: global heap size =3D 2097152 >>>>> bytes, <BSD>. >>>>> [Tue Feb 27 16:18:12 2018] , kstacks pool size =3D 524288 bytes >>>>> [Tue Feb 27 16:18:12 2018] . >>>>> [Tue Feb 27 16:18:12 2018] RTAI[sched]: hard timer type/freq =3D >>>>> lapic/437999993(Hz); timing: ONESHOT; >>>>> [Tue Feb 27 16:18:12 2018] linear timed lists. >>>>> [Tue Feb 27 16:18:12 2018] RTAI[sched]: Linux timer freq =3D 250 (Hz), >>>>> TimeBase freq =3D 3500000000 hz. >>>>> [Tue Feb 27 16:18:12 2018] RTAI[sched]: timer setup =3D 66 ns, resched >>>>> latency =3D 0 ns. >>>>> [Tue Feb 27 16:18:12 2018] FINAL CALIBRATION SUMMARY (ns): >>>>> KernelLatency >>>>> 0, UserLatency 0. >>>>> [Tue Feb 27 16:18:39 2018] RTAI[malloc]: unloaded. >>>>> [Tue Feb 27 16:18:39 2018] RTAI[sched]: unloaded (forced >>>>> hard/soft/hard >>>>> transitions: traps 0, syscalls 0). >>>>> [Tue Feb 27 16:18:39 2018] I-pipe: head domain RTAI unregistered. >>>>> [Tue Feb 27 16:18:39 2018] RTAI[hal]: unmounted. >>>>> >>>>> >>>>> On 02/24/2018 12:20 AM, josu wrote: >>>>>> Yes, of course, I can=92t get real time (much less, 100 us period), >>>>>> only test the procedure. However, setting deadline to 1 (and not >>>>>> 0), the programs run, without hanging the VM. >>>>>> >>>>>> Although the meaning is not at all the same that in a real system, >>>>>> the latency test are able to run with your run script (setting a >>>>>> number greater than 0). This is the result: >>>>>> >>>>>> root@debianRB:/usr/realtime/testsuite/user/latency# ./run >>>>>> RTAI Testsuite - USER space latency test (output data in nanoseconds) >>>>>> >>>>>> *** latency verification tool with real time hardened user space >>>>>> processes/threads *** >>>>>> *** period =3D 100000 (ns), avrgtime =3D 1 (s) *** >>>>>> >>>>>> RTAI Testsuite - USER latency (all data in nanoseconds) >>>>>> 2018/02/23 22:47:06 >>>>>> RTH| lat min| ovl min| lat avg| lat max| ovl max| >>>>>> overruns >>>>>> RTD| 0| 0| 25068| 870103| >>>>>> 870103| 67 >>>>>> RTD| 0| 0| 25608| 1007163| >>>>>> 1007163| 95 >>>>>> RTD| 0| 0| 25288| 886190| >>>>>> 1007163| 124 >>>>>> RTD| 0| 0| 25275| 985017| >>>>>> 1007163| 155 >>>>>> RTD| 0| 0| 24564| 832423| >>>>>> 1007163| 179 >>>>>> RTD| 0| 0| 25620| 199742| >>>>>> 1007163| 194 >>>>>> RTD| 0| 0| 25393| 881237| >>>>>> 1007163| 226 >>>>>> RTD| 0| 0| 25067| 874862| >>>>>> 1007163| 253 >>>>>> RTD| 0| 0| 25247| 827829| >>>>>> 1007163| 274 >>>>>> RTD| 0| 0| 25566| 855516| >>>>>> 1007163| 310 >>>>>> RTD| 0| 0| 24899| 1035597| >>>>>> 1035597| 360 >>>>>> RTD| 0| 0| 25403| 202992| >>>>>> 1035597| 376 >>>>>> RTD| 0| 0| 25288| 771628| >>>>>> 1035597| 398 >>>>>> RTD| 0| 0| 25288| 771628| >>>>>> 1035597| 398 >>>>>> ^C >>>>>>>>> S =3D 98.696, EXECTIME =3D 0.0663255 >>>>>>> On 24 Feb 2018, at 00:11, Paolo Mantegazza >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>> VM changes the matter a lot. Anticipating the scheduling >>>>>>> deadline by 1 ns does not change anything with respect to 0. >>>>>>> In any case it served to indicate that you cannot run the RTAI >>>>>>> tests, simply because VM cannot provide true real time. In >>>>>>> particular it cannot support a 100 us period. >>>>>>> I dare saying that if you configure RTAI to use a lower >>>>>>> calibrating frequency, say 2000 Hz , in place of the default >>>>>>> 10000 Hz, you might succeed in trying the RTAI latency tests, >>>>>>> along with the hidden latency calibration. >>>>>>> In any case hard real time will likely be a no hope matter. >>>>>>> >>>>>>> Paolo >>>>>>> >>>>>>> ________________________________________ >>>>>>> From: josu [[email protected]] >>>>>>> Sent: Friday, February 23, 2018 10:57 PM >>>>>>> To: Paolo Mantegazza >>>>>>> Cc: RTAI RTAI >>>>>>> Subject: Re: [Rtai] RTAI in debian9 >>>>>>> >>>>>>> Hi again: >>>>>>> >>>>>>> First, I want to clarify than I am testing with a VM. Sorry for >>>>>>> the possible misunderstanding. I have been very busy and I have >>>>>>> forgot to say this important point. >>>>>>> >>>>>>> In any case, I have test your script and the system hangs. >>>>>>> However, making the next change: >>>>>>> >>>>>>> insmod $MODULES/rtai_sched.ko kernel_latency=3D1 user_latency=3D1 >>>>>>> >>>>>>> I have obtained the result attached below. >>>>>>> >>>>>>> josu >>>>>>> >>>>>>> [cid:[email protected]] >>>>>>> >>>>>>> On 23 Feb 2018, at 18:56, Paolo Mantegazza >>>>>>> <[email protected]<mailto:[email protected]>> >>>>>>> wrote: >>>>>>> >>>>>>> May you try the "testsuite/user/latency" test, in your RTAI >>>>>>> install dir, after saving and substituting the existing "run" >>>>>>> script with the one below and then let me know what happened? >>>>>>> Do it on an unloaded machined and, if possible, after seeing a >>>>>>> stabilized average latency, attach to your answer also a short >>>>>>> snapshot of what appears on the screen. >>>>>>> Thanks, Paolo >>>>>>> -------------------------------------------------------------------= ---------------------------------------- >>>>>>> >>>>>>> #!/bin/sh >>>>>>> >>>>>>> prefix=3D`rtai-config --prefix` >>>>>>> >>>>>>> if test "$prefix" =3D ""; then >>>>>>> echo "ERROR: please set your PATH variable to <rtai-install>/bin" >>>>>>> exit >>>>>>> fi >>>>>>> >>>>>>> MODULES=3D$prefix/modules >>>>>>> >>>>>>> libpath=3D`rtai-config --library-dir` >>>>>>> >>>>>>> if test "$libpath" =3D ""; then >>>>>>> echo "ERROR: please set your PATH variable to <rtai-install>/bin" >>>>>>> exit >>>>>>> fi >>>>>>> >>>>>>> export LD_LIBRARY_PATH=3D$libpath:$LD_LIBRARY_PATH >>>>>>> >>>>>>> cleanup () { >>>>>>> rmmod rtai_msg >>>>>>> rmmod rtai_mbx >>>>>>> rmmod rtai_sem >>>>>>> rmmod rtai_sched >>>>>>> rmmod rtai_hal >>>>>>> } >>>>>>> >>>>>>> trap cleanup INT ALRM TERM HUP KILL >>>>>>> >>>>>>> insmod $MODULES/rtai_hal.ko >>>>>>> insmod $MODULES/rtai_sched.ko kernel_latency=3D0 user_latency=3D0 >>>>>>> insmod $MODULES/rtai_sem.ko; >>>>>>> insmod $MODULES/rtai_mbx.ko; >>>>>>> insmod $MODULES/rtai_msg.ko; >>>>>>> >>>>>>> ./latency & >>>>>>> until(`ps -a | grep -q latency`); do sleep 1; done >>>>>>> ./display >>>>>>> >>>>>>> cleanup >>>>>>> -------------------------------------------------------------------= ---------------------------- >>>>>>> >>>>>>> >>>>>>> ________________________________________ >>>>>>> From: josu jugo [[email protected]<mailto:[email protected]>] >>>>>>> Sent: Friday, February 23, 2018 5:37 PM >>>>>>> To: Paolo Mantegazza >>>>>>> Cc: RTAI RTAI >>>>>>> Subject: Re: [Rtai] RTAI in debian9 >>>>>>> >>>>>>> Hi again: >>>>>>> >>>>>>> I have test a simple program obtained with rtai-lab (scicoslab >>>>>>> version), monitoring with qrtailab and the system is working. >>>>>>> >>>>>>> I have load rtai_sched with kernel_latency=3D1000 and >>>>>>> user_latency=3D1000 >>>>>>> >>>>>>> josu >>>>>>> >>>>>>> On 23 Feb 2018, at 12:51, josu jugo >>>>>>> <[email protected]<mailto:[email protected]><mailto:[email protected]= us>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> <Screen Shot 2018-02-23 at 22.47.37.png> >>>>>> _______________________________________________ >>>>>> Rtai mailing list >>>>>> [email protected] >>>>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai >>>>> _______________________________________________ >>>>> Rtai mailing list >>>>> [email protected] >>>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai >>> >> > > _______________________________________________ > Rtai mailing list > [email protected] > https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai _______________________________________________ Rtai mailing list [email protected] https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai