Re: rt_task_wait_period hangs system

Before I'll access your system, let's see if there is any trivial problem left.

From your LInux config I see that there are 2 forgotten suggestions, as given in README.CONF_RMRKS.

One is that AUDITSYSCALL is enabled, which, since there seems to be no problem from user space, should not be the cause of our troubles.

The other one instead, i.e.: CONFIG_HAVE_CC_STACKPROTECTOR=y, CONFIG_CC_STACKPROTECTOR=y, CONFIG_CC_STACKPROTECTOR_STRONG=y, maybe.
Thus, disable all of the STACKPROTECTOR options, including the REGULAR one, recompile the kernel and retry. 
Then let me know.

If we'll still be in trouble, the I'll access your system.

Paolo

________________________________________
From: Rtai [[email protected]] on behalf of Jan Benda [[email protected]]
Sent: Wednesday, March 21, 2018 6:01 PM
To: [email protected]
Subject: Re: [Rtai] rt_task_wait_period hangs system

Thank you Paolo for the quick response!

On 03/21/2018 05:21 PM, Paolo Mantegazza wrote:
> First of all, if you can let me in your system I'm ready to take the
> burden of trying to understand what is happening.
> Even if in 20 years of RTAI development-usage I lost my system just
> three times. Nonetheless it would be better if  such a system can go
> ashtray without any trouble for anybody. I do not want to be hold
> responsible for anything.
> On your side you must be ready to be flooded by emails (or phone calls)
> to push the reset switch of your system.

I am ready and send you an email with the necessary details.

> I'm a bit curious to see why neither the calibration nor the kernel
> space latency test are running. The latter is a bit strange indeed,
> especially in view of the fact that 4.1 is OK. In fact the changes of
> 5.1 should affect user space, mostly.

This is my imrepssion as well - its strange. In particular given the
latency test without overruns.

> BTW, have you run also the kthread latency test? Let ime know about it
> too, please.

Attached is the kthreads test. This works as well. Only the switch test
does not report on the terminal (it complains about /var/log/messages
not being there - it is indeed not enabled), but the output of dmesg
shows the results (at end of the attached file).

> As a final comment I do not share the fact that max latencies in the
> range of 10-15 (us) are not great. To me they are indeed, to the point
> that of being a bit too optimistic. Likely taken for short time checks
> with Linux unloaded.

With "not great" I only meant that I know that with some effort I might
get them better. And you are right, they are taken on a machine without
load as a first check whether things are working. I am actually amazed
that on a first shot without any tweaking I got these latencies. This is
my general impression with rtai-5.1 that - in case it works - I get
already quite good latencies. First shot means that I start out with the
configuration of a matching lowlatency kernel, change a few kernel
configs, and run localmodconfig.

Jan

> To have them better I usually have to resort to CPU isolation, matched
> to a few other things I forgot.
>
> Paolo.
>
> On 03/21/2018 04:21 PM, Jan Benda wrote:
>> Hi,
>>
>> a follow up on Pauls issue:
>>
>> I installed rtai-5.1 with the hal-linux-4.4.115-x86-10.patch on
>> several intel i3, i5 and i7 machines. The results are mixed - some
>> work out of the box without tweaking (which is really cool) but others
>> hang up when insmoding rtai_sched - i.e. when it runs the calibration.
>> Loading rtai_sched with kernel_latency=10 user_latency=10, as
>> suggested, works always (no calibration).
>>
>> I modified the realtime/bin/rtai-load script such that it loads
>> rtai_sched without calibration:
>>
>> -- rtai-load   2018-03-20 18:32:15.317723122 +0100
>> +++ /home/benda/rtai-load       2018-03-21 15:53:59.619659835 +0100
>> @@ -158,7 +158,9 @@
>>       # one. If the latter fails too, assume the feature is built into
>>       # the scheduler or not needed. Obviously, we should _always_ find
>>       # the scheduler...
>> -     if test -r $rtai_moddir/rtai_$mod$modext; then
>> +     if test "x$mod" = "sched"; then
>> +        $sudo $insmod $rtai_moddir/rtai_$mod$modext kernel_latency=10
>> user_latency=10
>> +     elif test -r $rtai_moddir/rtai_$mod$modext; then
>>          test $verbose = 1 && echo "+ $insmod
>> $rtai_moddir/rtai_$mod$modext"
>>          $sudo $insmod $rtai_moddir/rtai_$mod$modext
>>         if test $? = 0 ; then
>>
>> With that I can run all user tests (see latencies* file for test
>> results (they are not great, but ok for a first shot) and infos on the
>> machine, config* is the kernel configuration).
>>
>> Note that on the same machine we had rtai 4.1 running without problems.
>>
>> However, when I run the kern latency test the machine hangs up - not
>> when insmoding rtai_sched but when executing the periodic task in
>> kernel space. So comedi is not to blame here.
>>
>> Hanging up means the LEDs on the keyboard blink, the computer is
>> inacessible, and I do not find any hints in the kernel log.
>>
>> I assume that the calibration feature in rtai_sched also runs a
>> periodic task.
>>
>> So something goes wrong with the period tasks.
>>
>> Any ideas?
>>
>> I also could provide access to the machines.
>>
>> Best,
>> Jan
>>
>>
>>
>> On 03/15/2018 12:46 PM, Paul Pfeiffer wrote:
>>> Dear RTAI community,
>>>
>>> after installation of RTAI 5.1 and running the latency tests, I am
>>> trying to run production code, which leads to a system crash.
>>> Debgugging lead me to conclude that it is the call to
>>> rt_task_wait_period, that produced the crash.
>>>
>>> System info:
>>> RTAi 5.1, Kernel 4.9.76, Intel i5-6600K
>>>
>>> In detail, the call happens in a custom kernel module, that first
>>> chooses between oneshot and periodic mode (both lead to the same crash)
>>>
>>> ```
>>> #ifdef ONESHOT_MODE
>>>    rt_set_oneshot_mode();
>>>    start_rt_timer(1);
>>> #else
>>>    rt_set_periodic_mode();
>>>    #endif
>>>
>>> ```
>>>
>>>   registers a real time task
>>>
>>> ```
>>> rt_linux_use_fpu( usesFPU );      /* declare if we use the FPU */
>>>
>>> retVal = rt_task_init( &dynClampTask.rtTask, dynclamp_loop, dummy,
>>> stackSize, priority, usesFPU, signal );
>>> ```
>>>
>>> and then makes it periodic
>>>
>>> ```
>>> rt_task_make_periodic( &dynClampTask.rtTask, rt_get_time() +
>>> periodTicks, periodTicks )
>>> ```
>>> Both of this calls are successful.
>>>
>>> In the task dynclamp_loop, the only calls to RTAI are
>>> ```
>>> rt_get_cpu_time_in_ns()
>>> ```
>>> which poses no problem and
>>>
>>> ```
>>> rt_task_wait_period()
>>> ```
>>> which kills the system.  This has worked in previous RTAI versions
>>> and on other machines, so we were wondering whether this a 5.1
>>> specific problem or related to our new machine? For completeness, I
>>> attach the source code of the kernel module, the functions of
>>> interest are dynclamp_loop, init_dynclamp_loop and
>>> cleanup_dynclamp_loop. Any ideas?
>>>
>>> Maybe related, I have recently been reporting my diffuculties
>>> installing RTAI 5.1, especially the system hig up when rtai_sched was
>>> loaded. Thanks to the help of Paolo, see thread "RTAI in debian 9",
>>> this problem was fixed via manual generation of a calibration file.
>>>
>>> Sorry, for the long question, but we are really eager to use the
>>> latest RTAI version and would like to understand this problem.
>>>
>>> Best, Paul
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Rtai mailing list
>>> [email protected]
>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>>>
>>
>>
>>
>>
>> _______________________________________________
>> Rtai mailing list
>> [email protected]
>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>
>


--
----------------------------------------------------------------------

Prof. Dr. Jan Benda
Abteilung Neuroethologie
Institut fuer Neurobiologie
Universitaet Tuebingen
Auf der Morgenstelle 28 E                 phone: +49 / 7071 - 29 74573
72076 Tuebingen                             fax: +49 / 7071 - 29 5144

----------------------------------------------------------------------
_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai