Re: rtai task causes rcu problems on exit

I don't think it's the kernel. I think it's changes to RTAI. I'm going 
to need to spend some time with a simplified RTAI application to see if 
I can execute it hard real time without causing the problem.

Under RTAI 5.0-test with kernel 3.10.32 I get slightly different 
behavior, but essentially the same problem. The hard realtime task 
essentially hangs the system. If I run the exact same task with 
identical code using RTAI 4.0 on kernel 3.8.13, it works perfectly.



On 3/13/2016 20:59, JB wrote:
> Interesting, it sounds very very similar to what I'm seeing. 
> Everything works great after starting the task and synchronizing the 
> tasks/processes, but either the first time or the time after that, I 
> lose system responsiveness and get the warnings. So do I understand 
> you correctly in that you are saying we have a bug in our shutdown 
> sequence? It has worked flawlessly for years. Did something change in 
> how realtime is stopped? If so, what is the appropriate shutdown 
> sequence?
>
> When I have my system back I will look into that and try kernel 3.10.
>
> JB
>
>
> On 3/13/2016 17:24, Paolo Mantegazza wrote:
>> Perhaps, I've been able to reproduce your problem. It did not appear 
>> under heavy RTAI load. My PC was barely usable, but once at the 
>> closing of the tasks.
>> I tried a few other times but everything was OK. Since some of the 
>> many example I launched start-stop the hard timer, while a few other 
>> check if that has to be done or not, it is then likely that I closed 
>> one that stopped the timer., so that the remaining timed task began 
>> seeing overruns and looped without a pause, making the RCU messages 
>> appear.
>> I do not know if something like that could happen in you case also, 
>> just my try to help.
>>
>> Paolo
>>
>> ________________________________________
>> From: JB [[email protected]]
>> Sent: Sunday, March 13, 2016 9:40 PM
>> To: Paolo Mantegazza; [email protected]
>> Subject: Re: [Rtai] rtai task causes rcu problems on exit
>>
>> Indeed, I'll see what I can do and report back.
>>
>>
>> On 3/13/2016 14:19, Paolo Mantegazza wrote:
>>> BTW, RTAI 5 supports 3.10 also, which is closer (in time) to 3.8. 
>>> Maybe that the RCU diagnosis trouble you is not there yet.
>>> A nice chance to contribute a checking for it also.
>>> Paolo
>>>
>>> ________________________________________
>>> From: Rtai [[email protected]] on behalf of JB [[email protected]]
>>> Sent: Sunday, March 13, 2016 7:21 PM
>>> To: [email protected]
>>> Subject: Re: [Rtai] rtai task causes rcu problems on exit
>>>
>>> Thanks Paolo!  Unfortunately, it definitely has to do with the
>>> combination of upgrade to 3.18.22 and rtai-5.0-test1.
>>>
>>> Previously I was running RTAI 4.0 on a 3.8.13 kernel and prior to that
>>> it was earlier 3.x and 2.6.x and 2.4.x kernels. The real-time 
>>> processing
>>> components of the application have remained largely unchanged beyond 
>>> the
>>> minor changes to move off deprecated things (like better SMP support in
>>> the application when moving from 2.4 to 2.6 and migrating from rtai
>>> fifos to rtai mailboxes in later version of 2.6. Other applications 
>>> have
>>> always been very responsive! Just as soon as I tried to go to 3.18.22
>>> and rtai-5.0-test1 did this start happening.
>>>
>>> I had already read the documentation at the URL you gave me. However,
>>> while it does talk about settings to disable RCU, those settings appear
>>> to be ignored. I attempted to enable the suppression of RCU and the
>>> setting was completely ignored. The RTAI application we use is running
>>> at 2.4kHz and never overruns. It's nothing too aggressive and certainly
>>> shouldn't be causing this.
>>>
>>> I upgraded because I was getting a crash on some CPUs with rtai-4.0 and
>>> 3.8.13. However, I've spent quite a bit of time on trying to figure out
>>> what's wrong and without busting out all the kernel debugging and
>>> hacking tools and climbing into it's guts I'm unable to determine the
>>> cause. So, I think I'll go back to 3.8.13 and see if I can make that
>>> work again.
>>>
>>>
>>>
>>>
>>> On 3/13/2016 04:50, Paolo Mantegazza wrote:
>>>> My assumption is that your task steals too much time too Linux 
>>>> somewhere, at least to annoy the some read-copy-update stuff.
>>>> Somewhere on the net I found:
>>>> "You probably have a real time application that is consuming all 
>>>> cpu (some bad implementation) and because of its realtime 
>>>> scheduling priority the system doesn't have enough resources 
>>>> available for other tasks.". Nothe that (s)hewas not talking of 
>>>> hard real time but simple of the real time oprion available to a 
>>>> super user under standard Linux POSIX.
>>>> I do not know if I and the guy on the net are right but if your 
>>>> RTAI task is running fine, since RTAI very hardly locks out any 
>>>> LINUX activity what above might be even a truer justification for a 
>>>> CPU stall, in LINUX view. After all RTAI is a LINUX staller.
>>>> Maybe the following can help:
>>>> shttps://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
>>>> Paolo
>>>> ________________________________________
>>>> From: Rtai [[email protected]] on behalf of JB [[email protected]]
>>>> Sent: Sunday, March 13, 2016 4:02 AM
>>>> To: [email protected]
>>>> Subject: Re: [Rtai] rtai task causes rcu problems on exit
>>>>
>>>> Correction, they occur the entire time the real time task is 
>>>> running. Is
>>>> there a way to disable this somehow or configure it so it is 
>>>> ignored. It
>>>> is cause multi-second delays in the execution of other programs 
>>>> running
>>>> on the same system as the real-time task.
>>>>
>>>>
>>>> On 3/12/2016 19:52, JB wrote:
>>>>> Hi,
>>>>> I'm a little confused about some behavior I'm currently seeing after
>>>>> upgrading to kernel 3.18.22 and RTAI 5.0-test1. After my real-time
>>>>> task ends the first, time all seems fine. However, if I start it up a
>>>>> second time, then stop it, it doesn't stop for a while. After a 
>>>>> while,
>>>>> it stops and the messages below show up in the logs. Is RTAI
>>>>> conflicting in some way with this new RCU feature of the kernel?
>>>>>
>>>>>
>>>>> Mar 12 11:56:00 localhost kernel: [  249.450228] INFO: rcu_preempt
>>>>> detected stalls on CPUs/tasks: {} (detected by 0, t=21104 jiffies,
>>>>> g=11573, c=11572, q=30
>>>>> Mar 12 11:56:00 localhost kernel: [  249.450234] INFO: Stall ended
>>>>> before state dump start
>>>>> Mar 12 11:56:00 localhost kernel: INFO: rcu_preempt detected 
>>>>> stalls on
>>>>> CPUs/tasks: {} (detected by 0, t=21104 jiffies, g=11573, c=11572, 
>>>>> q=3068)
>>>>> Mar 12 11:56:00 localhost kernel: INFO: Stall ended before state dump
>>>>> start
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625505] INFO: rcu_preempt
>>>>> detected stalls on CPUs/tasks: { 1} (detected by 0, t=21524 jiffies,
>>>>> g=11575, c=11574, q=
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625511] Task dump for CPU 1:
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625512] swapper/1       R
>>>>> running task    12624     0      1 0x00200000
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625516] ffff880077417d00
>>>>> 0000000000000082 0000000000000000 0000000000000000
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625519] ffff880037096300
>>>>> ffff880079fcda00 00000000fffffffa ffffc90004704230
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625521] ffffffffa0170e80
>>>>> ffffffffa0170e80 0000000000000001 ffffffffa01b5708
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625524] Call Trace:
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625530] [<ffffffffa015ed27>]
>>>>> rt_timer_handler+0x477/0x7e0 [rtai_sched]
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625532] [<ffffffffa015da1a>]
>>>>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched]
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625536] [<ffffffff810d63f3>]
>>>>> __ipipe_dispatch_irq+0xd3/0x1b0
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625539] [<ffffffff81031503>]
>>>>> __ipipe_handle_irq+0x73/0x190
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625542] [<ffffffff816a16c0>]
>>>>> apic_timer_interrupt+0x60/0x90
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625544] [<ffffffff813192f3>]
>>>>> ? __this_cpu_preempt_check+0x13/0x20
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625593] [<ffffffffa180cd60>]
>>>>> ? _nv008584rm+0x90/0x3e0 [nvidia]
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625596] [<ffffffff810919d0>]
>>>>> ? rcu_eqs_enter_common.isra.43+0x60/0x120
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625598] [<ffffffff813192d7>]
>>>>> ? debug_smp_processor_id+0x17/0x20
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625601] [<ffffffff8100c540>]
>>>>> ? mwait_idle+0x60/0xa0
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625603] [<ffffffff8100d0ba>]
>>>>> arch_cpu_idle+0xa/0x10
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625606] [<ffffffff810826a9>]
>>>>> cpu_startup_entry+0x319/0x470
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625609] [<ffffffff810a5807>]
>>>>> ? clockevents_config_and_register+0x27/0x30
>>>>> Mar 12 11:56:22 localhost kernel: [  271.625611] [<ffffffff81032473>]
>>>>> start_secondary+0x143/0x150
>>>>> Mar 12 11:56:22 localhost kernel: INFO: rcu_preempt detected 
>>>>> stalls on
>>>>> CPUs/tasks: { 1} (detected by 0, t=21524 jiffies, g=11575, c=11574,
>>>>> q=1230)
>>>>> Mar 12 11:56:22 localhost kernel: Task dump for CPU 1:
>>>>> Mar 12 11:56:22 localhost kernel: swapper/1       R running task
>>>>> 12624     0      1 0x00200000
>>>>> Mar 12 11:56:22 localhost kernel: ffff880077417d00 0000000000000082
>>>>> 0000000000000000 0000000000000000
>>>>> Mar 12 11:56:22 localhost kernel: ffff880037096300 ffff880079fcda00
>>>>> 00000000fffffffa ffffc90004704230
>>>>> Mar 12 11:56:22 localhost kernel: ffffffffa0170e80 ffffffffa0170e80
>>>>> 0000000000000001 ffffffffa01b5708
>>>>> Mar 12 11:56:22 localhost kernel: Call Trace:
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015ed27>]
>>>>> rt_timer_handler+0x477/0x7e0 [rtai_sched]
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015da1a>]
>>>>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched]
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810d63f3>]
>>>>> __ipipe_dispatch_irq+0xd3/0x1b0
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff81031503>]
>>>>> __ipipe_handle_irq+0x73/0x190
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff816a16c0>]
>>>>> apic_timer_interrupt+0x60/0x90
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192f3>] ?
>>>>> __this_cpu_preempt_check+0x13/0x20
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa180cd60>] ?
>>>>> _nv008584rm+0x90/0x3e0 [nvidia]
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810919d0>] ?
>>>>> rcu_eqs_enter_common.isra.43+0x60/0x120
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192d7>] ?
>>>>> debug_smp_processor_id+0x17/0x20
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100c540>] ?
>>>>> mwait_idle+0x60/0xa0
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100d0ba>]
>>>>> arch_cpu_idle+0xa/0x10
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810826a9>]
>>>>> cpu_startup_entry+0x319/0x470
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810a5807>] ?
>>>>> clockevents_config_and_register+0x27/0x30
>>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff81032473>]
>>>>> start_secondary+0x143/0x150
>>>>>
>>>>> _______________________________________________
>>>>> Rtai mailing list
>>>>> [email protected]
>>>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>>>> _______________________________________________
>>>> Rtai mailing list
>>>> [email protected]
>>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>>> _______________________________________________
>>> Rtai mailing list
>>> [email protected]
>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>
> _______________________________________________
> Rtai mailing list
> [email protected]
> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai