Re: rtai task causes rcu problems on exit

Thanks Paolo!  Unfortunately, it definitely has to do with the 
combination of upgrade to 3.18.22 and rtai-5.0-test1.

Previously I was running RTAI 4.0 on a 3.8.13 kernel and prior to that 
it was earlier 3.x and 2.6.x and 2.4.x kernels. The real-time processing 
components of the application have remained largely unchanged beyond the 
minor changes to move off deprecated things (like better SMP support in 
the application when moving from 2.4 to 2.6 and migrating from rtai 
fifos to rtai mailboxes in later version of 2.6. Other applications have 
always been very responsive! Just as soon as I tried to go to 3.18.22 
and rtai-5.0-test1 did this start happening.

I had already read the documentation at the URL you gave me. However, 
while it does talk about settings to disable RCU, those settings appear 
to be ignored. I attempted to enable the suppression of RCU and the 
setting was completely ignored. The RTAI application we use is running 
at 2.4kHz and never overruns. It's nothing too aggressive and certainly 
shouldn't be causing this.

I upgraded because I was getting a crash on some CPUs with rtai-4.0 and 
3.8.13. However, I've spent quite a bit of time on trying to figure out 
what's wrong and without busting out all the kernel debugging and 
hacking tools and climbing into it's guts I'm unable to determine the 
cause. So, I think I'll go back to 3.8.13 and see if I can make that 
work again.




On 3/13/2016 04:50, Paolo Mantegazza wrote:
> My assumption is that your task steals too much time too Linux somewhere, at least to annoy the some read-copy-update stuff.
> Somewhere on the net I found:
> "You probably have a real time application that is consuming all cpu (some bad implementation) and because of its realtime scheduling priority the system doesn't have enough resources available for other tasks.". Nothe that (s)hewas not talking of hard real time but simple of the real time oprion available to a super user under standard Linux POSIX.
> I do not know if I and the guy on the net are right but if your RTAI task is running fine, since RTAI very hardly locks out any LINUX activity what above might be even a truer justification for a CPU stall, in LINUX view. After all RTAI is a LINUX staller.
> Maybe the following can help:
> shttps://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
> Paolo
> ________________________________________
> From: Rtai [[email protected]] on behalf of JB [[email protected]]
> Sent: Sunday, March 13, 2016 4:02 AM
> To: [email protected]
> Subject: Re: [Rtai] rtai task causes rcu problems on exit
>
> Correction, they occur the entire time the real time task is running. Is
> there a way to disable this somehow or configure it so it is ignored. It
> is cause multi-second delays in the execution of other programs running
> on the same system as the real-time task.
>
>
> On 3/12/2016 19:52, JB wrote:
>> Hi,
>> I'm a little confused about some behavior I'm currently seeing after
>> upgrading to kernel 3.18.22 and RTAI 5.0-test1. After my real-time
>> task ends the first, time all seems fine. However, if I start it up a
>> second time, then stop it, it doesn't stop for a while. After a while,
>> it stops and the messages below show up in the logs. Is RTAI
>> conflicting in some way with this new RCU feature of the kernel?
>>
>>
>> Mar 12 11:56:00 localhost kernel: [  249.450228] INFO: rcu_preempt
>> detected stalls on CPUs/tasks: {} (detected by 0, t=21104 jiffies,
>> g=11573, c=11572, q=30
>> Mar 12 11:56:00 localhost kernel: [  249.450234] INFO: Stall ended
>> before state dump start
>> Mar 12 11:56:00 localhost kernel: INFO: rcu_preempt detected stalls on
>> CPUs/tasks: {} (detected by 0, t=21104 jiffies, g=11573, c=11572, q=3068)
>> Mar 12 11:56:00 localhost kernel: INFO: Stall ended before state dump
>> start
>> Mar 12 11:56:22 localhost kernel: [  271.625505] INFO: rcu_preempt
>> detected stalls on CPUs/tasks: { 1} (detected by 0, t=21524 jiffies,
>> g=11575, c=11574, q=
>> Mar 12 11:56:22 localhost kernel: [  271.625511] Task dump for CPU 1:
>> Mar 12 11:56:22 localhost kernel: [  271.625512] swapper/1       R
>> running task    12624     0      1 0x00200000
>> Mar 12 11:56:22 localhost kernel: [  271.625516]  ffff880077417d00
>> 0000000000000082 0000000000000000 0000000000000000
>> Mar 12 11:56:22 localhost kernel: [  271.625519]  ffff880037096300
>> ffff880079fcda00 00000000fffffffa ffffc90004704230
>> Mar 12 11:56:22 localhost kernel: [  271.625521]  ffffffffa0170e80
>> ffffffffa0170e80 0000000000000001 ffffffffa01b5708
>> Mar 12 11:56:22 localhost kernel: [  271.625524] Call Trace:
>> Mar 12 11:56:22 localhost kernel: [  271.625530] [<ffffffffa015ed27>]
>> rt_timer_handler+0x477/0x7e0 [rtai_sched]
>> Mar 12 11:56:22 localhost kernel: [  271.625532] [<ffffffffa015da1a>]
>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched]
>> Mar 12 11:56:22 localhost kernel: [  271.625536] [<ffffffff810d63f3>]
>> __ipipe_dispatch_irq+0xd3/0x1b0
>> Mar 12 11:56:22 localhost kernel: [  271.625539] [<ffffffff81031503>]
>> __ipipe_handle_irq+0x73/0x190
>> Mar 12 11:56:22 localhost kernel: [  271.625542] [<ffffffff816a16c0>]
>> apic_timer_interrupt+0x60/0x90
>> Mar 12 11:56:22 localhost kernel: [  271.625544] [<ffffffff813192f3>]
>> ? __this_cpu_preempt_check+0x13/0x20
>> Mar 12 11:56:22 localhost kernel: [  271.625593] [<ffffffffa180cd60>]
>> ? _nv008584rm+0x90/0x3e0 [nvidia]
>> Mar 12 11:56:22 localhost kernel: [  271.625596] [<ffffffff810919d0>]
>> ? rcu_eqs_enter_common.isra.43+0x60/0x120
>> Mar 12 11:56:22 localhost kernel: [  271.625598] [<ffffffff813192d7>]
>> ? debug_smp_processor_id+0x17/0x20
>> Mar 12 11:56:22 localhost kernel: [  271.625601] [<ffffffff8100c540>]
>> ? mwait_idle+0x60/0xa0
>> Mar 12 11:56:22 localhost kernel: [  271.625603] [<ffffffff8100d0ba>]
>> arch_cpu_idle+0xa/0x10
>> Mar 12 11:56:22 localhost kernel: [  271.625606] [<ffffffff810826a9>]
>> cpu_startup_entry+0x319/0x470
>> Mar 12 11:56:22 localhost kernel: [  271.625609] [<ffffffff810a5807>]
>> ? clockevents_config_and_register+0x27/0x30
>> Mar 12 11:56:22 localhost kernel: [  271.625611] [<ffffffff81032473>]
>> start_secondary+0x143/0x150
>> Mar 12 11:56:22 localhost kernel: INFO: rcu_preempt detected stalls on
>> CPUs/tasks: { 1} (detected by 0, t=21524 jiffies, g=11575, c=11574,
>> q=1230)
>> Mar 12 11:56:22 localhost kernel: Task dump for CPU 1:
>> Mar 12 11:56:22 localhost kernel: swapper/1       R  running task
>> 12624     0      1 0x00200000
>> Mar 12 11:56:22 localhost kernel: ffff880077417d00 0000000000000082
>> 0000000000000000 0000000000000000
>> Mar 12 11:56:22 localhost kernel: ffff880037096300 ffff880079fcda00
>> 00000000fffffffa ffffc90004704230
>> Mar 12 11:56:22 localhost kernel: ffffffffa0170e80 ffffffffa0170e80
>> 0000000000000001 ffffffffa01b5708
>> Mar 12 11:56:22 localhost kernel: Call Trace:
>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015ed27>]
>> rt_timer_handler+0x477/0x7e0 [rtai_sched]
>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015da1a>]
>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched]
>> Mar 12 11:56:22 localhost kernel: [<ffffffff810d63f3>]
>> __ipipe_dispatch_irq+0xd3/0x1b0
>> Mar 12 11:56:22 localhost kernel: [<ffffffff81031503>]
>> __ipipe_handle_irq+0x73/0x190
>> Mar 12 11:56:22 localhost kernel: [<ffffffff816a16c0>]
>> apic_timer_interrupt+0x60/0x90
>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192f3>] ?
>> __this_cpu_preempt_check+0x13/0x20
>> Mar 12 11:56:22 localhost kernel: [<ffffffffa180cd60>] ?
>> _nv008584rm+0x90/0x3e0 [nvidia]
>> Mar 12 11:56:22 localhost kernel: [<ffffffff810919d0>] ?
>> rcu_eqs_enter_common.isra.43+0x60/0x120
>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192d7>] ?
>> debug_smp_processor_id+0x17/0x20
>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100c540>] ?
>> mwait_idle+0x60/0xa0
>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100d0ba>]
>> arch_cpu_idle+0xa/0x10
>> Mar 12 11:56:22 localhost kernel: [<ffffffff810826a9>]
>> cpu_startup_entry+0x319/0x470
>> Mar 12 11:56:22 localhost kernel: [<ffffffff810a5807>] ?
>> clockevents_config_and_register+0x27/0x30
>> Mar 12 11:56:22 localhost kernel: [<ffffffff81032473>]
>> start_secondary+0x143/0x150
>>
>> _______________________________________________
>> Rtai mailing list
>> [email protected]
>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> _______________________________________________
> Rtai mailing list
> [email protected]
> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai