Re: rtai task causes rcu problems on exit

RTAI 5 has had enhancements but  I do not see anything that should change its former behavior.  The main change is we left the legacy mode, which makes things better and easier in my view/hope.
Recall that you have the possibility of checking how much RTAI tasks consume, through "cat /proc/rtai/scheduler". Can you cross check the figures under 4 and 5 and see if there is any difference, letting me know.
Generally speaking my experience shows that RTAI can consume up to 70%  of the computational power available without loosing Linux responsiveness.  I've conducted a long series of stress test, running many examples together  without troubles.
In fact I'm somewhat helpless because it is an unknown problem for me. 
Paolo

________________________________________
From: Rtai [[email protected]] on behalf of JB [[email protected]]
Sent: Sunday, March 13, 2016 7:21 PM
To: [email protected]
Subject: Re: [Rtai] rtai task causes rcu problems on exit

Thanks Paolo!  Unfortunately, it definitely has to do with the
combination of upgrade to 3.18.22 and rtai-5.0-test1.

Previously I was running RTAI 4.0 on a 3.8.13 kernel and prior to that
it was earlier 3.x and 2.6.x and 2.4.x kernels. The real-time processing
components of the application have remained largely unchanged beyond the
minor changes to move off deprecated things (like better SMP support in
the application when moving from 2.4 to 2.6 and migrating from rtai
fifos to rtai mailboxes in later version of 2.6. Other applications have
always been very responsive! Just as soon as I tried to go to 3.18.22
and rtai-5.0-test1 did this start happening.

I had already read the documentation at the URL you gave me. However,
while it does talk about settings to disable RCU, those settings appear
to be ignored. I attempted to enable the suppression of RCU and the
setting was completely ignored. The RTAI application we use is running
at 2.4kHz and never overruns. It's nothing too aggressive and certainly
shouldn't be causing this.

I upgraded because I was getting a crash on some CPUs with rtai-4.0 and
3.8.13. However, I've spent quite a bit of time on trying to figure out
what's wrong and without busting out all the kernel debugging and
hacking tools and climbing into it's guts I'm unable to determine the
cause. So, I think I'll go back to 3.8.13 and see if I can make that
work again.




On 3/13/2016 04:50, Paolo Mantegazza wrote:
> My assumption is that your task steals too much time too Linux somewhere, at least to annoy the some read-copy-update stuff.
> Somewhere on the net I found:
> "You probably have a real time application that is consuming all cpu (some bad implementation) and because of its realtime scheduling priority the system doesn't have enough resources available for other tasks.". Nothe that (s)hewas not talking of hard real time but simple of the real time oprion available to a super user under standard Linux POSIX.
> I do not know if I and the guy on the net are right but if your RTAI task is running fine, since RTAI very hardly locks out any LINUX activity what above might be even a truer justification for a CPU stall, in LINUX view. After all RTAI is a LINUX staller.
> Maybe the following can help:
> shttps://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
> Paolo
> ________________________________________
> From: Rtai [[email protected]] on behalf of JB [[email protected]]
> Sent: Sunday, March 13, 2016 4:02 AM
> To: [email protected]
> Subject: Re: [Rtai] rtai task causes rcu problems on exit
>
> Correction, they occur the entire time the real time task is running. Is
> there a way to disable this somehow or configure it so it is ignored. It
> is cause multi-second delays in the execution of other programs running
> on the same system as the real-time task.
>
>
> On 3/12/2016 19:52, JB wrote:
>> Hi,
>> I'm a little confused about some behavior I'm currently seeing after
>> upgrading to kernel 3.18.22 and RTAI 5.0-test1. After my real-time
>> task ends the first, time all seems fine. However, if I start it up a
>> second time, then stop it, it doesn't stop for a while. After a while,
>> it stops and the messages below show up in the logs. Is RTAI
>> conflicting in some way with this new RCU feature of the kernel?
>>
>>
>> Mar 12 11:56:00 localhost kernel: [  249.450228] INFO: rcu_preempt
>> detected stalls on CPUs/tasks: {} (detected by 0, t=21104 jiffies,
>> g=11573, c=11572, q=30
>> Mar 12 11:56:00 localhost kernel: [  249.450234] INFO: Stall ended
>> before state dump start
>> Mar 12 11:56:00 localhost kernel: INFO: rcu_preempt detected stalls on
>> CPUs/tasks: {} (detected by 0, t=21104 jiffies, g=11573, c=11572, q=3068)
>> Mar 12 11:56:00 localhost kernel: INFO: Stall ended before state dump
>> start
>> Mar 12 11:56:22 localhost kernel: [  271.625505] INFO: rcu_preempt
>> detected stalls on CPUs/tasks: { 1} (detected by 0, t=21524 jiffies,
>> g=11575, c=11574, q=
>> Mar 12 11:56:22 localhost kernel: [  271.625511] Task dump for CPU 1:
>> Mar 12 11:56:22 localhost kernel: [  271.625512] swapper/1       R
>> running task    12624     0      1 0x00200000
>> Mar 12 11:56:22 localhost kernel: [  271.625516]  ffff880077417d00
>> 0000000000000082 0000000000000000 0000000000000000
>> Mar 12 11:56:22 localhost kernel: [  271.625519]  ffff880037096300
>> ffff880079fcda00 00000000fffffffa ffffc90004704230
>> Mar 12 11:56:22 localhost kernel: [  271.625521]  ffffffffa0170e80
>> ffffffffa0170e80 0000000000000001 ffffffffa01b5708
>> Mar 12 11:56:22 localhost kernel: [  271.625524] Call Trace:
>> Mar 12 11:56:22 localhost kernel: [  271.625530] [<ffffffffa015ed27>]
>> rt_timer_handler+0x477/0x7e0 [rtai_sched]
>> Mar 12 11:56:22 localhost kernel: [  271.625532] [<ffffffffa015da1a>]
>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched]
>> Mar 12 11:56:22 localhost kernel: [  271.625536] [<ffffffff810d63f3>]
>> __ipipe_dispatch_irq+0xd3/0x1b0
>> Mar 12 11:56:22 localhost kernel: [  271.625539] [<ffffffff81031503>]
>> __ipipe_handle_irq+0x73/0x190
>> Mar 12 11:56:22 localhost kernel: [  271.625542] [<ffffffff816a16c0>]
>> apic_timer_interrupt+0x60/0x90
>> Mar 12 11:56:22 localhost kernel: [  271.625544] [<ffffffff813192f3>]
>> ? __this_cpu_preempt_check+0x13/0x20
>> Mar 12 11:56:22 localhost kernel: [  271.625593] [<ffffffffa180cd60>]
>> ? _nv008584rm+0x90/0x3e0 [nvidia]
>> Mar 12 11:56:22 localhost kernel: [  271.625596] [<ffffffff810919d0>]
>> ? rcu_eqs_enter_common.isra.43+0x60/0x120
>> Mar 12 11:56:22 localhost kernel: [  271.625598] [<ffffffff813192d7>]
>> ? debug_smp_processor_id+0x17/0x20
>> Mar 12 11:56:22 localhost kernel: [  271.625601] [<ffffffff8100c540>]
>> ? mwait_idle+0x60/0xa0
>> Mar 12 11:56:22 localhost kernel: [  271.625603] [<ffffffff8100d0ba>]
>> arch_cpu_idle+0xa/0x10
>> Mar 12 11:56:22 localhost kernel: [  271.625606] [<ffffffff810826a9>]
>> cpu_startup_entry+0x319/0x470
>> Mar 12 11:56:22 localhost kernel: [  271.625609] [<ffffffff810a5807>]
>> ? clockevents_config_and_register+0x27/0x30
>> Mar 12 11:56:22 localhost kernel: [  271.625611] [<ffffffff81032473>]
>> start_secondary+0x143/0x150
>> Mar 12 11:56:22 localhost kernel: INFO: rcu_preempt detected stalls on
>> CPUs/tasks: { 1} (detected by 0, t=21524 jiffies, g=11575, c=11574,
>> q=1230)
>> Mar 12 11:56:22 localhost kernel: Task dump for CPU 1:
>> Mar 12 11:56:22 localhost kernel: swapper/1       R  running task
>> 12624     0      1 0x00200000
>> Mar 12 11:56:22 localhost kernel: ffff880077417d00 0000000000000082
>> 0000000000000000 0000000000000000
>> Mar 12 11:56:22 localhost kernel: ffff880037096300 ffff880079fcda00
>> 00000000fffffffa ffffc90004704230
>> Mar 12 11:56:22 localhost kernel: ffffffffa0170e80 ffffffffa0170e80
>> 0000000000000001 ffffffffa01b5708
>> Mar 12 11:56:22 localhost kernel: Call Trace:
>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015ed27>]
>> rt_timer_handler+0x477/0x7e0 [rtai_sched]
>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015da1a>]
>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched]
>> Mar 12 11:56:22 localhost kernel: [<ffffffff810d63f3>]
>> __ipipe_dispatch_irq+0xd3/0x1b0
>> Mar 12 11:56:22 localhost kernel: [<ffffffff81031503>]
>> __ipipe_handle_irq+0x73/0x190
>> Mar 12 11:56:22 localhost kernel: [<ffffffff816a16c0>]
>> apic_timer_interrupt+0x60/0x90
>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192f3>] ?
>> __this_cpu_preempt_check+0x13/0x20
>> Mar 12 11:56:22 localhost kernel: [<ffffffffa180cd60>] ?
>> _nv008584rm+0x90/0x3e0 [nvidia]
>> Mar 12 11:56:22 localhost kernel: [<ffffffff810919d0>] ?
>> rcu_eqs_enter_common.isra.43+0x60/0x120
>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192d7>] ?
>> debug_smp_processor_id+0x17/0x20
>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100c540>] ?
>> mwait_idle+0x60/0xa0
>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100d0ba>]
>> arch_cpu_idle+0xa/0x10
>> Mar 12 11:56:22 localhost kernel: [<ffffffff810826a9>]
>> cpu_startup_entry+0x319/0x470
>> Mar 12 11:56:22 localhost kernel: [<ffffffff810a5807>] ?
>> clockevents_config_and_register+0x27/0x30
>> Mar 12 11:56:22 localhost kernel: [<ffffffff81032473>]
>> start_secondary+0x143/0x150
>>
>> _______________________________________________
>> Rtai mailing list
>> [email protected]
>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> _______________________________________________
> Rtai mailing list
> [email protected]
> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai