Re: Rtai Digest, Vol 99, Issue 31
JB <[email protected]>
| Newsgroups | gmane.linux.real-time.rtai |
|---|---|
| Message-ID | <[email protected]> |
Thank you!!!!! I don't know if it was something with my kernel config (which I'm happy to share if needed) or what but your kernel config combined with probably the same sched.c that Paulo sent me a couple days ago as well has in fact done the trick! The behavior now very closely resembles the prior behavior. On 3/18/2016 14:36, Francescodario Cuzzocrea wrote: > [Call for Testing ] > > I also had these kind of problems on my machines with RTAI 5.0-test1. > Today I tried the modified sched.c that Paolo sent to me a couple of > days ago and now everything works fine (at least on my machines, for > my applications (primary simulink applications generated by real time > workshop with rtai target)). > I'm using kernel 3.18 on arch linux. > I'm attacching to this mail the modified sched.c and my kernel .config > file as reference in order to enable more people to test the changes. > Thank You. > > Il giorno mer 16 mar 2016 alle ore 13:38 <[email protected] > <mailto:[email protected]>> ha scritto: > > Send Rtai mailing list submissions to > [email protected] <mailto:[email protected]> > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai > or, via email, send a message with subject or body 'help' to > [email protected] <mailto:[email protected]> > > You can reach the person managing the list at > [email protected] <mailto:[email protected]> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Rtai digest..." > > > Today's Topics: > > 1. Re: rtai task causes rcu problems on exit (JB) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 15 Mar 2016 20:23:59 -0600 > From: JB <[email protected] <mailto:[email protected]>> > To: "[email protected] <mailto:[email protected]>" <[email protected] > <mailto:[email protected]>> > Subject: Re: [Rtai] rtai task causes rcu problems on exit > Message-ID: <[email protected] > <mailto:[email protected]>> > Content-Type: text/plain; charset=windows-1252; format=flowed > > I don't think it's the kernel. I think it's changes to RTAI. I'm going > to need to spend some time with a simplified RTAI application to > see if > I can execute it hard real time without causing the problem. > > Under RTAI 5.0-test with kernel 3.10.32 I get slightly different > behavior, but essentially the same problem. The hard realtime task > essentially hangs the system. If I run the exact same task with > identical code using RTAI 4.0 on kernel 3.8.13, it works perfectly. > > > > On 3/13/2016 20:59, JB wrote: > > Interesting, it sounds very very similar to what I'm seeing. > > Everything works great after starting the task and synchronizing the > > tasks/processes, but either the first time or the time after that, I > > lose system responsiveness and get the warnings. So do I understand > > you correctly in that you are saying we have a bug in our shutdown > > sequence? It has worked flawlessly for years. Did something > change in > > how realtime is stopped? If so, what is the appropriate shutdown > > sequence? > > > > When I have my system back I will look into that and try kernel > 3.10. > > > > JB > > > > > > On 3/13/2016 17:24, Paolo Mantegazza wrote: > >> Perhaps, I've been able to reproduce your problem. It did not > appear > >> under heavy RTAI load. My PC was barely usable, but once at the > >> closing of the tasks. > >> I tried a few other times but everything was OK. Since some of the > >> many example I launched start-stop the hard timer, while a few > other > >> check if that has to be done or not, it is then likely that I > closed > >> one that stopped the timer., so that the remaining timed task began > >> seeing overruns and looped without a pause, making the RCU messages > >> appear. > >> I do not know if something like that could happen in you case also, > >> just my try to help. > >> > >> Paolo > >> > >> ________________________________________ > >> From: JB [[email protected] <mailto:[email protected]>] > >> Sent: Sunday, March 13, 2016 9:40 PM > >> To: Paolo Mantegazza; [email protected] <mailto:[email protected]> > >> Subject: Re: [Rtai] rtai task causes rcu problems on exit > >> > >> Indeed, I'll see what I can do and report back. > >> > >> > >> On 3/13/2016 14:19, Paolo Mantegazza wrote: > >>> BTW, RTAI 5 supports 3.10 also, which is closer (in time) to 3.8. > >>> Maybe that the RCU diagnosis trouble you is not there yet. > >>> A nice chance to contribute a checking for it also. > >>> Paolo > >>> > >>> ________________________________________ > >>> From: Rtai [[email protected] > <mailto:[email protected]>] on behalf of JB [[email protected] > <mailto:[email protected]>] > >>> Sent: Sunday, March 13, 2016 7:21 PM > >>> To: [email protected] <mailto:[email protected]> > >>> Subject: Re: [Rtai] rtai task causes rcu problems on exit > >>> > >>> Thanks Paolo! Unfortunately, it definitely has to do with the > >>> combination of upgrade to 3.18.22 and rtai-5.0-test1. > >>> > >>> Previously I was running RTAI 4.0 on a 3.8.13 kernel and prior > to that > >>> it was earlier 3.x and 2.6.x and 2.4.x kernels. The real-time > >>> processing > >>> components of the application have remained largely unchanged > beyond > >>> the > >>> minor changes to move off deprecated things (like better SMP > support in > >>> the application when moving from 2.4 to 2.6 and migrating from > rtai > >>> fifos to rtai mailboxes in later version of 2.6. Other > applications > >>> have > >>> always been very responsive! Just as soon as I tried to go to > 3.18.22 > >>> and rtai-5.0-test1 did this start happening. > >>> > >>> I had already read the documentation at the URL you gave me. > However, > >>> while it does talk about settings to disable RCU, those > settings appear > >>> to be ignored. I attempted to enable the suppression of RCU > and the > >>> setting was completely ignored. The RTAI application we use is > running > >>> at 2.4kHz and never overruns. It's nothing too aggressive and > certainly > >>> shouldn't be causing this. > >>> > >>> I upgraded because I was getting a crash on some CPUs with > rtai-4.0 and > >>> 3.8.13. However, I've spent quite a bit of time on trying to > figure out > >>> what's wrong and without busting out all the kernel debugging and > >>> hacking tools and climbing into it's guts I'm unable to > determine the > >>> cause. So, I think I'll go back to 3.8.13 and see if I can > make that > >>> work again. > >>> > >>> > >>> > >>> > >>> On 3/13/2016 04:50, Paolo Mantegazza wrote: > >>>> My assumption is that your task steals too much time too Linux > >>>> somewhere, at least to annoy the some read-copy-update stuff. > >>>> Somewhere on the net I found: > >>>> "You probably have a real time application that is consuming all > >>>> cpu (some bad implementation) and because of its realtime > >>>> scheduling priority the system doesn't have enough resources > >>>> available for other tasks.". Nothe that (s)hewas not talking of > >>>> hard real time but simple of the real time oprion available to a > >>>> super user under standard Linux POSIX. > >>>> I do not know if I and the guy on the net are right but if your > >>>> RTAI task is running fine, since RTAI very hardly locks out any > >>>> LINUX activity what above might be even a truer justification > for a > >>>> CPU stall, in LINUX view. After all RTAI is a LINUX staller. > >>>> Maybe the following can help: > >>>> shttps://www.kernel.org/doc/Documentation/RCU/stallwarn.txt > <http://www.kernel.org/doc/Documentation/RCU/stallwarn.txt> > >>>> Paolo > >>>> ________________________________________ > >>>> From: Rtai [[email protected] > <mailto:[email protected]>] on behalf of JB [[email protected] > <mailto:[email protected]>] > >>>> Sent: Sunday, March 13, 2016 4:02 AM > >>>> To: [email protected] <mailto:[email protected]> > >>>> Subject: Re: [Rtai] rtai task causes rcu problems on exit > >>>> > >>>> Correction, they occur the entire time the real time task is > >>>> running. Is > >>>> there a way to disable this somehow or configure it so it is > >>>> ignored. It > >>>> is cause multi-second delays in the execution of other programs > >>>> running > >>>> on the same system as the real-time task. > >>>> > >>>> > >>>> On 3/12/2016 19:52, JB wrote: > >>>>> Hi, > >>>>> I'm a little confused about some behavior I'm currently > seeing after > >>>>> upgrading to kernel 3.18.22 and RTAI 5.0-test1. After my > real-time > >>>>> task ends the first, time all seems fine. However, if I > start it up a > >>>>> second time, then stop it, it doesn't stop for a while. After a > >>>>> while, > >>>>> it stops and the messages below show up in the logs. Is RTAI > >>>>> conflicting in some way with this new RCU feature of the kernel? > >>>>> > >>>>> > >>>>> Mar 12 11:56:00 localhost kernel: [ 249.450228] INFO: > rcu_preempt > >>>>> detected stalls on CPUs/tasks: {} (detected by 0, t=21104 > jiffies, > >>>>> g=11573, c=11572, q=30 > >>>>> Mar 12 11:56:00 localhost kernel: [ 249.450234] INFO: Stall > ended > >>>>> before state dump start > >>>>> Mar 12 11:56:00 localhost kernel: INFO: rcu_preempt detected > >>>>> stalls on > >>>>> CPUs/tasks: {} (detected by 0, t=21104 jiffies, g=11573, > c=11572, > >>>>> q=3068) > >>>>> Mar 12 11:56:00 localhost kernel: INFO: Stall ended before > state dump > >>>>> start > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625505] INFO: > rcu_preempt > >>>>> detected stalls on CPUs/tasks: { 1} (detected by 0, t=21524 > jiffies, > >>>>> g=11575, c=11574, q= > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625511] Task dump > for CPU 1: > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625512] swapper/1 > R > >>>>> running task 12624 0 1 0x00200000 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625516] ffff880077417d00 > >>>>> 0000000000000082 0000000000000000 0000000000000000 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625519] ffff880037096300 > >>>>> ffff880079fcda00 00000000fffffffa ffffc90004704230 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625521] ffffffffa0170e80 > >>>>> ffffffffa0170e80 0000000000000001 ffffffffa01b5708 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625524] Call Trace: > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625530] > [<ffffffffa015ed27>] > >>>>> rt_timer_handler+0x477/0x7e0 [rtai_sched] > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625532] > [<ffffffffa015da1a>] > >>>>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched] > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625536] > [<ffffffff810d63f3>] > >>>>> __ipipe_dispatch_irq+0xd3/0x1b0 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625539] > [<ffffffff81031503>] > >>>>> __ipipe_handle_irq+0x73/0x190 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625542] > [<ffffffff816a16c0>] > >>>>> apic_timer_interrupt+0x60/0x90 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625544] > [<ffffffff813192f3>] > >>>>> ? __this_cpu_preempt_check+0x13/0x20 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625593] > [<ffffffffa180cd60>] > >>>>> ? _nv008584rm+0x90/0x3e0 [nvidia] > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625596] > [<ffffffff810919d0>] > >>>>> ? rcu_eqs_enter_common.isra.43+0x60/0x120 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625598] > [<ffffffff813192d7>] > >>>>> ? debug_smp_processor_id+0x17/0x20 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625601] > [<ffffffff8100c540>] > >>>>> ? mwait_idle+0x60/0xa0 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625603] > [<ffffffff8100d0ba>] > >>>>> arch_cpu_idle+0xa/0x10 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625606] > [<ffffffff810826a9>] > >>>>> cpu_startup_entry+0x319/0x470 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625609] > [<ffffffff810a5807>] > >>>>> ? clockevents_config_and_register+0x27/0x30 > >>>>> Mar 12 11:56:22 localhost kernel: [ 271.625611] > [<ffffffff81032473>] > >>>>> start_secondary+0x143/0x150 > >>>>> Mar 12 11:56:22 localhost kernel: INFO: rcu_preempt detected > >>>>> stalls on > >>>>> CPUs/tasks: { 1} (detected by 0, t=21524 jiffies, g=11575, > c=11574, > >>>>> q=1230) > >>>>> Mar 12 11:56:22 localhost kernel: Task dump for CPU 1: > >>>>> Mar 12 11:56:22 localhost kernel: swapper/1 R running task > >>>>> 12624 0 1 0x00200000 > >>>>> Mar 12 11:56:22 localhost kernel: ffff880077417d00 > 0000000000000082 > >>>>> 0000000000000000 0000000000000000 > >>>>> Mar 12 11:56:22 localhost kernel: ffff880037096300 > ffff880079fcda00 > >>>>> 00000000fffffffa ffffc90004704230 > >>>>> Mar 12 11:56:22 localhost kernel: ffffffffa0170e80 > ffffffffa0170e80 > >>>>> 0000000000000001 ffffffffa01b5708 > >>>>> Mar 12 11:56:22 localhost kernel: Call Trace: > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015ed27>] > >>>>> rt_timer_handler+0x477/0x7e0 [rtai_sched] > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015da1a>] > >>>>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched] > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810d63f3>] > >>>>> __ipipe_dispatch_irq+0xd3/0x1b0 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff81031503>] > >>>>> __ipipe_handle_irq+0x73/0x190 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff816a16c0>] > >>>>> apic_timer_interrupt+0x60/0x90 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192f3>] ? > >>>>> __this_cpu_preempt_check+0x13/0x20 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa180cd60>] ? > >>>>> _nv008584rm+0x90/0x3e0 [nvidia] > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810919d0>] ? > >>>>> rcu_eqs_enter_common.isra.43+0x60/0x120 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192d7>] ? > >>>>> debug_smp_processor_id+0x17/0x20 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100c540>] ? > >>>>> mwait_idle+0x60/0xa0 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100d0ba>] > >>>>> arch_cpu_idle+0xa/0x10 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810826a9>] > >>>>> cpu_startup_entry+0x319/0x470 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810a5807>] ? > >>>>> clockevents_config_and_register+0x27/0x30 > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff81032473>] > >>>>> start_secondary+0x143/0x150 > >>>>> > >>>>> _______________________________________________ > >>>>> Rtai mailing list > >>>>> [email protected] <mailto:[email protected]> > >>>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai > >>>> _______________________________________________ > >>>> Rtai mailing list > >>>> [email protected] <mailto:[email protected]> > >>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai > >>> _______________________________________________ > >>> Rtai mailing list > >>> [email protected] <mailto:[email protected]> > >>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai > > > > _______________________________________________ > > Rtai mailing list > > [email protected] <mailto:[email protected]> > > https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Rtai mailing list > [email protected] <mailto:[email protected]> > https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai > > ------------------------------ > > End of Rtai Digest, Vol 99, Issue 31 > ************************************ > > > > _______________________________________________ > Rtai mailing list > [email protected] > https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai _______________________________________________ Rtai mailing list [email protected] https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai