Re: Rtai Digest, Vol 99, Issue 33

Thank you for your reply and for trying out my config ! I'm glad that works
now. I think that was both coulprit of the config and of the scheduler.
I'll write an updated guide about kernel and rtai configuration when I will
have some spare time !

Il ven 18 mar 2016, 21:37 <[email protected]> ha scritto:

> Send Rtai mailing list submissions to
>         [email protected]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> or, via email, send a message with subject or body 'help' to
>         [email protected]
>
> You can reach the person managing the list at
>         [email protected]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Rtai digest..."
>
>
> Today's Topics:
>
>    1. Re: Rtai Digest, Vol 99, Issue 31 (Francescodario Cuzzocrea)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 18 Mar 2016 20:36:58 +0000
> From: Francescodario Cuzzocrea
>         <[email protected]>
> To: [email protected]
> Subject: Re: [Rtai] Rtai Digest, Vol 99, Issue 31
> Message-ID:
>         <
> CADnVkj_t_b6zpn02sOsi1CcJj9xXjLJ6MOdHfn_8ikU_GykMSQ@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> [Call for Testing ]
>
> I also had these kind of problems on my machines with RTAI 5.0-test1.
> Today I tried the modified sched.c that Paolo sent to me a couple of days
> ago and now everything works fine (at least on my machines, for my
> applications (primary simulink applications generated by real time workshop
> with rtai target)).
> I'm using kernel 3.18 on arch linux.
> I'm attacching to this mail the modified sched.c and my kernel .config file
> as reference in order to enable more people to test the changes.
> Thank You.
>
> Il giorno mer 16 mar 2016 alle ore 13:38 <[email protected]> ha
> scritto:
>
> > Send Rtai mailing list submissions to
> >         [email protected]
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> > or, via email, send a message with subject or body 'help' to
> >         [email protected]
> >
> > You can reach the person managing the list at
> >         [email protected]
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Rtai digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Re: rtai task causes rcu problems on exit (JB)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Tue, 15 Mar 2016 20:23:59 -0600
> > From: JB <[email protected]>
> > To: "[email protected]" <[email protected]>
> > Subject: Re: [Rtai] rtai task causes rcu problems on exit
> > Message-ID: <[email protected]>
> > Content-Type: text/plain; charset=windows-1252; format=flowed
> >
> > I don't think it's the kernel. I think it's changes to RTAI. I'm going
> > to need to spend some time with a simplified RTAI application to see if
> > I can execute it hard real time without causing the problem.
> >
> > Under RTAI 5.0-test with kernel 3.10.32 I get slightly different
> > behavior, but essentially the same problem. The hard realtime task
> > essentially hangs the system. If I run the exact same task with
> > identical code using RTAI 4.0 on kernel 3.8.13, it works perfectly.
> >
> >
> >
> > On 3/13/2016 20:59, JB wrote:
> > > Interesting, it sounds very very similar to what I'm seeing.
> > > Everything works great after starting the task and synchronizing the
> > > tasks/processes, but either the first time or the time after that, I
> > > lose system responsiveness and get the warnings. So do I understand
> > > you correctly in that you are saying we have a bug in our shutdown
> > > sequence? It has worked flawlessly for years. Did something change in
> > > how realtime is stopped? If so, what is the appropriate shutdown
> > > sequence?
> > >
> > > When I have my system back I will look into that and try kernel 3.10.
> > >
> > > JB
> > >
> > >
> > > On 3/13/2016 17:24, Paolo Mantegazza wrote:
> > >> Perhaps, I've been able to reproduce your problem. It did not appear
> > >> under heavy RTAI load. My PC was barely usable, but once at the
> > >> closing of the tasks.
> > >> I tried a few other times but everything was OK. Since some of the
> > >> many example I launched start-stop the hard timer, while a few other
> > >> check if that has to be done or not, it is then likely that I closed
> > >> one that stopped the timer., so that the remaining timed task began
> > >> seeing overruns and looped without a pause, making the RCU messages
> > >> appear.
> > >> I do not know if something like that could happen in you case also,
> > >> just my try to help.
> > >>
> > >> Paolo
> > >>
> > >> ________________________________________
> > >> From: JB [[email protected]]
> > >> Sent: Sunday, March 13, 2016 9:40 PM
> > >> To: Paolo Mantegazza; [email protected]
> > >> Subject: Re: [Rtai] rtai task causes rcu problems on exit
> > >>
> > >> Indeed, I'll see what I can do and report back.
> > >>
> > >>
> > >> On 3/13/2016 14:19, Paolo Mantegazza wrote:
> > >>> BTW, RTAI 5 supports 3.10 also, which is closer (in time) to 3.8.
> > >>> Maybe that the RCU diagnosis trouble you is not there yet.
> > >>> A nice chance to contribute a checking for it also.
> > >>> Paolo
> > >>>
> > >>> ________________________________________
> > >>> From: Rtai [[email protected]] on behalf of JB [
> [email protected]]
> > >>> Sent: Sunday, March 13, 2016 7:21 PM
> > >>> To: [email protected]
> > >>> Subject: Re: [Rtai] rtai task causes rcu problems on exit
> > >>>
> > >>> Thanks Paolo!  Unfortunately, it definitely has to do with the
> > >>> combination of upgrade to 3.18.22 and rtai-5.0-test1.
> > >>>
> > >>> Previously I was running RTAI 4.0 on a 3.8.13 kernel and prior to
> that
> > >>> it was earlier 3.x and 2.6.x and 2.4.x kernels. The real-time
> > >>> processing
> > >>> components of the application have remained largely unchanged beyond
> > >>> the
> > >>> minor changes to move off deprecated things (like better SMP support
> in
> > >>> the application when moving from 2.4 to 2.6 and migrating from rtai
> > >>> fifos to rtai mailboxes in later version of 2.6. Other applications
> > >>> have
> > >>> always been very responsive! Just as soon as I tried to go to 3.18.22
> > >>> and rtai-5.0-test1 did this start happening.
> > >>>
> > >>> I had already read the documentation at the URL you gave me. However,
> > >>> while it does talk about settings to disable RCU, those settings
> appear
> > >>> to be ignored. I attempted to enable the suppression of RCU and the
> > >>> setting was completely ignored. The RTAI application we use is
> running
> > >>> at 2.4kHz and never overruns. It's nothing too aggressive and
> certainly
> > >>> shouldn't be causing this.
> > >>>
> > >>> I upgraded because I was getting a crash on some CPUs with rtai-4.0
> and
> > >>> 3.8.13. However, I've spent quite a bit of time on trying to figure
> out
> > >>> what's wrong and without busting out all the kernel debugging and
> > >>> hacking tools and climbing into it's guts I'm unable to determine the
> > >>> cause. So, I think I'll go back to 3.8.13 and see if I can make that
> > >>> work again.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On 3/13/2016 04:50, Paolo Mantegazza wrote:
> > >>>> My assumption is that your task steals too much time too Linux
> > >>>> somewhere, at least to annoy the some read-copy-update stuff.
> > >>>> Somewhere on the net I found:
> > >>>> "You probably have a real time application that is consuming all
> > >>>> cpu (some bad implementation) and because of its realtime
> > >>>> scheduling priority the system doesn't have enough resources
> > >>>> available for other tasks.". Nothe that (s)hewas not talking of
> > >>>> hard real time but simple of the real time oprion available to a
> > >>>> super user under standard Linux POSIX.
> > >>>> I do not know if I and the guy on the net are right but if your
> > >>>> RTAI task is running fine, since RTAI very hardly locks out any
> > >>>> LINUX activity what above might be even a truer justification for a
> > >>>> CPU stall, in LINUX view. After all RTAI is a LINUX staller.
> > >>>> Maybe the following can help:
> > >>>> shttps://www.kernel.org/doc/Documentation/RCU/stallwarn.txt
> > >>>> Paolo
> > >>>> ________________________________________
> > >>>> From: Rtai [[email protected]] on behalf of JB [
> [email protected]
> > ]
> > >>>> Sent: Sunday, March 13, 2016 4:02 AM
> > >>>> To: [email protected]
> > >>>> Subject: Re: [Rtai] rtai task causes rcu problems on exit
> > >>>>
> > >>>> Correction, they occur the entire time the real time task is
> > >>>> running. Is
> > >>>> there a way to disable this somehow or configure it so it is
> > >>>> ignored. It
> > >>>> is cause multi-second delays in the execution of other programs
> > >>>> running
> > >>>> on the same system as the real-time task.
> > >>>>
> > >>>>
> > >>>> On 3/12/2016 19:52, JB wrote:
> > >>>>> Hi,
> > >>>>> I'm a little confused about some behavior I'm currently seeing
> after
> > >>>>> upgrading to kernel 3.18.22 and RTAI 5.0-test1. After my real-time
> > >>>>> task ends the first, time all seems fine. However, if I start it
> up a
> > >>>>> second time, then stop it, it doesn't stop for a while. After a
> > >>>>> while,
> > >>>>> it stops and the messages below show up in the logs. Is RTAI
> > >>>>> conflicting in some way with this new RCU feature of the kernel?
> > >>>>>
> > >>>>>
> > >>>>> Mar 12 11:56:00 localhost kernel: [  249.450228] INFO: rcu_preempt
> > >>>>> detected stalls on CPUs/tasks: {} (detected by 0, t=21104 jiffies,
> > >>>>> g=11573, c=11572, q=30
> > >>>>> Mar 12 11:56:00 localhost kernel: [  249.450234] INFO: Stall ended
> > >>>>> before state dump start
> > >>>>> Mar 12 11:56:00 localhost kernel: INFO: rcu_preempt detected
> > >>>>> stalls on
> > >>>>> CPUs/tasks: {} (detected by 0, t=21104 jiffies, g=11573, c=11572,
> > >>>>> q=3068)
> > >>>>> Mar 12 11:56:00 localhost kernel: INFO: Stall ended before state
> dump
> > >>>>> start
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625505] INFO: rcu_preempt
> > >>>>> detected stalls on CPUs/tasks: { 1} (detected by 0, t=21524
> jiffies,
> > >>>>> g=11575, c=11574, q=
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625511] Task dump for CPU
> 1:
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625512] swapper/1       R
> > >>>>> running task    12624     0      1 0x00200000
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625516] ffff880077417d00
> > >>>>> 0000000000000082 0000000000000000 0000000000000000
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625519] ffff880037096300
> > >>>>> ffff880079fcda00 00000000fffffffa ffffc90004704230
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625521] ffffffffa0170e80
> > >>>>> ffffffffa0170e80 0000000000000001 ffffffffa01b5708
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625524] Call Trace:
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625530]
> [<ffffffffa015ed27>]
> > >>>>> rt_timer_handler+0x477/0x7e0 [rtai_sched]
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625532]
> [<ffffffffa015da1a>]
> > >>>>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched]
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625536]
> [<ffffffff810d63f3>]
> > >>>>> __ipipe_dispatch_irq+0xd3/0x1b0
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625539]
> [<ffffffff81031503>]
> > >>>>> __ipipe_handle_irq+0x73/0x190
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625542]
> [<ffffffff816a16c0>]
> > >>>>> apic_timer_interrupt+0x60/0x90
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625544]
> [<ffffffff813192f3>]
> > >>>>> ? __this_cpu_preempt_check+0x13/0x20
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625593]
> [<ffffffffa180cd60>]
> > >>>>> ? _nv008584rm+0x90/0x3e0 [nvidia]
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625596]
> [<ffffffff810919d0>]
> > >>>>> ? rcu_eqs_enter_common.isra.43+0x60/0x120
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625598]
> [<ffffffff813192d7>]
> > >>>>> ? debug_smp_processor_id+0x17/0x20
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625601]
> [<ffffffff8100c540>]
> > >>>>> ? mwait_idle+0x60/0xa0
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625603]
> [<ffffffff8100d0ba>]
> > >>>>> arch_cpu_idle+0xa/0x10
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625606]
> [<ffffffff810826a9>]
> > >>>>> cpu_startup_entry+0x319/0x470
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625609]
> [<ffffffff810a5807>]
> > >>>>> ? clockevents_config_and_register+0x27/0x30
> > >>>>> Mar 12 11:56:22 localhost kernel: [  271.625611]
> [<ffffffff81032473>]
> > >>>>> start_secondary+0x143/0x150
> > >>>>> Mar 12 11:56:22 localhost kernel: INFO: rcu_preempt detected
> > >>>>> stalls on
> > >>>>> CPUs/tasks: { 1} (detected by 0, t=21524 jiffies, g=11575, c=11574,
> > >>>>> q=1230)
> > >>>>> Mar 12 11:56:22 localhost kernel: Task dump for CPU 1:
> > >>>>> Mar 12 11:56:22 localhost kernel: swapper/1       R running task
> > >>>>> 12624     0      1 0x00200000
> > >>>>> Mar 12 11:56:22 localhost kernel: ffff880077417d00 0000000000000082
> > >>>>> 0000000000000000 0000000000000000
> > >>>>> Mar 12 11:56:22 localhost kernel: ffff880037096300 ffff880079fcda00
> > >>>>> 00000000fffffffa ffffc90004704230
> > >>>>> Mar 12 11:56:22 localhost kernel: ffffffffa0170e80 ffffffffa0170e80
> > >>>>> 0000000000000001 ffffffffa01b5708
> > >>>>> Mar 12 11:56:22 localhost kernel: Call Trace:
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015ed27>]
> > >>>>> rt_timer_handler+0x477/0x7e0 [rtai_sched]
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa015da1a>]
> > >>>>> rtai_hirq_dispatcher+0x6a/0x1c0 [rtai_sched]
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810d63f3>]
> > >>>>> __ipipe_dispatch_irq+0xd3/0x1b0
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff81031503>]
> > >>>>> __ipipe_handle_irq+0x73/0x190
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff816a16c0>]
> > >>>>> apic_timer_interrupt+0x60/0x90
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192f3>] ?
> > >>>>> __this_cpu_preempt_check+0x13/0x20
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffffa180cd60>] ?
> > >>>>> _nv008584rm+0x90/0x3e0 [nvidia]
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810919d0>] ?
> > >>>>> rcu_eqs_enter_common.isra.43+0x60/0x120
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff813192d7>] ?
> > >>>>> debug_smp_processor_id+0x17/0x20
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100c540>] ?
> > >>>>> mwait_idle+0x60/0xa0
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff8100d0ba>]
> > >>>>> arch_cpu_idle+0xa/0x10
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810826a9>]
> > >>>>> cpu_startup_entry+0x319/0x470
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff810a5807>] ?
> > >>>>> clockevents_config_and_register+0x27/0x30
> > >>>>> Mar 12 11:56:22 localhost kernel: [<ffffffff81032473>]
> > >>>>> start_secondary+0x143/0x150
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> Rtai mailing list
> > >>>>> [email protected]
> > >>>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> > >>>> _______________________________________________
> > >>>> Rtai mailing list
> > >>>> [email protected]
> > >>>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> > >>> _______________________________________________
> > >>> Rtai mailing list
> > >>> [email protected]
> > >>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> > >
> > > _______________________________________________
> > > Rtai mailing list
> > > [email protected]
> > > https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Rtai mailing list
> > [email protected]
> > https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> >
> > ------------------------------
> >
> > End of Rtai Digest, Vol 99, Issue 31
> > ************************************
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.rtai.org/pipermail/rtai/attachments/20160318/2643d3f2/attachment.html
> >
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: config.x86_64
> Type: application/octet-stream
> Size: 117880 bytes
> Desc: not available
> URL: <
> http://mail.rtai.org/pipermail/rtai/attachments/20160318/2643d3f2/attachment.obj
> >
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: sched.c
> Type: text/x-csrc
> Size: 76664 bytes
> Desc: not available
> URL: <
> http://mail.rtai.org/pipermail/rtai/attachments/20160318/2643d3f2/attachment.c
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Rtai mailing list
> [email protected]
> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>
> ------------------------------
>
> End of Rtai Digest, Vol 99, Issue 33
> ************************************
>

_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai