Re: rt_make_soft_real_time hangs my process

Hi Duane,
I don't see the benefit of having thread B.1 switching back and forth
to/from HRT. Why not just leaving it as hard realtime?
Also I'd be wary of using shared memory to do synchronization between
threads or processes, since there is no atomicity guarantee. I think it
should be safer to use a semaphore or a message box.
Btw, are you doing heap memory (de-)allocations from the rt threads?

Marco

On Mon, Jun 6, 2016 at 7:32 PM, Duane AW. Jeffery <
[email protected]> wrote:

> Synchronization mechanism is just a counter of frames received.  Thread 3
> busy loops (with usleep(100)) until the value is changed by Thread 1 which
> uses an atomic increment on that value.
>
> Nothing terribly fancy, really.
>
> -Duane
>
> -----Original Message-----
> From: Shahbaz Youssefi [mailto:[email protected]]
> Sent: Monday, June 06, 2016 1:03 PM
> To: Duane AW. Jeffery <[email protected]>
> Cc: Paolo Mantegazza <[email protected]>; [email protected]
> Subject: Re: [Rtai] rt_make_soft_real_time hangs my process
>
> "When using these two cameras, the RT version worked until we added Thread
> 3 a couple weeks ago."
>
> So thread 3 is not even real-time, but introducing it causes a system
> hang? Did you perhaps add some synchronization mechanism between thread 1
> and thread 3 also? And perhaps that mechanism is an RTAI
> mutex/semaphore/etc?
>
> On Mon, Jun 6, 2016 at 12:29 PM, Duane AW. Jeffery <
> [email protected]> wrote:
> > Sorry for the confusion Shahbaz,
> >
> > I was merely trying to delineate stuff in the list that was actual code
> pertaining to the RT calls, as opposed to just acquisition and calculation
> code.  You are correct, the calculation thread (Process B, Thread 1) is
> supposed to be an RTAI task through the bulk of its lifecycle, except
> during init and cleanup.  It switches between hard and soft when it does
> the calculations.  It is the only thread of the three that attempts to go
> real time.
> >
> > I will try swapping in some rt_printk's for the printf's I was using to
> try and locate the hang.  Good idea.
> >
> > -Duane
> >
> > -----Original Message-----
> > From: Shahbaz Youssefi [mailto:[email protected]]
> > Sent: Monday, June 06, 2016 12:08 PM
> > To: Duane AW. Jeffery <[email protected]>
> > Cc: Paolo Mantegazza <[email protected]>; [email protected]
> > Subject: Re: [Rtai] rt_make_soft_real_time hangs my process
> >
> > See inline:
> >
> > On Mon, Jun 6, 2016 at 11:24 AM, Duane AW. Jeffery <
> [email protected]> wrote:
> >>
> >> System is a 6-core system started with Kernel param isolcpus=2-5 RTAI
> >> is insmod'd with IsolCpusMask = 0x3C The user starts Process A from
> >> their desktop.
> >> Process A initializes itself, and tries to connect to the shared memory
> segment.
> >> Process A fires a system() call to start Process B and redirect its
> stdout to a log file.
> >> Process A fires a second system() call to migrate Process B into the
> desired cpuset (which was created at boot time by a script).
> >> Process B initializes itself (in Thread 1), and connects to the
> >> shared memory segment RT CODE:
> >> Process B has a global RT_TASK mytask; Process B calls
> >> rt_allow_nonroot_hrt(); Process B calls mytask =
> >> rt_task_init_schmod(nam2num("HRTSK"), 2, 0, 0, SCHED_FIFO, 0x02);
> >> (Is this perhaps the problem because CPU Mask 0x02 isn't part of the
> >> 0x3C from the insmod?) END RT CODE
> >
> > Why is this called END RT CODE? Aren't the threads run in real-time?
> > Have you verified that your mutexes (and other locks, if any) are of the
> RTAI type, and that the threads are switched to real-time?
> >
> >> Both processes handshake through the shared memory segment, and then
> Process A notifies Process B to initialize the camera.
> >> Process B Thread 1 calls into an init camera routine Process B Thread
> >> 1 finds and opens a link to the camera.
> >> Process B Thread 1 starts Thread 2, and the 2 threads communicate
> through global variables.
> >> Process B Thread 1 starts the stream from the camera Process B Thread
> >> 2 sees the Camera Started flag, and starts acquiring frames from the
> camera's non-RT driver and storing them in memory.
> >> Process B Thread 1 returns from the init camera routine, and begins
> >> initializing the new feature Process B Thread 1 starts Thread 3, also
> communicating with Thread 1 through global variables.
> >> RT CODE:
> >> Process B Thread 1 checks rt_is_hard_timer_running() If no, then it
> >> calls rt_set_oneshot_mode(); and start_rt_timer(0); Process B Thread
> >> 1 calls rt_grow_and_lock_stack(40000); END RT CODE From this point
> >> on, Process B Thread 1 checks the variables shared with Thread 2 to
> determine if a new frame has arrived.
> >> On arrival of a new frame, Process B Thread 1 calls
> >> rt_make_hard_real_time(); and rt_task_use_fpu(mytask, 1); Process B
> Thread 1 computes the main result and notifies Thread 3 to compute its
> result for the new feature.
> >> Process B Thread 1 fills up the structures in the shared memory
> >> segment to return its data to Process A Process B Thread 1 calls
> >> rt_make_soft_real_time(); Process B Thread 1 loops to checking for new
> frames.
> >> When the user commands a shutdown, Process B Thread 1 calls
> >> rt_task_delete(mytask); and if it started the timer, additionally
> >> calls stop_rt_timer();
> >>
> >> Any ideas anyone has to what I might try are certainly welcome.  I've
> >> tried to trace where it locks up with printf's into Process B's
> >> stdout, and it seems to be stalling on the call to
> >> rt_make_soft_real_time();
> >
> > Just a side note, try `rt_printk` instead of `printf`, and check
> `dmesg`. `printf` would take your task out of hard real-time, which is
> likely not what you want. Also, if the system hangs, you may not see all
> the `printf` logs, but you have a better chance with `rt_printk`.
> > You can have an always-on-top terminal running:
> >
> >     tail -f /var/log/messages
> >
> > for example to show you the kernel logs as they happen. Adding a small
> busy loop after each `rt_printk` would help make sure you see the log
> before the hang.
> >
> >>
> >> -Duane
> >>
> >> -----Original Message-----
> >> From: Paolo Mantegazza [mailto:[email protected]]
> >> Sent: Friday, June 03, 2016 4:58 PM
> >> To: Duane AW. Jeffery <[email protected]>; '[email protected]'
> >> <[email protected]>
> >> Subject: RE: rt_make_soft_real_time hangs my process
> >>
> >> Trivially recalling that Linux threads are substantially processes
> sharing a memory space by default, every thread of a process can be made
> hard-soft as you like, under RTAI.
> >> So if you have a thread that in Linux's hands to acquire frames through
> a not hard real time RTAI driver and want that as soon as a complete frame
> has been acquired it is  process in hard real time immediately you have
> just to enable (using rt_thread_init or rt_task_init_schmod) the
> acquisition LINUX thread to using RTAI functions, mating it to another hard
> real time hardened RTAI thread that waits for being awaken by the
> acuisition thread. That done define a multiple frames buffer, let the
> acquisition frame acquire frames continuously passing  an available  frame
> index to the hard real time thread. The mechanisms to be used can be just a
> semephore with the index being a per process variable, a suspend-resum,  an
> rpc_receive with the index passed through and rt_send.
> >> Hoping I undertood your proble, sorry if my suggestions were trivial
> stuff.
> >>
> >> Paolo
> >>
> >> ________________________________________
> >> From: Duane AW. Jeffery [[email protected]]
> >> Sent: Friday, June 3, 2016 6:54 PM
> >> To: Paolo Mantegazza; '[email protected]'
> >> Subject: RE: rt_make_soft_real_time hangs my process
> >>
> >> Its running on a video processing software, and we're doing it to
> ensure our image analysis routine is not interrupted.
> >>
> >> Each time a new frame arrives from the camera, it goes hard, processes
> the frame, and returns soft.  Framerates range from 100 to ~9000 fps
> depending on ROI.
> >>
> >> I think I might know what's going on though.  The process that this
> runs on creates a separate thread for acquiring the frames from the camera
> with its affinity set so that it runs on a separate core, and the realtime
> part is only for the processing of a frame after it arrived.  Both
> functions run under the same process though (ie. ps aux only shows one
> item), and so I think that might be my problem.  Can I even legally call
> rt_make_hard_real_time() on a multithreaded process?
> >>
> >> -Duane
> >>
> >> -----Original Message-----
> >> From: Paolo Mantegazza [mailto:[email protected]]
> >> Sent: Friday, June 03, 2016 9:59 AM
> >> To: Duane AW. Jeffery <[email protected]>; '[email protected]'
> >> <[email protected]>
> >> Subject: RE: rt_make_soft_real_time hangs my process
> >>
> >> Can you provide the frequency of the hard7soft switches?
> >> If you a doing it for a controlled approach to Linux syscalls , try
> letting RTAI take care of it and see what happens. In such a view you might
> even try  using a Linux  server .
> >> That said, eventually this is the case in which it should not be too
> complex to set a neutral test up, so that I cantry to diagnose the problem
> you are wittnessing.
> >>
> >> Paolo
> >> ________________________________________
> >> From: Rtai [[email protected]] on behalf of Duane AW. Jeffery
> >> [[email protected]]
> >> Sent: Friday, June 3, 2016 3:12 PM
> >> To: '[email protected]'
> >> Subject: [Rtai] rt_make_soft_real_time hangs my process
> >>
> >> Hi all,
> >>
> >> I'm running a shiny new system on Linux 4.1.18 w/ RTAI 5 from
> >> vulcano, and it seems to be going well.  (System is derived from a
> >> Slackware
> >> 14.1 install)
> >>
> >> I seem to be having a small problem with one of my applications where
> it calls into rt_make_soft_real_time() which never returns.  It previously
> calls rt_make_hard_real_time(), and calls the pair in a loop, so it
> switches back and forth fairly frequently.
> >>
> >> Any ideas?
> >>
> >> -Duane
> >> _______________________________________________
> >> Rtai mailing list
> >> [email protected]
> >> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
> _______________________________________________
> Rtai mailing list
> [email protected]
> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>



-- 
Marco Pantaleoni

_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai