Re: rt_make_soft_real_time hangs my process

See inline:

On Mon, Jun 6, 2016 at 11:24 AM, Duane AW. Jeffery
<[email protected]> wrote:
>
> System is a 6-core system started with Kernel param isolcpus=2-5
> RTAI is insmod'd with IsolCpusMask = 0x3C
> The user starts Process A from their desktop.
> Process A initializes itself, and tries to connect to the shared memory segment.
> Process A fires a system() call to start Process B and redirect its stdout to a log file.
> Process A fires a second system() call to migrate Process B into the desired cpuset (which was created at boot time by a script).
> Process B initializes itself (in Thread 1), and connects to the shared memory segment
> RT CODE:
> Process B has a global RT_TASK mytask;
> Process B calls rt_allow_nonroot_hrt();
> Process B calls mytask = rt_task_init_schmod(nam2num("HRTSK"), 2, 0, 0, SCHED_FIFO, 0x02);  (Is this perhaps the problem because CPU Mask 0x02 isn't part of the 0x3C from the insmod?)
> END RT CODE

Why is this called END RT CODE? Aren't the threads run in real-time?
Have you verified that your mutexes (and other locks, if any) are of
the RTAI type, and that the threads are switched to real-time?

> Both processes handshake through the shared memory segment, and then Process A notifies Process B to initialize the camera.
> Process B Thread 1 calls into an init camera routine
> Process B Thread 1 finds and opens a link to the camera.
> Process B Thread 1 starts Thread 2, and the 2 threads communicate through global variables.
> Process B Thread 1 starts the stream from the camera
> Process B Thread 2 sees the Camera Started flag, and starts acquiring frames from the camera's non-RT driver and storing them in memory.
> Process B Thread 1 returns from the init camera routine, and begins initializing the new feature
> Process B Thread 1 starts Thread 3, also communicating with Thread 1 through global variables.
> RT CODE:
> Process B Thread 1 checks rt_is_hard_timer_running()
> If no, then it calls rt_set_oneshot_mode(); and start_rt_timer(0);
> Process B Thread 1 calls rt_grow_and_lock_stack(40000);
> END RT CODE
> From this point on, Process B Thread 1 checks the variables shared with Thread 2 to determine if a new frame has arrived.
> On arrival of a new frame, Process B Thread 1 calls rt_make_hard_real_time(); and rt_task_use_fpu(mytask, 1);
> Process B Thread 1 computes the main result and notifies Thread 3 to compute its result for the new feature.
> Process B Thread 1 fills up the structures in the shared memory segment to return its data to Process A
> Process B Thread 1 calls rt_make_soft_real_time();
> Process B Thread 1 loops to checking for new frames.
> When the user commands a shutdown, Process B Thread 1 calls rt_task_delete(mytask); and if it started the timer, additionally calls stop_rt_timer();
>
> Any ideas anyone has to what I might try are certainly welcome.  I've tried to trace where it locks up with printf's into Process B's stdout, and it seems to be stalling on the call to rt_make_soft_real_time();

Just a side note, try `rt_printk` instead of `printf`, and check
`dmesg`. `printf` would take your task out of hard real-time, which is
likely not what you want. Also, if the system hangs, you may not see
all the `printf` logs, but you have a better chance with `rt_printk`.
You can have an always-on-top terminal running:

    tail -f /var/log/messages

for example to show you the kernel logs as they happen. Adding a small
busy loop after each `rt_printk` would help make sure you see the log
before the hang.

>
> -Duane
>
> -----Original Message-----
> From: Paolo Mantegazza [mailto:[email protected]]
> Sent: Friday, June 03, 2016 4:58 PM
> To: Duane AW. Jeffery <[email protected]>; '[email protected]' <[email protected]>
> Subject: RE: rt_make_soft_real_time hangs my process
>
> Trivially recalling that Linux threads are substantially processes sharing a memory space by default, every thread of a process can be made hard-soft as you like, under RTAI.
> So if you have a thread that in Linux's hands to acquire frames through a not hard real time RTAI driver and want that as soon as a complete frame has been acquired it is  process in hard real time immediately you have just to enable (using rt_thread_init or rt_task_init_schmod) the acquisition LINUX thread to using RTAI functions, mating it to another hard real time hardened RTAI thread that waits for being awaken by the acuisition thread. That done define a multiple frames buffer, let the acquisition frame acquire frames continuously passing  an available  frame index to the hard real time thread. The mechanisms to be used can be just a semephore with the index being a per process variable, a suspend-resum,  an rpc_receive with the index passed through and rt_send.
> Hoping I undertood your proble, sorry if my suggestions were trivial stuff.
>
> Paolo
>
> ________________________________________
> From: Duane AW. Jeffery [[email protected]]
> Sent: Friday, June 3, 2016 6:54 PM
> To: Paolo Mantegazza; '[email protected]'
> Subject: RE: rt_make_soft_real_time hangs my process
>
> Its running on a video processing software, and we're doing it to ensure our image analysis routine is not interrupted.
>
> Each time a new frame arrives from the camera, it goes hard, processes the frame, and returns soft.  Framerates range from 100 to ~9000 fps depending on ROI.
>
> I think I might know what's going on though.  The process that this runs on creates a separate thread for acquiring the frames from the camera with its affinity set so that it runs on a separate core, and the realtime part is only for the processing of a frame after it arrived.  Both functions run under the same process though (ie. ps aux only shows one item), and so I think that might be my problem.  Can I even legally call rt_make_hard_real_time() on a multithreaded process?
>
> -Duane
>
> -----Original Message-----
> From: Paolo Mantegazza [mailto:[email protected]]
> Sent: Friday, June 03, 2016 9:59 AM
> To: Duane AW. Jeffery <[email protected]>; '[email protected]' <[email protected]>
> Subject: RE: rt_make_soft_real_time hangs my process
>
> Can you provide the frequency of the hard7soft switches?
> If you a doing it for a controlled approach to Linux syscalls , try letting RTAI take care of it and see what happens. In such a view you might even try  using a Linux  server .
> That said, eventually this is the case in which it should not be too complex to set a neutral test up, so that I cantry to diagnose the problem you are wittnessing.
>
> Paolo
> ________________________________________
> From: Rtai [[email protected]] on behalf of Duane AW. Jeffery [[email protected]]
> Sent: Friday, June 3, 2016 3:12 PM
> To: '[email protected]'
> Subject: [Rtai] rt_make_soft_real_time hangs my process
>
> Hi all,
>
> I'm running a shiny new system on Linux 4.1.18 w/ RTAI 5 from vulcano, and it seems to be going well.  (System is derived from a Slackware 14.1 install)
>
> I seem to be having a small problem with one of my applications where it calls into rt_make_soft_real_time() which never returns.  It previously calls rt_make_hard_real_time(), and calls the pair in a loop, so it switches back and forth fairly frequently.
>
> Any ideas?
>
> -Duane
> _______________________________________________
> Rtai mailing list
> [email protected]
> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai