Re: rt_make_soft_real_time hangs my process

It seems that you have more or less understood my problem, but for some reason (unless I'm misunderstanding what you mean), my problem doesn't seem to fit with your description of how it should work.

This program has been working (and in continuous development through a number of developers) for quite some time (read: almost 10 years) in a selectable approach.  We can compile it for realtime or non-realtime (by setting preprocessor directives to include/exclude <rtai_lxrt.h> and the various rt_... calls).  The result, was that both the RT version and the non-RT version function as expected, but the non-RT version sometimes drops frames due to the calculations getting interrupted.  Naturally, that is an unwanted occurrence, so we prefer to use the RT-compiled version.

This seemed to work OK, right up until we added support for a new camera which required its own acquisition thread.  At that point, the new camera's version would only run properly when compiled non-RT.  The version compiled for RT hangs the machine.  Because the division of labour bought some extra cycles, it didn't seem to matter too much that the new camera wasn't running RT, and so we didn't worry about it at the time, choosing to just run the non-RT version for that camera.

More recently, we added a computationally expensive feature which required its own calculation thread on all cameras in order to not bog down the main calculations, and when we did that, it broke the RT support for all cameras.  This is what lead me to believe the hangs are related to the threads, and ultimately to this discussion.

Perhaps we're just not instantiating the threads correctly?  Or doing something otherwise not allowed.  These processes also use shared memory and cpusets, so it's a bit tough to know which exact piece of advanced process control might be breaking things.

I will try to more fully describe the architecture, and hopefully someone on the mailing list knows something about where I might look for the problem.

We have:
Process A: a non-RT user-space GTK2-based X app for display and control - The user interacts directly with this process.
Process B: an optionally RT user-space daemon which handles calculations and camera acquisition - this process runs in the background, controlled by the user's actions via Process A.

Process A and Process B communicate through a shared memory segment.  Process B is the process which can be compiled for RT or non-RT.

Process B has 3 threads.
Thread 1 is its main calculation thread.  For 2 of the 4 supported cameras, this thread also acquires frames from the camera.  When using these two cameras, the RT version worked until we added Thread 3 a couple weeks ago.
Thread 2 is the acquisition thread for the other 2 cameras.
Thread 3 is the calculation thread for our new feature. 

The sequence of operation goes like this:

System is a 6-core system started with Kernel param isolcpus=2-5
RTAI is insmod'd with IsolCpusMask = 0x3C
The user starts Process A from their desktop.
Process A initializes itself, and tries to connect to the shared memory segment.  
Process A fires a system() call to start Process B and redirect its stdout to a log file.  
Process A fires a second system() call to migrate Process B into the desired cpuset (which was created at boot time by a script).
Process B initializes itself (in Thread 1), and connects to the shared memory segment
RT CODE:
Process B has a global RT_TASK mytask; 
Process B calls rt_allow_nonroot_hrt();
Process B calls mytask = rt_task_init_schmod(nam2num("HRTSK"), 2, 0, 0, SCHED_FIFO, 0x02);  (Is this perhaps the problem because CPU Mask 0x02 isn't part of the 0x3C from the insmod?)
END RT CODE
Both processes handshake through the shared memory segment, and then Process A notifies Process B to initialize the camera.
Process B Thread 1 calls into an init camera routine
Process B Thread 1 finds and opens a link to the camera.
Process B Thread 1 starts Thread 2, and the 2 threads communicate through global variables.  
Process B Thread 1 starts the stream from the camera
Process B Thread 2 sees the Camera Started flag, and starts acquiring frames from the camera's non-RT driver and storing them in memory.
Process B Thread 1 returns from the init camera routine, and begins initializing the new feature
Process B Thread 1 starts Thread 3, also communicating with Thread 1 through global variables.
RT CODE:
Process B Thread 1 checks rt_is_hard_timer_running()
If no, then it calls rt_set_oneshot_mode(); and start_rt_timer(0);
Process B Thread 1 calls rt_grow_and_lock_stack(40000);
END RT CODE
From this point on, Process B Thread 1 checks the variables shared with Thread 2 to determine if a new frame has arrived.
On arrival of a new frame, Process B Thread 1 calls rt_make_hard_real_time(); and rt_task_use_fpu(mytask, 1);
Process B Thread 1 computes the main result and notifies Thread 3 to compute its result for the new feature.
Process B Thread 1 fills up the structures in the shared memory segment to return its data to Process A
Process B Thread 1 calls rt_make_soft_real_time();
Process B Thread 1 loops to checking for new frames.
When the user commands a shutdown, Process B Thread 1 calls rt_task_delete(mytask); and if it started the timer, additionally calls stop_rt_timer();

Any ideas anyone has to what I might try are certainly welcome.  I've tried to trace where it locks up with printf's into Process B's stdout, and it seems to be stalling on the call to rt_make_soft_real_time();

-Duane

-----Original Message-----
From: Paolo Mantegazza [mailto:[email protected]] 
Sent: Friday, June 03, 2016 4:58 PM
To: Duane AW. Jeffery <[email protected]>; '[email protected]' <[email protected]>
Subject: RE: rt_make_soft_real_time hangs my process

Trivially recalling that Linux threads are substantially processes sharing a memory space by default, every thread of a process can be made hard-soft as you like, under RTAI.
So if you have a thread that in Linux's hands to acquire frames through a not hard real time RTAI driver and want that as soon as a complete frame has been acquired it is  process in hard real time immediately you have just to enable (using rt_thread_init or rt_task_init_schmod) the acquisition LINUX thread to using RTAI functions, mating it to another hard real time hardened RTAI thread that waits for being awaken by the acuisition thread. That done define a multiple frames buffer, let the acquisition frame acquire frames continuously passing  an available  frame index to the hard real time thread. The mechanisms to be used can be just a semephore with the index being a per process variable, a suspend-resum,  an rpc_receive with the index passed through and rt_send.
Hoping I undertood your proble, sorry if my suggestions were trivial stuff.

Paolo

________________________________________
From: Duane AW. Jeffery [[email protected]]
Sent: Friday, June 3, 2016 6:54 PM
To: Paolo Mantegazza; '[email protected]'
Subject: RE: rt_make_soft_real_time hangs my process

Its running on a video processing software, and we're doing it to ensure our image analysis routine is not interrupted.

Each time a new frame arrives from the camera, it goes hard, processes the frame, and returns soft.  Framerates range from 100 to ~9000 fps depending on ROI.

I think I might know what's going on though.  The process that this runs on creates a separate thread for acquiring the frames from the camera with its affinity set so that it runs on a separate core, and the realtime part is only for the processing of a frame after it arrived.  Both functions run under the same process though (ie. ps aux only shows one item), and so I think that might be my problem.  Can I even legally call rt_make_hard_real_time() on a multithreaded process?

-Duane

-----Original Message-----
From: Paolo Mantegazza [mailto:[email protected]]
Sent: Friday, June 03, 2016 9:59 AM
To: Duane AW. Jeffery <[email protected]>; '[email protected]' <[email protected]>
Subject: RE: rt_make_soft_real_time hangs my process

Can you provide the frequency of the hard7soft switches?
If you a doing it for a controlled approach to Linux syscalls , try letting RTAI take care of it and see what happens. In such a view you might even try  using a Linux  server .
That said, eventually this is the case in which it should not be too complex to set a neutral test up, so that I cantry to diagnose the problem you are wittnessing.

Paolo
________________________________________
From: Rtai [[email protected]] on behalf of Duane AW. Jeffery [[email protected]]
Sent: Friday, June 3, 2016 3:12 PM
To: '[email protected]'
Subject: [Rtai] rt_make_soft_real_time hangs my process

Hi all,

I'm running a shiny new system on Linux 4.1.18 w/ RTAI 5 from vulcano, and it seems to be going well.  (System is derived from a Slackware 14.1 install)

I seem to be having a small problem with one of my applications where it calls into rt_make_soft_real_time() which never returns.  It previously calls rt_make_hard_real_time(), and calls the pair in a loop, so it switches back and forth fairly frequently.

Any ideas?

-Duane
_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai