Re: rt_make_soft_real_time hangs my process

"When using these two cameras, the RT version worked until we added
Thread 3 a couple weeks ago."

So thread 3 is not even real-time, but introducing it causes a system
hang? Did you perhaps add some synchronization mechanism between
thread 1 and thread 3 also? And perhaps that mechanism is an RTAI
mutex/semaphore/etc?

On Mon, Jun 6, 2016 at 12:29 PM, Duane AW. Jeffery
<[email protected]> wrote:
> Sorry for the confusion Shahbaz,
>
> I was merely trying to delineate stuff in the list that was actual code pertaining to the RT calls, as opposed to just acquisition and calculation code.  You are correct, the calculation thread (Process B, Thread 1) is supposed to be an RTAI task through the bulk of its lifecycle, except during init and cleanup.  It switches between hard and soft when it does the calculations.  It is the only thread of the three that attempts to go real time.
>
> I will try swapping in some rt_printk's for the printf's I was using to try and locate the hang.  Good idea.
>
> -Duane
>
> -----Original Message-----
> From: Shahbaz Youssefi [mailto:[email protected]]
> Sent: Monday, June 06, 2016 12:08 PM
> To: Duane AW. Jeffery <[email protected]>
> Cc: Paolo Mantegazza <[email protected]>; [email protected]
> Subject: Re: [Rtai] rt_make_soft_real_time hangs my process
>
> See inline:
>
> On Mon, Jun 6, 2016 at 11:24 AM, Duane AW. Jeffery <[email protected]> wrote:
>>
>> System is a 6-core system started with Kernel param isolcpus=2-5 RTAI
>> is insmod'd with IsolCpusMask = 0x3C The user starts Process A from
>> their desktop.
>> Process A initializes itself, and tries to connect to the shared memory segment.
>> Process A fires a system() call to start Process B and redirect its stdout to a log file.
>> Process A fires a second system() call to migrate Process B into the desired cpuset (which was created at boot time by a script).
>> Process B initializes itself (in Thread 1), and connects to the shared
>> memory segment RT CODE:
>> Process B has a global RT_TASK mytask; Process B calls
>> rt_allow_nonroot_hrt(); Process B calls mytask =
>> rt_task_init_schmod(nam2num("HRTSK"), 2, 0, 0, SCHED_FIFO, 0x02);  (Is
>> this perhaps the problem because CPU Mask 0x02 isn't part of the 0x3C
>> from the insmod?) END RT CODE
>
> Why is this called END RT CODE? Aren't the threads run in real-time?
> Have you verified that your mutexes (and other locks, if any) are of the RTAI type, and that the threads are switched to real-time?
>
>> Both processes handshake through the shared memory segment, and then Process A notifies Process B to initialize the camera.
>> Process B Thread 1 calls into an init camera routine Process B Thread
>> 1 finds and opens a link to the camera.
>> Process B Thread 1 starts Thread 2, and the 2 threads communicate through global variables.
>> Process B Thread 1 starts the stream from the camera Process B Thread
>> 2 sees the Camera Started flag, and starts acquiring frames from the camera's non-RT driver and storing them in memory.
>> Process B Thread 1 returns from the init camera routine, and begins
>> initializing the new feature Process B Thread 1 starts Thread 3, also communicating with Thread 1 through global variables.
>> RT CODE:
>> Process B Thread 1 checks rt_is_hard_timer_running() If no, then it
>> calls rt_set_oneshot_mode(); and start_rt_timer(0); Process B Thread 1
>> calls rt_grow_and_lock_stack(40000); END RT CODE From this point on,
>> Process B Thread 1 checks the variables shared with Thread 2 to determine if a new frame has arrived.
>> On arrival of a new frame, Process B Thread 1 calls
>> rt_make_hard_real_time(); and rt_task_use_fpu(mytask, 1); Process B Thread 1 computes the main result and notifies Thread 3 to compute its result for the new feature.
>> Process B Thread 1 fills up the structures in the shared memory
>> segment to return its data to Process A Process B Thread 1 calls
>> rt_make_soft_real_time(); Process B Thread 1 loops to checking for new frames.
>> When the user commands a shutdown, Process B Thread 1 calls
>> rt_task_delete(mytask); and if it started the timer, additionally
>> calls stop_rt_timer();
>>
>> Any ideas anyone has to what I might try are certainly welcome.  I've
>> tried to trace where it locks up with printf's into Process B's
>> stdout, and it seems to be stalling on the call to
>> rt_make_soft_real_time();
>
> Just a side note, try `rt_printk` instead of `printf`, and check `dmesg`. `printf` would take your task out of hard real-time, which is likely not what you want. Also, if the system hangs, you may not see all the `printf` logs, but you have a better chance with `rt_printk`.
> You can have an always-on-top terminal running:
>
>     tail -f /var/log/messages
>
> for example to show you the kernel logs as they happen. Adding a small busy loop after each `rt_printk` would help make sure you see the log before the hang.
>
>>
>> -Duane
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: Friday, June 03, 2016 4:58 PM
>> To: Duane AW. Jeffery <[email protected]>; '[email protected]'
>> <[email protected]>
>> Subject: RE: rt_make_soft_real_time hangs my process
>>
>> Trivially recalling that Linux threads are substantially processes sharing a memory space by default, every thread of a process can be made hard-soft as you like, under RTAI.
>> So if you have a thread that in Linux's hands to acquire frames through a not hard real time RTAI driver and want that as soon as a complete frame has been acquired it is  process in hard real time immediately you have just to enable (using rt_thread_init or rt_task_init_schmod) the acquisition LINUX thread to using RTAI functions, mating it to another hard real time hardened RTAI thread that waits for being awaken by the acuisition thread. That done define a multiple frames buffer, let the acquisition frame acquire frames continuously passing  an available  frame index to the hard real time thread. The mechanisms to be used can be just a semephore with the index being a per process variable, a suspend-resum,  an rpc_receive with the index passed through and rt_send.
>> Hoping I undertood your proble, sorry if my suggestions were trivial stuff.
>>
>> Paolo
>>
>> ________________________________________
>> From: Duane AW. Jeffery [[email protected]]
>> Sent: Friday, June 3, 2016 6:54 PM
>> To: Paolo Mantegazza; '[email protected]'
>> Subject: RE: rt_make_soft_real_time hangs my process
>>
>> Its running on a video processing software, and we're doing it to ensure our image analysis routine is not interrupted.
>>
>> Each time a new frame arrives from the camera, it goes hard, processes the frame, and returns soft.  Framerates range from 100 to ~9000 fps depending on ROI.
>>
>> I think I might know what's going on though.  The process that this runs on creates a separate thread for acquiring the frames from the camera with its affinity set so that it runs on a separate core, and the realtime part is only for the processing of a frame after it arrived.  Both functions run under the same process though (ie. ps aux only shows one item), and so I think that might be my problem.  Can I even legally call rt_make_hard_real_time() on a multithreaded process?
>>
>> -Duane
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: Friday, June 03, 2016 9:59 AM
>> To: Duane AW. Jeffery <[email protected]>; '[email protected]'
>> <[email protected]>
>> Subject: RE: rt_make_soft_real_time hangs my process
>>
>> Can you provide the frequency of the hard7soft switches?
>> If you a doing it for a controlled approach to Linux syscalls , try letting RTAI take care of it and see what happens. In such a view you might even try  using a Linux  server .
>> That said, eventually this is the case in which it should not be too complex to set a neutral test up, so that I cantry to diagnose the problem you are wittnessing.
>>
>> Paolo
>> ________________________________________
>> From: Rtai [[email protected]] on behalf of Duane AW. Jeffery
>> [[email protected]]
>> Sent: Friday, June 3, 2016 3:12 PM
>> To: '[email protected]'
>> Subject: [Rtai] rt_make_soft_real_time hangs my process
>>
>> Hi all,
>>
>> I'm running a shiny new system on Linux 4.1.18 w/ RTAI 5 from vulcano,
>> and it seems to be going well.  (System is derived from a Slackware
>> 14.1 install)
>>
>> I seem to be having a small problem with one of my applications where it calls into rt_make_soft_real_time() which never returns.  It previously calls rt_make_hard_real_time(), and calls the pair in a loop, so it switches back and forth fairly frequently.
>>
>> Any ideas?
>>
>> -Duane
>> _______________________________________________
>> Rtai mailing list
>> [email protected]
>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai