Re: L4Re: region mapper/manager concurrency considerations and resource limits

Hi Paul,


On 2/9/21 1:20 AM, Paul Boddie wrote:
> On Monday, 8 February 2021 09:54:08 CET Christian Ludwig wrote:
>>
>> On Sunday, 07.02.2021, 01:54 +0100 Paul Boddie wrote:
>>
>>> What considerations need to be given to concurrency with regard to
>>> the use of the l4re_rm_attach function and similar functionality? Is
>>> access to the region mapper/manager from different threads in the same
>>> task advisable, are there thread safety issues, and if so, can these be
>>> sensibly worked around?
>>
>> Calls to the region manager are IPC calls and are synchronized by a
>> wait-queue in the kernel. The region manager thread only serves one
>> client at a time. In fact if you use multiple threads in a task it
>> serves all page-faults for all threads already. It's thread-safe by
>> design.
> 
> OK. I wasn't really sure about the concurrency aspects of the region manager 
> itself. Some frustrating experience with my own code led me to question this 
> rather than just take it for granted.
> 
>> However, the capability slot allocator (l4re_util_cap_alloc) is not
>> thread-safe. You need to introduce some synchronization if you use it
>> from multiple threads.
> 
> This is very useful to know. I imagine that thread safety is regarded as 
> unnecessary for many use-cases, but it does surprise me that it has to be 
> introduced to make the allocator usable in a multithreaded program. Then 
> again, I might be doing things that are not particularly normal for L4Re 
> programs.

I agree that an out-of-the-box mechanism is helpful here and will
prevent headaches in many situations, plus you wouldn't need to know
about this. However, the (internal) discussion did not yet reach consensus.

Once you know about this shortcoming, a thread-safety wrapper around
cap_alloc is fortunately straight forward.


> What kind of resource limits apply to L4Re tasks? In my work, I am allocating 
> capabilities for objects acting as dataspaces. Things seem to work fairly well 
> up to a certain point, and then capability allocation seems to fail, and 
> program failure is then not particularly graceful.
> 
> (I currently observe that creating 160 threads, each with its own IPC gate 
> capability seems to be unproblematic, but more than this will produce errors 
> that manifest themselves either in pthread library code or in my own code 
> where a critical section is supposed to be enforced.)

It looks like there is a capability limit per task at 4096 for
L4Re::Util::cap_alloc.
160 thread (2 caps) + 160 IPC gates = 3*160 -> 480 caps

If you are handing out more than 3600 Dataspaces you might be running
into this limit.
Now I'm fuzzy on the details, so I'll just provide a general direction.
I don't know if you can replenish the Util::cap_alloc by providing
additional memory to it or if you need to setup your own allocator (see
l4re-core/l4re/util/libs/cap_alloc.cc). For the latter, you can set it
up with a bigger piece of memory, to increase the number of manage
capability slots.


On 2/7/21 1:54 AM, Paul Boddie wrote:
> I might also ask about support in L4Re for C++ threading constructs. When 
> developing my simulation, I decided to use the standard C++ support for 
> threading, mutexes, condition variables, and so on, as opposed to using 
> pthread functionality. I imagine that C++ library support for these things 
> just wraps the pthread functionality, but I wonder if there are not other 
> considerations that I might need to be aware of.

You are right, std::thread wraps libpthread. I guess you know about
l4re-core/libstdc++-headers/include/thread-l4?

To my knowledge the behavior for the C++ constructs don't differ from
the pthread ones. Be careful with std::thread.join(), the thread needs
to cooperate and terminate/exit by itself. If it doesn't, your caller
waits for a long time.

If you directly interact with the UTCB or use low-level functions which
setup the UTCB, make sure to not do any other function calls, as they
might change the UTCB. (That's also a high scorer in the headache list.)



Cheers,
Philipp