Re: Xenomai 4 Allocator Fragmentation Questions

Philippe Gerum <[email protected]> writes:

> Kevin Strell <[email protected]> writes:
>
> >
> > 1. Other than the compaction procedure, is there anything we can do
> > with the default Xenomai allocator to improve the performance in
> > regards to fragmentation?
>
> Use multiple heaps if possible, to confine the problematic allocation
> patterns.
>
> > 2. When is the correct time to run the compaction procedure in regards
> > to when we should call evl_init()?
>
> Those are unrelated issues.
>
> > 3. If the fragmentation is a known issue of the default allocator, is
> > there any other allocator that is recommended?
> >
>
> An allocator which meets real-time requirements while still limiting
> external fragmentation to the minimum may be difficult to find because
> both goals are somewhat conflicting (notably because garbage collection
> may not fit well when it comes to time complexity). You may want to have
> a look at respins of the TLSF allocator, some of them were designed to
> address the fragmentation issue the original one had - which was the
> reason for developing heapmem years ago.
>
> I'm interested by any result in this area, feedback welcome.

--

> Kevin Strell <[email protected]> writes:
>
> > We have been able to avoid the fragmentation problems by switching to
> > using the rpmalloc library (https://github.com/mjansson/rpmalloc)
>
> A note regarding this allocator, which documentation reads as follows
> under the "Worst case scenarios" section:
>
> "Since each heap maps a span of memory pages per page type, a thread
> that allocates just a few blocks of each size class (16, 32, ...) for
> many size classes will commit a memory page for each used size class,
> while only using a small fraction of the committed memory. However,
> memory pages are committed on demand and blocks are initialized only as
> needed, ..."
>
> Assuming that "committing pages" may mean mapping them to the current
> address space and/or performing some kind of memcontrol work (msync?),
> you may want to make sure that rpmalloc can pre-commit pages, so that
> this does not happen in time-critical work loops. Because rpmalloc
> operates on a per-thread basis, you would need to ensure this for each
> thread attached to the evl core specifically. Failing to do so would
> certainly cause those real-time threads to be demoted to the in-band
> stage. You may want to check this using the related health monitoring
> service [1] (EVL_HMDIAG_SYSDEMOTE).
>
> To sum up, rpmalloc addresses the fragmentation issue by dedicating each
> memory page to dealing with a particular block size, for allocation
> _and_ release, so it does not have to resort to dynamic garbage
> collection. However, in order to meet real-time requirements, you may
> want to make sure that alloc and release operations won't ever issue a
> regular system call under the hood.
>
> [1] https://v4.xenomai.org/core/user-api/thread/index.html#health-monitoring

Thank you for your feedback.

We originally had a process that used a single large heap that was
shared among multiple threads:

* Main thread: allocates upon initialization only.
* Lower-priority thread A: allocates/frees during normal operation.
* Lower-priority thread B: allocates/frees during normal operation.

Memory was allocated using the non-serializing/unlocked libevl heap
API, which was obviously problematic when threads A and B run
concurrently. We have historically avoided the use of mutexes and
similar mechanisms, so we attempted to continue using the unlocked API
with a large heap for the main thread and a smaller heap for one of the
lower-priority threads. The smaller heap was much more likely to be
fully consumed, which is where our concerns about fragmentation came
from.

When reverting to use a single large heap and the serializing libevl
heap API, we haven't seen any issues with memory allocation nor any
concerns about using mutexes. We had to use a small workaround because
there are multiple processes that each use their own `struct evl_heap`,
and the name of the mutex created internally by evl_init_heap() is only
unique within the same process (and not between multiple processes).

 struct evl_heap heap;
 ret = evl_new_mutex(&heap.lock, "heap:%.3d", getpid())
 /* ... */
 ret = evl_init_heap_unlocked(&heap, mem, size);

rpmalloc did end up meeting real-time performance requirements with
the evaluation that we had done, but I believe that we are going to
abandon this evaluation since the libevl serialized heap API works well
for our use case.

Regards,
Bryan