Re: L4Re: region mapper/manager concurrency considerations and resource limits

On Wednesday, 24 February 2021 11:40:15 CET Philipp Eppelt wrote:
> 
> I'm always happy to read from you.

That's nice to know!

> Dataspaces and regions are two different things. The Dataspace is a
> piece of memory, which you can divide up and map to different places.
> You tell the region manager the goto-dataspace in case of a page fault
> and the region manager will map a piece (aka region) from the dataspace
> to the faulting address.

This seems to be consistent with my current understanding. When I talk about 
detaching dataspaces, it is just my lazy shorthand for talking about removing 
the association between the memory region situated at a given base address and 
the object acting as a dataspace.

Of course, upon a page fault, the dataspace presents a flexpage to the region 
manager to satisfy an access for some, if not all, of the region associated 
with the dataspace. Here, I interpret the span of any given flexpage as being 
part of a region, with an entire region corresponding to the entire span of 
memory associated with a dataspace. I hope that is correct.

> You can have multiple regions in the region manager referring to the
> same dataspace as backing memory. The region manager stores a start and
> size in your virtual address space, a dataspace cap and an offset into
> this dataspace, which corresponds to the start of your region.

OK. In my case, I do have multiple regions but they tend to reference 
different dataspaces. However, these dataspaces may "export" the same memory 
pages. Something like this:

(a1, s1) -> d1 -> mem[o1:o1+s1]
(a2, s2) -> d2 -> mem[o2:o2+s2]

Here, d1 and d2 are provided by the same task. Offsets o1 and o2 are 
configured for the dataspaces in advance. Obviously, any offsets within the 
mapped regions are applied within the defined memory ranges.

I don't actually specify offsets into dataspaces when attaching them, but I 
could imagine that this would just change the offset parameter involved in map 
requests. This is different to my strategy where each dataspace is configured 
to export a particular region of something resembling a file.

> Of course you can have multiple entries in the region mapper referring
> to the same dataspace and even the same offset.

I feel that this might be a useful alternative to having multiple dataspaces 
providing different "spans" within a file.

> Now the question is: What happens if you detach a region from the region
> mapper?
> You need to specify just the address or the address and a size to detach
> a single region.
> In case of just an address you can use any address within a region to
> unmap it. So this affects just a single region entry in the region manager.
> In case of the address-size call you can detach multiple regions at once.
> In the process, the region's memory is unmapped from your local address
> space.

If I understand this correctly, detaching a region that is being serviced by a 
dataspace will cause the virtual memory associations for addresses within that 
region to be invalidated. And just specifying the base address of that region 
should be sufficient to detach the region in its entirety.

> So if I understand your case correctly, you attach a dataspace to some
> region (a1, s1) -> (d1, o1) and later unmap this single region. This
> should unmap all mappings between a1 and a1+s1.

This is what I would expect, yes. I detach the region...

l4re_rm_detach(a1)

...or...

l4re_rm_detach_unmap(a1, L4RE_THIS_TASK_CAP)

...and I also call l4re_util_cap_free_um on the dataspace capability. However, 
I am not completely familiar with the difference between l4re_util_cap_free 
and l4_util_cap_free_um.

> However, if there are other regions attached, e.g. (a2, s2) -> (d1, o2),
> this will still remain and as soon as you unmap the d1-capability, you
> have stale entries in your region map.

What happens when a task tries to access the memory within a2 to a2+s2? Are 
there virtual memory associations that may still provide access to the memory 
exported by the now-unmapped capability?

> To ensure you detach all regions in an area you can use detach(start,
> size, ds, task), however, I just noticed there is no C-version provided
> for the corresponding C++ interface.

For my purposes, the size parameter would not be required if there is still a 
way to detach the entire region.

> To answer your question: If you invoke detach on a specific region, all
> mappings in this and only this region are unmapped. Other regions served
> by the same dataspace are not touched.

My probably-incomplete mental model is that when my dataspaces send flexpage 
items in response to map requests, there is something that establishes a 
specific association between the eventual region and the memory being exported 
by a dataspace. So, in response to...

map(offset, hot_spot, flags)

...we somehow end up with...

(a1+offset, flexpage_size) -> mem[o1+offset:o1+offset+flexpage_size]

And I guess that there is a specific way for the region manager to request 
that this mapping be removed without other mappings to the same memory being 
affected.

(Meanwhile, I know from previous discussions and experimentation that I can 
indeed invalidate mappings for a given area of exported memory from within a 
dataspace, affecting all involved regions.)

> I'm still curious about your setup, because your experience seems to
> differ. Can you give me some info about your attach calls?

I tend to perform the attach operation as follows:

l4re_rm_attach(addr, size, L4RE_RM_SEARCH_ADDR, ds, 0, L4_PAGESHIFT)

This then yields a suitable address, as I would expect. So, I attach regions 
and eventually end up with something like this:

(a1, s1) -> d1 -> mem[o1:o1+s1]

Then I detach the region and attach another, getting the same base address:

(a1, s2) -> d2 -> mem[o2:o2+s2]

But it seems as if the memory accessible via the new region still exposes 
pages via the old mapping. This does not happen all the time, but only under 
certain conditions that I am trying to characterise.

I also saw it with a region that overlapped the old one instead of having 
precisely the same base address:

(a1+0x1000, s2) -> d2 -> mem[o2:o2+s2]

Here, an access to the new base of a1+0x1000 appeared to expose mem[o1+0x1000] 
instead of mem[o2].

> I hope this sheds some light and my assumptions on what you are doing
> are not too far fetched.

I worry that what I am doing is actually the thing that is far fetched! It 
remains possible, as always, that I am doing something wrong and overlooking 
it. At the same time, I want to make sure that my understanding and 
expectations are correct.

Thanks for following up!

Paul