Re: Creating tasks and the l4_task_map function

Adam,

Thanks for the reply again!

On Monday, 18 April 2022 23:26:03 CEST you wrote:
> Hi Paul,
> 
> On Tue Apr 12, 2022 at 01:09:40 +0200, Paul Boddie wrote:
> > 
> > OK, I did see that function being used, too, but I also found plenty of
> > other things in my perusal of the different files. Obviously, being able
> > to extend the UTCB memory is an important consideration.
> 
> It is, because, for example, one might not know how many threads a task
> will have, especially the component that creates the task.

Right. And so, in the Moe_app_model the size of the UTCB is dependent of the 
default number of threads. It still seems that I have to provide a UTCB 
flexpage to l4_factory_create_task, however.

[...]

> > Right. I see that the factory function actually sends the flexpage in the
> > IPC call (using l4_factory_create_add_fpage_u), thus mapping it in the
> > task. I find it hard to follow where this message is actually handled (I
> > presume that Moe acts as the factory) or what the factory actually does
> > with the flexpage, but I presume that it ultimately causes it to be
> > mapped in the new task.
>
> It is handled in Fiasco.

OK. I think I found this in Task::create (src/kern/task.cpp).

[...]

> > However, I wonder about the "chicken and egg" situation in new tasks. It
> > seems to me that the way things work is that a new task in L4Re is
> > typically populated with the l4re binary containing the region
> > mapper/manager (RM). This seems to be initiated here (in launch_loader):
>
> Yes, the l4re binary loads the application and then serves as its pager.

OK. And it seems that the RM provided by this binary is able to indicate the 
receive window for flexpages when asking for mappings from dataspaces.

[Pagers for other tasks]

> That's how it works. Moe also has region managers that are used for the
> l4re binary to be paged. When a page fault is resolved, then there is
> someone sending memory via a flexpage to the task in question. In our
> case it's the dataspace manager which sends the memory via an 'map'
> call. Here it does not matter whether the l4re binary faulted or the
> application, because in the end the task is the receiver of the flex
> page, not the particular application (which are runnig in the same
> task).

So, the one thing I didn't understand until I started digging around in the 
Fiasco sources and also implementing my own page fault handler was the scope 
of the receive window for issued flexpages, but it appears that the whole 
address space is indicated as the receive window in 
Thread::handle_page_fault_pager (src/kern/thread-ipc.cpp).

[Mapping memory using l4_task_map]

> Jdb has facilities to check how the address spaces look like, exactly to
> debug issue like you describe. You can press 's' to see all the tasks in
> the system, navigate onto them, and then press 'p' to see the page-table
> view. Here you can navigate the page tables and verify that the pages at
> some virtual address is actually pointing to the physical location they
> should point at. For a particular physical address (page frame number)
> you can also show the mapping hierarchy via the 'm'.

The "coherency" problems actually turned out to be me forgetting the 
appropriate alignment for the mapped flexpages. But I have discovered a few 
things about jdb in my attempts to troubleshoot my code, including 's' to look 
at tasks. I found the page table view bewildering, though, rather hoping for a 
nice summary of mapped pages instead. However, the object space view was very 
useful in establishing that capabilities were being mapped.

I did manage to get l4_task_map to work between two fully initialised tasks 
created by Ned as follows:

l4_fpage_t payload_fpage = l4_fpage((l4_addr_t) buf,
                                    l4util_log2(L4_PAGESIZE * NUM_PAGES),
                                    L4_FPAGE_RW);

l4_task_map(recipient, L4RE_THIS_TASK_CAP, payload_fpage, 0x3000000);

This permitted the recipient task to access the memory in the appropriately 
aligned buf region, this being mapped into the region starting at 0x3000000 in 
the recipient.

I also managed to achieve the same thing using IPC between the two tasks 
instead, having the recipient indicate a receive window in the buffer 
registers as follows:

br[0] = l4_map_control(0, 0, L4_MAP_ITEM_MAP);
br[1] = l4_fpage(0x3000000, l4util_log2(L4_PAGESIZE * NUM_PAGES), 0).raw;

And then constructing the send flexpage as follows:

mr[0] = l4_map_control(0, L4_FPAGE_CACHEABLE, 0);
mr[1] = l4_fpage((l4_addr_t) buf, l4util_log2(L4_PAGESIZE * NUM_PAGES),
                 L4_FPAGE_RW).raw;

Alternatively, setting the receive window to the entire address space...

br[1] = l4_fpage_all().raw;

...and indicating a different send base also worked:

mr[0] = l4_map_control(0x3000000, L4_FPAGE_CACHEABLE, 0);

This approximating to the l4_task_map scenario.

However, trying this with a completely newly created task, I cannot seem to 
get l4_task_map to map memory into the task, and my page fault handler does 
not seem to be able to respond to a page fault message with such a flexpage 
and have the request satisfied. This means that there is some detail I am 
overlooking, but I have yet to determine what it is.

One thing I have tried to do is to get Fiasco to report what it is doing when 
processing page fault messages, but this is quite challenging. Really, I just 
want to establish whether the reply to a page fault message gets interpreted 
correctly and causes mappings to be established in the page tables.

Doing some "old school" tracing in various routines like transfer_msg_items by 
detecting the appropriate fault conditions, enabling tracing, and then 
producing output on the console doesn't seem to yield any indications of the 
messages being processed, but perhaps I misunderstand the flow of control from 
Thread::handle_page_fault_pager within the kernel when the IPC is initiated.

Reviewing the old thread on this broader topic, I found this advice:

http://os.inf.tu-dresden.de/pipermail/l4-hackers/2014/015441.html

This does yield page fault trace log entries of the following form:

pf:  00bc pfa=0000000001000ae3 ip=0000000001000ae3 (rp) spc=0xffffffff13dc5ad8 
err=15

Here, I presume that the error is R_aborted (src/abi/l4_error.cpp), meaning 
that the page fault was not handled.

Looking at advice about IPC tracing...

http://os.inf.tu-dresden.de/pipermail/l4-hackers/2014/015475.html

...I can also get log entries that I think might indicate some element of 
success in terms of the messages being sent, with this being the fault 
message:

     00be answ [fffffffffffe0002] L=8000fcf3 err=0 (OK) (1000ae5,1000ae3)
ipc: 00be wait->[C:INV] DID=be L=0 TO=INF

And elsewhere:

     00be answ [00000040] L=0 err=0 (OK) (1000038,400595)
ipc: 00be send rcap->[C:INV] DID=bc L=0 [0000000000000040]
               (0000000001000038,0000000000400595) TO=INF

Here, I am attempting to resolve the page fault caused by execution at 
0x1000ae3 by sending a flexpage providing memory in one task at 0x400000 to 
the recipient at 0x1000000.

None of this really explains why the page fault handler keeps getting called 
with the same details, sending the same message, and so on.

[...]

> For sure this is an area where the code is pretty involved. Back in the
> old days we had an IDL compiler that grew and grew and was in the end
> not easy to maintain. When we switched to the capability system and thus
> changing all the APIs we had the choice of whether adapting the IDL
> compiler or doing something different. Back then it was a major hassle
> to parse the input because eventually one wants to have the whole
> language understood by the IDL compiler to be able to use all sorts of
> types (of course one could make compromises there). With C / C++ that is
> not so easy, at least back then. Now there's LLVM and that's a major
> improvement in this area. Still the actual tool needs to be implemented
> and maintained. Now, as we all see, we have opted for the "do something
> else" option. With C++ as our main language and the possibilities with
> it, there was the idea to implement the "IDL thing" purely with C++
> directly in the code. That's what we have now. For me, all the code
> around it is the IDL compiler, and abstractly, if it would not be in
> header files it would sit somewhere else but it would be there in one
> form or another.

The unfortunate thing about the "do something else" solution is that it 
becomes difficult to determine what the interfaces are for components: the 
details are all presumably present, but they are encoded in ways that are not 
particularly readable. That might not seem to matter if the existing classes 
are usable and readily understandable, but my own experience was that I found 
myself trying to understand the low-level details before I could hope to 
understand why the C++ API did things in a particular way, which is really the 
wrong way round. I must have spent hours staring at Gen_fpage and related 
types in one way or another.

> > This might just sound like me complaining, but I also have some concerns
> > about being able to verify the behaviour of some of the code. For
> > example, I recently found that my dataspace implementation was getting
> > requests from a region mapper/manager with an opcode of 0x100000000,
> > which doesn't make any sense to me at all, given that the dataspace
> > interface code in L4Re implicitly defines opcodes that are all likely to
> > be very small integers. At first I obviously blamed my own code, but then
> > I found that in the IPC call implementation found here...
> > 
> > pk/l4re-core/l4sys/include/cxx/ipc_iface
> > 
> > ...if I explicitly cleared the first message register before this
> > statement...> 
> >   int send_bytes =
> >   
> >     Args::template write_op<Do_in_data>(mrs->mr, 0, Mr_bytes,
> >     
> >                                         Opt::Opcode, a...);
> > 
> > ...then the opcode was produced as expected again.
> 
> Which does not fully make sense to me because the message registers seem
> to be written from 0 on. Anyway, do you have an example maybe?

I only found this to happen when a program of mine had fetched many pages from 
a dataspace via the RM. It would be useful to understand the conditions under 
which this occurs, and I obviously suspect that I must be doing something 
wrong, but I can't see how my dataspace implementation would corrupt the 
opcode sent by the RM in its own IPC messages. But I do think it is odd that 
somehow, the rather opaque code above doesn't manage to fully initialise the 
message registers.

Anyway, I will aim to continue my investigations and hopefully make some kind 
of progress.

Thanks for following up once again!

Paul