Re: L4Re: Allocating pinned memory and obtaining its physical address

Hi Paul,

On 5/23/23 00:51, Paul Boddie wrote:
> On Sunday, 23 April 2023 10:09:52 CEST Jakub Jermář wrote:
>>
>> The idea is that you have a dataspace and a DMA space. In order to
>> obtain a DMA address that you can then use to program your I/O device,
>> you need to map a portion of the dataspace into the DMA space. After use
>> you unmap it from the DMA space.
>>
>> See for example the implementation of the Inout buffer in our nvme-driver:
>>
>> https://github.com/kernkonzept/nvme-driver/blob/master/server/src/
> inout_buffer.h#L46
> 
> I also looked at the main.cc file to see how I might obtain a DMA space and
> assign it to a DMA domain. Obtaining a DMA space appears to involve invoking a
> factory create operation using the L4RE_PROTO_DMA_SPACE protocol.
> 
> Here, I found a difference between the L4Re C and C++ libraries, with the
> latter referencing a "user_factory" but defining it as the memory allocator,
> so I just used the mem_alloc member of the environment structure in C for the
> factory.

In C++ mem_alloc and user_factory are nearly the same. Mem_alloc extends 
the factory protocol with allocation for dataspaces. At the moment, this 
is effectively a convenience wrapper, as both are using the same 
capability index and thus talk to the same server.
The reason for the user_factory name is to be able to write less 
confusing code, e.g. where one would use mem_alloc to create some 
non-memory objects.


> 
> With a DMA space, it should supposedly be possible to specify ~0U (or -1, I
> guess) as the domain identifier when assigning the space to a domain, but this
> does not work and yields the following warning:
> 
> IO      | vbus vbus does not support a global DMA domain
> 
> This along with an error indicating "No such object found".
> 
> The main.cc file in the nvme-driver performs a search for a DMA domain
> resource, but replicating this myself produces a few issues. First of all, a
> search from the root device doesn't manage to find a resource offering a DMA
> domain if I traverse devices and their resources.
> 
> However, I see that the hw_root_bus.cc file in pkg/io/io/server/src adds root
> resources for a DMA domain, amongst other things, and if I also investigate
> the resources of the root device itself, then I do find such resources.
> Strangely, these resources appear to have an identifier of zero after they
> have been added to the bus, even though they get set up with expected
> identifiers such as "DMAD".
> 
> Meanwhile, in the main.cc file in pkg/io/io/server/src, a DMA domain resource
> is explicitly created when no iommu capability is provided to the server. This
> actually appears in the dump of the devices and resources at the system bus
> level, as produced by Io, but not within the vbus.
> 
> So, I think I need some suggestions as to how I can actually obtain a domain
> identifier and maybe how a domain should be made available in the .io file.
> Sorry if I failed to understand what to do, but none of this is particularly
> obvious to me.

The path you are on sounds correct. A reason for a missing DMA domain 
resource on the vbus is (usually) the lack of a DMA capable device on 
said vbus.

https://github.com/kernkonzept/manifest/wiki/NVMeWithLinux#vbus-configuration-file

shows a vbus configuration file that places all PCI/storage devices on 
the vbus. I presume your MIPS board lacks PCI, so you have to write the 
entries for System_bus and the Vbus yourself.

The helpful sections in another tutorial are:
https://github.com/kernkonzept/manifest/wiki/HwPassThrough#ios-config-file
https://github.com/kernkonzept/manifest/wiki/HwPassThrough#vbus-configuration-file

I found your mercurial instance for the l4re topic but didn't find any 
io & vbus configuration. Can you point me to your config files?
And/Or send along the Io output with maximum verbosity (-vvvvvvv)?


Cheers,
Philipp

_______________________________________________
l4-hackers mailing list
[email protected]
https://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers