Re: [OPAE] IOMMU in Xeon+FPGA and the perception of memory by OPAE apps

--===============6772999615544407911==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

Hi Furkan,

thank you for your question! Let me try to clarify a couple of points before
addressing your question directly.

OPAE is targeted at supporting a wide range of programmable accelerator
platforms, and its fundamental concepts are kept intentionally broad to acc=
ount
for differences in platform implementation. As such, it may include features
that are not available on all supported platforms or may differ in realizat=
ion.

I believe you're referring to the FPGA platform that's part of the current =
Intel
Hardware Accelerator Research Program (that was the target platform of the
presentation at FPL2017). That platform does not include an IOMMU, so any
addresses returned by fpgaGetIOAddress() will indeed be physical addresses.
(Trivia: fpgaGetIOAddress() was originally called fpgaGetIOVA(), but we cha=
nged
it precisely because of the reasons you're outlining.) It appears that not =
all
variable names carrying this IO address have been renamed to account for th=
is,
though; I'm sorry for any confusion this may have caused.

If you look at other potential platforms that OPAE could support (or even at
actual platforms OPAE supports right now, such as http://altera.com/pac), y=
ou
will find examples of configurations that do have an IOMMU (the PAC, for
example, is a PCIe card, so it lives behind any IOMMU of the platform it's
plugged into).

An IOMMU does not necessarily provide the same virtual-to-phyiscal translat=
ions
that are created for software applications by the MMU -- IO devices in
virtualized systems may have their own IO virtual address space (hence the
name "IO Virtual Address"). In many virtualized systems, the IOMMU will only
provide a single mapping for each virtual machine. To allow AFUs to use true
"process virtual addresses" instead of "IO addresses" (which may be virtual=
 or
physical depending on IOMMU availability) you'll need additional logic in t=
he
FPGA to do the translation in sync with the OS page tables.

As it happens, the "intel-fpga-bbb" repository

	https://github.com/OPAE/intel-fpga-bbb

includes an IP block (we call then "basic building blocks" or BBBs) for this
named MPF -- you may recall a section of the FPL 2017 workshop that explain=
ed
it. MPF includes a feature for virtual-to-physical address translations (VT=
P)
for AFUs. If you instantiate this block in your AFU design, your AFU can us=
e the
virtual addresses of its owning process (you can share pointers) -- limited=
 to
explicitly shared buffers, of course --, and there is no need to use
fpgaGetIOAddress() at all.

On Fri, Feb 16, 2018 at 04:12:22PM +0100, Furkan Turan wrote:
> Hello Dears,
> =

> I would like to consult on you about the memory view of AFUs.
> =

> In OPAE, fpgaGetIOAddress is declared to get IO address, which is returned
> as updating the ioaddr parameter passed to it. A variable for this parame=
ter
> in one example code is name as buf_pa, as it will return the IO address f=
or
> the buffer and that will be the physical address "pa". However, in some
> other examples, e.g. hello_fpga.c, the same variable is names as iova whi=
ch
> basically stands for IO virtual address.
> =

> As far as I remember the presentation in FPL2017, it was mentioned that
> there is no IOMMU right now; therefore, iova and physical address should =
be
> same things. However, the opae.github.io explains the functions referring=
 to
> IOMMU, so maybe Intel has already introduced it. In that case, AFUs should
> not work with physical addresses, but indeed have virtual addresses which
> will be translated to physical by the IOMMU.
> =

> I would like to consult on you about this:
> * Is IOMMU available or not?

It depends on the target platform. The integrated Xeon+FPGA system deployed=
 as
part of HARP does not; a deployment of the PAC card may very well do (as may
other platforms).

> * If there is IOMMU, then how the page tables are maintained: by OS, or by
> OPAE drivers?

The IOMMU translations are managed by the operating system. The OPAE driver=
 uses
the Linux kernel DMA subsystem, which is aware of (and manages) these mappi=
ngs.

> * Can this provide the same virtual addresses on both SW and HW side in t=
he
> future, making the fpgaGetIOAddress function obsolete?

Yes, using MPF (see above). Other (future?) platforms may also provide other
solutions to this -- it has been done before for other accelerator technolo=
gies.

> * May the IOMMU allow getting rid of the contagious memory space allocati=
on
> requirement for the buffers in the future?

That's an interesting question. I believe you could program an IOMMU to mir=
ror
the page mappings of an MMU and thus allow page-level translations. I don't=
 know
how that would be implemented using the Linux kernel, though (I believe it
currently only creates mappings for explicitly requested DMA buffers, which=
 have
to be contiguous (duh)). You may also run into issues where the IOMMU is al=
ready
used for translating between guest physical and host physical addresses for
virtual machines -- you'll need something like a second-level address
translation mechanism for that (similar to what MMUs do with EPT/RVI).

I hope this answers your questions.

Thank you
- Enno


--===============6772999615544407911==--