[SPDK] Re: --no-huge on IOMMU/VFIO platform

On 12/2/2021 7:12 PM, niketkandya(a)google.com wrote:
>  From my reading of the code and docs this is what I understand:
> 1. Huge pages are needed by SPDK/DPDK so that user pages are always present and pinned.
> 2. In systems with IOMMU and VFIO support, there should be no need for huge pages. spdk_mem_register on an externally allocated buffer should do the same pinning.
> 3. This is also what happens to DPDK's internal heaps in the memory callbacks (memory_iter_cb).
> 
> When I tried to use --no-huge with SPDK it failed with error in alignment/size restrictions for DPDK's internal heap. I was able to hack DPDK code to return 2MB aligned addresses for its internal heap and this worked.
> 
> So, to confirm my understanding here: for SPDK to work without hugepages, just a guarantee that (2MB)aligned addresses are returned by DPDK's internal heap should be enough?

I can't think of any other requirements. We store virtual to physical 
mappings in a radix tree with 2MB granularity, so as long as you can 
assign a physical/io address to every virtual address in 2MB units, I 
think it should be fine. I'd be curious to see how you changed DPDK 
because I'd love to be able to officially support SPDK with no huge pages.

I *think* the step of programming the IOMMU to associate the virtual 
address to the io address pins the pages. I'm about 99% sure that's the 
case. If for some reason it's not the case, you'd also need to mlock all 
the memory.

> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>