Re: dma_opt_mapping_size returns way too low sizes when using IOMMU

[adding Adrian who originall added the cap to NVMe to work around a
softlockup, see commit 3710e2b056cb ("nvme-pci: clamp max_hw_sectors
based on DMA optimized limitation")]

On Mon, Aug 17, 2026 at 06:04:24PM +0100, Robin Murphy wrote:
>> page order, i.e. the idea was that we should not be getting streaming IOs 
>> larger than that value. But in looking at lore, 8 was very originally 
>> proposed, but I can't see any discussion on why that changed or any 
>> relation to page max order.
>
> From a quick skim I can't spot any specific mention in the original threads 
> or accompanying paper, but I have a feeling it was as much about just 
> balancing memory consumption with observable performance improvement, and 
> larger sizes weren't significantly impactful to the original testing, which 
> did seem to be focused on networking rather than storage.

Yeah, network transfer sizes tend to be much lower than those for
storage.

> Since these days we have the flexibly-sized depot, and now even 
> lazy-allocation of the CPU magazines, we might be in a better position to 
> consider scaling IOVA_RANGE_CACHE_MAX_SIZE up without too much inherent 
> overhead (I guess the base cost per level will just be 2*NR_CPUS + 1 
> pointers now).

That would be useful.  Is just incrementing it enough or do we need more
fine tuning?

> The larger sizes we cache, though, the more it's also going 
> to blow up IOVA consumption if freeing/reallocation isn't well balanced 
> across CPUs, so for devices/systems with limited IOVA space, at some point 
> the flush_rcache case is liable to start adding more overhead than the 
> caching itself saves...

Storage drivers tend to steer I/O completions to the submitting CPU by
default.  There are some exceptions, though.  For example polling with
a dedicated polling thread will usually not do this.  Then again the
overlap of polled I/O and IOMMU usage should be about 0 right now.
This could change with the dmabuf I/O path, but that would lock down
the IOVA for the lifetime of the dmabuf and sidestep the entire
issue here.

> FWIW, my understanding of the idea we settled on for opt_mapping_size was 
> that it was never meant to be an absolute target, just a boundary between 
> where callers can expect "ideal" mapping performance, and where it may 
> start scaling with size/contention/etc., and thus it's for the callers to 
> judge whether larger sizes are still worth it for their particular 
> use-case.

That would suggest the approach done in the SAS transport class makes
more sense and NVMe should just cap to the max mapping size.  Of course
even that might be the wrong answer.  For example for HDD we really do
want much larger I/O sizes even if it slows down the IOMMU, but right
now HDD on NVMe isn't really beyond the proof of concept stage, at
least discounting things like smart NICs as an in-between layer.