Re: [PATCH v8 00/10] nvme-multipath: introduce latency I/O policy

On 15/08/2026 20:34, Nilay Shroff wrote:
> Hi,
>
> This series introduces a new latency I/O policy for NVMe native
> multipath. Existing policies such as numa, round-robin, and queue-depth
> are static and do not adapt to real-time transport performance. The numa
> selects the path closest to the NUMA node of the current CPU, optimizing
> memory and path locality, but ignores actual path performance. The
> round-robin distributes I/O evenly across all paths, providing fairness
> but not performance awareness. The queue-depth reacts to instantaneous
> queue occupancy, avoiding heavily loaded paths, but does not account for
> actual latency, throughput, or link speed.
>
> The new latency policy addresses these gaps selecting paths dynamically
> based on measured I/O latency for both PCIe and fabrics. Latency is
> derived by passively sampling I/O completions. Each path is assigned a
> weight proportional to its latency score, and I/Os are then forwarded
> accordingly. As condition changes (e.g. latency spikes, bandwidth
> differences), path weights are updated, automatically steering traffic
> toward better-performing paths.
>
> Early results show reduced tail latency under mixed workloads and
> improved throughput by exploiting higher-speed links more effectively.
> For example, with NVMf/TCP using two paths (one throttled with ~30 ms
> delay), fio results with random read/write/rw workloads (direct I/O)
> showed:

TBH, I do not know if this measurement represent any real-life
scenario. I do think that occasional packet drops are a real-life 
scenario, and
it would be a worthy use-case to optimize for. Can you perhaps measure
how the path selectors compare in this case?

>
>          numa         round-robin   queue-depth  adaptive
>          -----------  -----------   -----------  ---------
> READ:   50.0 MiB/s   105 MiB/s     230 MiB/s    350 MiB/s
> WRITE:  65.9 MiB/s   125 MiB/s     385 MiB/s    446 MiB/s
> RW:     R:30.6 MiB/s R:56.5 MiB/s  R:122 MiB/s  R:175 MiB/s
>          W:30.7 MiB/s W:56.5 MiB/s  W:122 MiB/s  W:175 MiB/s

And I'm assuming there are zero downsides for the normal
case?

>
> This pathcset includes totla 8 patches:
> [PATCH 1/10] block: expose blk_stat_{enable,disable}_accounting()
>    - Make blk_stat APIs available to block drivers.
>    - Needed for per-path latency measurement.
>
> [PATCH 2/10] block: record I/O request start time for passthru request
>    - Record I/O start time for I/O passthru requests.
>    - This is prep patch which allows measuring I/O completion latency
>      for passthru requests.
>
> [PATCH 3/10] block: support nesting for blk-mq flag QUEUE_FLAG_SAME_FORCE
>    - Support nesting for QUEUE_FLAG_SAME_FORCE as multiple users
>      could toggle QUEUE_FLAG_SAME_FORCE.
>
> [PATCH 4/10] nvme-multipath: pass I/O type to nvme_find_path()
>    - This is the prep patch which updates nvme_find_path() signature
>      
> [PATCH 5/10] nvme-multipath: add latency I/O policy
>    - Implement path scoring based on latency (EWMA).
>    - Distribute I/O proportionally to per-path weights.
>
> [PATCH 6/10] nvme: add generic debugfs support
>    - Introduce generic debugfs support for NVMe module
>
> [PATCH 7/10] nvme-multipath: add debugfs attribute latency_ewma_shift
>    - Adds a debugfs attribute to control ewma shift
>
> [PATCH 8/10] nvme-multipath: add debugfs attribute latency_batch_timeout
>    - Adds a debugfs attribute to control latency batch window interval
>
> [PATCH 9/10] nvme-multipath: add debugfs attribute latency_stat
>    - Add “latency_stat” under per-path and head debugfs directories to
>      expose latency policy state and statistics.
>
> [PATCH 10/10] nvme-multipath: add documentation for latency I/O policy
>    - Includes documentation for latency I/O multipath policy.
>
> LSFMM discussion:
> =================
> During lsfmm 2026, it was decided to rename this I/O policy from
> "adaptive" to "latency". This series reflects that rename.
>
> The discussion at lsfmm also focused extensively on the latency
> measurement model, including whether latency should be tracked
> per-CPU or per-NUMA, and whether separate I/O-size buckets should
> be maintained for different request sizes.
>
> After detailed discussion and evaluation of throughput results, the
> consensus was to initially measure I/O completion latency on a
> per-CPU basis. The available performance data showed that the
> per-CPU implementation already provides sufficient averaging across
> CPUs while keeping the design relatively simple.
>
> The use of additional I/O-size buckets did not demonstrate meaningful
> throughput improvement in the general case and would introduce extra
> complexity into the fast path and accounting logic. As a result, the
> consensus was to avoid I/O-size bucketing for now and keep the policy
> focused on per-CPU latency measurement.
>
> If future real-world workloads demonstrate a clear benefit from
> I/O-size-aware latency accounting, the policy can be extended later
> to support it.
>
> As ususal, feedback and suggestions are most welcome!

Nilay, do we have evidence that round-robin/queue-depth are better
for any workload? As a user, I would be very confused with the amount
of path selectors I have available and which should I choose.