Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling

On Thu, Aug 06, 2026 at 11:10:58AM +0800, Fengnan Chang wrote:
> Each Solidigm SB5PH27X038T device used for testing can deliver about 3.2M
> 4 KiB random-read IOPS.  Four of them should be good for about 12.8M IOPS,
> but interrupt-driven completion tops out at 5.59M, only about 44% of that.
> 
> Polling gets rid of that cost, but polling every queue all the time burns
> CPU and hurts the sparse or bursty queues that interrupts handle just fine.
> So instead of a global switch, let each queue make the call on its own,
> from how fast it has been completing lately, and re-check often enough that
> the decision tracks the workload rather than a fixed tunable.
> 
> Each queue runs a small loop with three stages.  First it samples its
> completion rate while still on interrupts.  Only if that rate is high
> enough to fill a small batch inside a bounded latency window does it mask
> its own IRQ and start draining the CQ from a high-resolution timer, with
> each wait sized to collect roughly one batch.  It keeps polling as long as
> it keeps up with that rate; the moment it stalls or slows down it turns the
> IRQ back on and backs off, waiting longer the further behind it fell.
> A queue that doesn't benefit drops back quickly and only gets retried once
> in a while, so polling stays on the queues that are actually
> interrupt-bound and everything else keeps running on the untouched IRQ
> path.
> 
> Measured with 4 KiB random reads on Solidigm SB5PH27X038T, adaptive on
> versus off:
> 
>                                    QD32      QD64      QD128
>   one device, one job             +18.44%   +24.35%   +26.38%
>   four devices, eight jobs        +83.76%   +99.02%   +96.26%
>

Did a quick test on single NVMe with QD64, one job, 4K random reads via
io_uring, I see ~23% improvement: 569K -> 701K IOPS.

Do you expect the fixed batch size, sample size, maximum delay, episode
length, and backoff multiplier to work across devices with different
latency profiles and across different workloads, or should these values
somehow adapt to observed queue behavior?

> +/*
> + * Stop polling and turn the queue's IRQ back on.  @elapsed is how long the
> + * episode ran after it started falling behind, or 0 if it ended cleanly.
> + * The bigger @elapsed is, the more completions we missed, and the longer we
> + * wait before sampling this queue again, so a queue that polling doesn't
> + * help is left alone most of the time.
> + */

@elapsed is the total polling-episode duration, not the time since the
queue fell behind. It is then used with the total completion count to
estimate cumulative deficit. Could the comment be reworded accordingly?

> +/*
> + * Called from the IRQ handler after a reap that found something.  If we're
> + * still in backoff, just count it down.  Otherwise time how long
> + * NVME_ADAPTIVE_SAMPLE_CQES completions take to get the average gap between
> + * them.  If that looks worth polling (see the filter below) mask the IRQ and
> + * switch to poll mode; if not, leave the queue on interrupts.
> + */
> +static void nvme_adaptive_sample(struct nvme_queue *nvmeq,
> +				 unsigned int completions)
> +{
> +	struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
> +	unsigned int sample;
> +	unsigned long flags;
> +	u64 delta, interval, now;
> +
> +	if (adaptive->retry_completions) {
> +		if (adaptive->retry_completions != U64_MAX)
> +			adaptive->retry_completions -= min_t(u64, completions,
> +							 adaptive->retry_completions);
> +		return;
> +	}
> +	if (!adaptive->start_ns) {
> +		adaptive->start_ns = ktime_get_ns();
> +		return;
> +	}
> +	adaptive->completions += completions;
> +	if (adaptive->completions < NVME_ADAPTIVE_SAMPLE_CQES)
> +		return;
> +
> +	now = ktime_get_ns();
> +	delta = now - adaptive->start_ns;
> +	sample = adaptive->completions;
> +	adaptive->start_ns = now;
> +	adaptive->completions = 0;
> +	if (!delta || delta > (u64)NVME_ADAPTIVE_SAMPLE_CQES *
> +			       NVME_ADAPTIVE_MAX_DELAY_NS)
> +		return;
> +	/*
> +	 * Is this rate worth polling?  The 100/99 factor trims 1% off the gap so
> +	 * a queue sitting right on the threshold isn't pulled in.  Skip it if
> +	 * it's too slow to fill a batch within MAX_DELAY, and also skip it if
> +	 * it's already fast enough to batch by itself.  The hardware's own
> +	 * coalescing already handles that case, so leave it on interrupts.
> +	 */
> +	interval = div64_u64(delta * 100, sample * 99);
> +	if (!interval || interval > NVME_ADAPTIVE_MAX_DELAY_NS ||

The comment says this rejects queues that cannot fill a target batch
within MAX_DELAY, but this condition only checks whether one completion
interval exceeds MAX_DELAY. Should this instead account for TARGET_BATCH
or this comment is describing a different policy?

> +	    (delta > NSEC_PER_MSEC &&
> +	     interval <= NVME_ADAPTIVE_MAX_DELAY_NS /
> +			 NVME_ADAPTIVE_TARGET_BATCH))
> +		return;
> +

For a normal 256-CQE sample with an interval of 2us or less, delta will
be about 512us or less, so the 1ms condition prevents this exclusion
from firing. Is this intended to detect only samples that substantially
overshoot 256 CQEs? If so, could the comment make that narrower intent
explicit?