Re: Evenly distribute jobs and iodepth over a 1 TiB device so that every byte is written to in parallel

Hello Thomas,

On Tue, 15 Jul 2025 at 06:18, Thomas Glanzmann <[email protected]> wrote:
>
> I have a 1 TiB NVMe namespace from a NetApp connected via two distinct
> direct links to a Linux system over NVMe/TCP. I would like to generate
> read and write I/O using multiple jobs/iodepth so that every byte of the
> device is being written to in parallel with the maximum number of
> available parallel inflight I/Os. The NetApp does deduplication
> and compression by default so I want to generate random data. Because I
> think if I don't do refill_buffers, the NetApp gets that the same data
> is used over and over again and dedups it. I tried:

It depends on just how clever it is. By default fio uses
scramble_buffers
(https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-scramble_buffers
) which may be enough but you would have to check. The trick would be
to see what happens with a single stream as that's easier to reason
about.

> fio --ioengine=libaio --refill_buffers --filesize=25G --ramp_time=2s \
> --runtime=1m --numjobs=40 --direct=1 --verify=0 --randrepeat=0 \
> --group_reporting --filename=/dev/nvme0n1 --name=1mhqd --blocksize=1m \
> --iodepth=1638 --readwrite=write

If I'm reading this correctly you have 40 jobs all sequentially
writing the same 25G of the device at the same time. This is
problematic because if you send two or more write I/Os for the same
area to storage at the same time then something in your data path
could legitimately just throw all but one of them away and say "I'm
done" (because one I/O overwrites the others) - you're essentially
saying "I don't care about the data" and in certain setups such
behaviour is undefined. One change you could make is to have each of
the 40 jobs write to a different region to the others e.g. by using
offset_increment
https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-offset_increment
and size https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-size
.

<snip>

> So, I ran:
>
> (live) [~] pv < /dev/urandom > /dev/nvme0n1
> 1.00TiB 0:59:14 [ 294MiB/s] [======================>] 100%
>
> And afterwards more physical space was used:
>
> na2501::*> aggr show -fields physical-used
> aggregate      physical-used
> -------------- -------------
> dataFA_4_p0_i1 1.15TB

Do you get similar results in terms of space used with a single fio
stream? Start small and then work your way up!

> So, what is the best way to use fio to write random data to every byte of this
> 1 TiB device in parallel?
>
>         - Is there a command line parameter?
>         - Or should I create 40 25.6 GiB (1024/40) partitions and give them as
>           colon separated list to fio?

See above.

> I also would like to determine the number of queues and queue depth? Is there a
> command available. When I run:

fio does report statistics about what queue depth each job internally
reached but these may be different to what your device sees for a
variety of reasons (e.g. splitting a coalescing done by the block
layer). See IO depths/IO submit/IO complete over on
https://fio.readthedocs.io/en/latest/fio_doc.html#interpreting-the-output
. But perhaps you're thinking of device queue depths?

> fio --ioengine=libaio --refill_buffers --filesize=8G --ramp_time=2s \
> --runtime=1m --numjobs=40 --direct=1 --verify=0 --randrepeat=0 \
> --group_reporting --filename=/dev/nvme0n1 --name=4khqd --blocksize=4k \
> --iodepth=1638 --readwrite=randwrite

Given you're running 40 jobs I'd be surprised if you can hit a depth
of over 1000 per job (that would be over 65000 I/Os in total) without
some serious tuning. You may want to look at
/sys/block/[disk]/queue/nr_requests (see
https://www.kernel.org/doc/Documentation/block/queue-sysfs.rst ) and
/sys/block/[disk]/device/queue_depth but you may also find you run
into libaio limits...

> And also watch 'iostat -xm 2' I can see aqu-sz is 194.87 per path and
> 391.96 for the multipathed device nvme0n1. So I kind of know it but
> would like to have a command on Linux that shows me the available queues
> and queue depths.

I'm fairly sure iostat is the right way to go (unless you wanted to
write some BPF tracing).

-- 
Sitsofe