[SPDK] Something cool coming for IOAT....

So coming this release is a bunch of support for DSA but that silicon feature won’t be ready for a while.  However, it did inspire a big revamp of what we used to call the “copy framework” now it’s known as the “accelerator framework”.  DSA has a pretty cool batching feature that’s been coded into the general accel_fw API and is pretty easy to use and yes it supports for the IOAT engine and, for compatibility reasons only, the SW engine. The low level IOAT library already had it’s own version of batching via the ability to create a bunch of copy/fill descriptors and selectively send them later.  Now one API can be used to batch all supported commands (copy, fill, compare, dual-cast, crc32c) for any of the engines.  Note that for IOAT you can batch any combination of these through the accel_fw API however the only ones that will get hw benefit from batching for IOAT are copy/fill as that’s all it supports (DSA can batch everything).

Anyway, fun news for a Sat… here’s IOAT with 512B copies, 1024 outstanding IO and 4 cores using the new accel_perf app (many patches still under review).  Below that the same exact thing but with batching. There’s likely a wide variety of benefits, I’m guessing this isn’t either the best or worst case but I’ll save you the math – this is a 167% improvement, not too shabby ☺

[peluse(a)localhost examples]$ sudo ./accel_perf -q 1024 --wait-for-rpc -w copy -m 0xf -o 512
SPDK Configuration:
Core mask:      0xf

Accel Perf Configuration:
Workload Type:  copy
Transfer size:  512 bytes
Queue depth:    1024
Run time:       5 seconds
Batching:       Disabled
Verify:         No

Starting SPDK v20.07-pre git sha1 527d01212 / DPDK 19.11.2 initialization...
[ DPDK EAL parameters: (null) --no-shconf -c 0xf --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid27832 ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized
[2020-07-18 19:33:12.857089] app.c: 652:spdk_app_start: *NOTICE*: Total cores available: 4
[2020-07-18 19:33:12.996874] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 1
[2020-07-18 19:33:12.997441] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 2
[2020-07-18 19:33:12.997966] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 3
[2020-07-18 19:33:12.998396] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 0
[2020-07-18 19:33:16.094129] accel_engine.c: 510:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
EAL:   using IOMMU type 1 (Type 1)
[2020-07-18 19:33:16.809376] accel_engine_ioat.c: 701:accel_engine_ioat_init: *NOTICE*: Accel engine updated to use IOAT engine.
Running for 5 seconds...

Core           Transfers     Bandwidth     Failed     Miscompares
-----------------------------------------------------------------
         3     1966784/s     960 MiB/s          0           0
         2     1966680/s     960 MiB/s          0           0
         1     1965504/s     959 MiB/s          0           0
         0     1963873/s     958 MiB/s          0           0
==================================================================
Total:         7862841/s    3839 MiB/s          0           0

[peluse(a)localhost examples]$ sudo ./accel_perf -q 1024 --wait-for-rpc -w copy -m 0xf -o 512 -b 256
SPDK Configuration:
Core mask:      0xf

Accel Perf Configuration:
Workload Type:  copy
Transfer size:  512 bytes
Queue depth:    1024
Run time:       5 seconds
Batching:       256 reqs
Verify:         No

Starting SPDK v20.07-pre git sha1 527d01212 / DPDK 19.11.2 initialization...
[ DPDK EAL parameters: (null) --no-shconf -c 0xf --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid27818 ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized
[2020-07-18 19:32:55.246468] app.c: 652:spdk_app_start: *NOTICE*: Total cores available: 4
[2020-07-18 19:32:55.386718] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 1
[2020-07-18 19:32:55.387289] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 2
[2020-07-18 19:32:55.387802] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 3
[2020-07-18 19:32:55.388245] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 0
[2020-07-18 19:32:58.219421] accel_engine.c: 510:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
EAL:   using IOMMU type 1 (Type 1)
[2020-07-18 19:32:58.937321] accel_engine_ioat.c: 701:accel_engine_ioat_init: *NOTICE*: Accel engine updated to use IOAT engine.
Running for 5 seconds...

Core           Transfers     Bandwidth     Failed     Miscompares
-----------------------------------------------------------------
         3     3299942/s    1611 MiB/s          0           0
         2     3298748/s    1610 MiB/s          0           0
         1     3298916/s    1610 MiB/s          0           0
         0     3300608/s    1611 MiB/s          0           0
==================================================================
Total:        13198216/s    6444 MiB/s          0           0