[SPDK] Re: Something cool coming for IOAT....

I guess I should comment on the latency cost of batching before someone calls me out on it ( Clearly there's a penalty and updates will be made to the tool to measure latency as well....

On 7/18/20, 4:42 PM, "Luse, Paul E" <paul.e.luse(a)intel.com> wrote:

    So coming this release is a bunch of support for DSA but that silicon feature won’t be ready for a while.  However, it did inspire a big revamp of what we used to call the “copy framework” now it’s known as the “accelerator framework”.  DSA has a pretty cool batching feature that’s been coded into the general accel_fw API and is pretty easy to use and yes it supports for the IOAT engine and, for compatibility reasons only, the SW engine. The low level IOAT library already had it’s own version of batching via the ability to create a bunch of copy/fill descriptors and selectively send them later.  Now one API can be used to batch all supported commands (copy, fill, compare, dual-cast, crc32c) for any of the engines.  Note that for IOAT you can batch any combination of these through the accel_fw API however the only ones that will get hw benefit from batching for IOAT are copy/fill as that’s all it supports (DSA can batch everything).

    Anyway, fun news for a Sat… here’s IOAT with 512B copies, 1024 outstanding IO and 4 cores using the new accel_perf app (many patches still under review).  Below that the same exact thing but with batching. There’s likely a wide variety of benefits, I’m guessing this isn’t either the best or worst case but I’ll save you the math – this is a 167% improvement, not too shabby ☺

    [peluse(a)localhost examples]$ sudo ./accel_perf -q 1024 --wait-for-rpc -w copy -m 0xf -o 512
    SPDK Configuration:
    Core mask:      0xf

    Accel Perf Configuration:
    Workload Type:  copy
    Transfer size:  512 bytes
    Queue depth:    1024
    Run time:       5 seconds
    Batching:       Disabled
    Verify:         No

    Starting SPDK v20.07-pre git sha1 527d01212 / DPDK 19.11.2 initialization...
    [ DPDK EAL parameters: (null) --no-shconf -c 0xf --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid27832 ]
    EAL: No available hugepages reported in hugepages-1048576kB
    EAL: VFIO support initialized
    [2020-07-18 19:33:12.857089] app.c: 652:spdk_app_start: *NOTICE*: Total cores available: 4
    [2020-07-18 19:33:12.996874] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 1
    [2020-07-18 19:33:12.997441] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 2
    [2020-07-18 19:33:12.997966] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 3
    [2020-07-18 19:33:12.998396] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 0
    [2020-07-18 19:33:16.094129] accel_engine.c: 510:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
    EAL:   using IOMMU type 1 (Type 1)
    [2020-07-18 19:33:16.809376] accel_engine_ioat.c: 701:accel_engine_ioat_init: *NOTICE*: Accel engine updated to use IOAT engine.
    Running for 5 seconds...

    Core           Transfers     Bandwidth     Failed     Miscompares
    -----------------------------------------------------------------
             3     1966784/s     960 MiB/s          0           0
             2     1966680/s     960 MiB/s          0           0
             1     1965504/s     959 MiB/s          0           0
             0     1963873/s     958 MiB/s          0           0
    ==================================================================
    Total:         7862841/s    3839 MiB/s          0           0

    [peluse(a)localhost examples]$ sudo ./accel_perf -q 1024 --wait-for-rpc -w copy -m 0xf -o 512 -b 256
    SPDK Configuration:
    Core mask:      0xf

    Accel Perf Configuration:
    Workload Type:  copy
    Transfer size:  512 bytes
    Queue depth:    1024
    Run time:       5 seconds
    Batching:       256 reqs
    Verify:         No

    Starting SPDK v20.07-pre git sha1 527d01212 / DPDK 19.11.2 initialization...
    [ DPDK EAL parameters: (null) --no-shconf -c 0xf --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid27818 ]
    EAL: No available hugepages reported in hugepages-1048576kB
    EAL: VFIO support initialized
    [2020-07-18 19:32:55.246468] app.c: 652:spdk_app_start: *NOTICE*: Total cores available: 4
    [2020-07-18 19:32:55.386718] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 1
    [2020-07-18 19:32:55.387289] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 2
    [2020-07-18 19:32:55.387802] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 3
    [2020-07-18 19:32:55.388245] reactor.c: 371:reactor_run: *NOTICE*: Reactor started on core 0
    [2020-07-18 19:32:58.219421] accel_engine.c: 510:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
    EAL:   using IOMMU type 1 (Type 1)
    [2020-07-18 19:32:58.937321] accel_engine_ioat.c: 701:accel_engine_ioat_init: *NOTICE*: Accel engine updated to use IOAT engine.
    Running for 5 seconds...

    Core           Transfers     Bandwidth     Failed     Miscompares
    -----------------------------------------------------------------
             3     3299942/s    1611 MiB/s          0           0
             2     3298748/s    1610 MiB/s          0           0
             1     3298916/s    1610 MiB/s          0           0
             0     3300608/s    1611 MiB/s          0           0
    ==================================================================
    Total:        13198216/s    6444 MiB/s          0           0
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org