[SPDK] Re: DIF/DIX acceleration in SPDK

Hi Ben,

Do you plan to replace DPDK calls  in "reduce" block by  "accel" framework?

Having plug-in system for HW offloads is a great idea.

Best regards
Sasha

On 11-Jun-20 11:00 PM, Walker, Benjamin wrote:
>
>> -----Original Message-----
>> From: Luse, Paul E <paul.e.luse(a)intel.com>
>> Sent: Thursday, June 11, 2020 11:22 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: [SPDK] Re: DIF/DIX acceleration in SPDK
>>
>> That’s a great question. I can only comment on what’s been made public and
>> there’s no mention of those things in the announcement here
>> https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator
>>
>> As I’m sure you’re aware, we already support compression/crypto via Intel QAT
>> https://www.intel.com/content/www/us/en/architecture-and-
>> technology/intel-quick-assist-technology-overview.html
> I think there are some interesting open questions in this area on how we model the software framework, even if DSA never supports crypto/compression. Right now you've coded up the 'accel' framework library and it can handle a bunch of these types of storage offloads by redirecting to either DSA, CBDMA, or ISA-L (CPU) as needed. You've even made it a plug-in system, so other vendors can add their offload hardware into it. That's a great starting point.
>
> Simultaneously, parts of SPDK leverage compression and crypto offload by directly calling into the DPDK framework. That code has a similar concept - plugin drivers for various pieces of hardware that can do crypto or compression type things, with ISA-l as the fallback.
>
> What isn't clear to me, as of right now, is if we should continue to model these two things as separate components, or if we should try to unite them into a single accel framework. Under the hood, we're going to delegate crypto and compression offloads to DPDK because that's where the drivers live, and the more storage-specific offloads that the 'accel' framework currently does will stay in SPDK. But we could build a more unified abstraction layer for SPDK users on top at least just for convenience. I don't know how much demand there would be for people to plug their own hardware into a framework like that, or from people who want to consume an API like that.
>
>> Thx
>> Paul
>>
>> From: Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
>> Sent: Thursday, June 11, 2020 11:14 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: [SPDK] Re: DIF/DIX acceleration in SPDK
>>
>> Hi Paul,
>>
>> does the offload engine support (or plan to support in the future) more complex
>> compute-intensive storage tasks such as compression/decompression, crypto
>> (encryption/strong hashing) etc.?
>> Thanks,
>> Andrey
>>
>> On Thu, Jun 11, 2020, 18:46 Luse, Paul E
>> <paul.e.luse(a)intel.com<mailto:paul.e.luse(a)intel.com>> wrote:
>> [cid:image001.gif(a)01D63FC9.365AE030]
>> Hi Everyone,
>>
>> This is primarily for Shuhei but please feel free, anyone, to respond ☺
>>
>> Adding support for Intel’s next generation offload engine is going well (Note,
>> the feature is not available in HW yet, I’m using a simulator to do dev/test).
>> Currently support exists, or is about to land on master, for:
>>
>> Copy, fill, dual-cast, CRC32C, compare and the ability to submit batches of
>> commands.
>>
>> Currently these are only being used by a new tool in /examples/accel/perf but
>> once they all land and I’ve added some more tests, we’ll start using them in
>> SPDK modules – the most notable uses will be for CRC32C 9iscsi) and DIF/DIX
>> throughout the stack.  There will be other uses (compare, fill, copy, etc) as well
>> but those are the big ones.
>>
>> I’ve just now started looking at DIF/DIX and have determined that using these
>> within SPDK won’t be quite as straightforward as some of the others. I’ll explain
>> what I’m thinking after briefly summering the DSA DIF/DIX functions (more detail
>> is available in the public spec at
>> https://software.intel.com/content/www/us/en/develop/download/intel-data-
>> streaming-accelerator-preliminary-architecture-specification.html)
>>
>> Note: there is no SGL support in any of these, all are single src and/or dst:
>>
>>
>>    *   DIF Check: The DIF Check operation computes the Data Integrity Field (DIF)
>> on the source data and compares the computed DIF to the DIF contained in the
>> source data.
>>    *   DIF Insert: The DIF Insert operation copies memory from the Source Address
>> to the Destination Address, while computing the Data Integrity Field (DIF) on the
>> source data and inserting the DIF into the output data.
>>    *   DIF Strip: The DIF Strip operation copies memory from the Source Address to
>> the Destination Address, removing the Data Integrity Field (DIF). It optionally
>> computes the DIF on the source data and compares the computed DIF to the DIF
>> contained in the source data.
>>    *   DIF Update: The DIF Update operation copies memory from the Source
>> Address to the Destination Address. It optionally computes the Data Integrity
>> Field (DIF) on the source data and compares the computed DIF to the DIF
>> contained in the data. It simultaneously computes the DIF on the source data
>> using Destination DIF fields in the descriptor and inserts the computed DIF into
>> the output data.
>>
>> Upon initial review of the relatively complex implementation of DIF?DIX we have
>> in SPDK I have the following observations that I’m hoping to get some feedback
>> on:
>>
>>
>>    *   It looks like we require SGL in most if not all cases. I can go through them
>> one by one but wanted to get an initial feel mainly from Shuhei on how lack of
>> SGL support impacts our ability to use DIF?DIX offload w/DSA before I start
>> adding support ☺
>>    *   With the exception of DIF Check, all of the DSA functions include a copy (I
>> can only assume they figured a use case where they are moving data from a
>> host buffer into a different memory subsystem in prep for DMA’ing to disk).  It
>> looks like most if not all of our calculations are done on fixed buffers. I see a
>> few copy functions in diff.c but I don’t see them used anywhere
>>
>> I’m almost thinking the DSA functions are too “simple” for our current
>> implementation but wonder if there’s some refactoring we can do to make use
>> of them. I don’t know if the DSA CRC32C engine calculates the same exact CRC
>> as the DIF/DIX functions but if so (I can verify) at a minum maybe use just
>> accelerate the CRCs called from funcs within diff.c
>>
>> Thoughts? We can chat in a community meeting soon too but email might be
>> easier to get us all on the amge page first.
>>
>> Thanks!!
>> Paul
>>
>> _______________________________________________
>> SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
>> To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-
>> leave(a)lists.01.org>
>> _______________________________________________
>> SPDK mailing list -- spdk(a)lists.01.org
>> To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org