[SPDK] Re: DIF/DIX acceleration in SPDK

松本周平 <shuheimatsumoto at gmai l.com>
Newsgroups dev.linux.lists.spdk
Message-ID <CAFEtCnB5JPfAzk9_k6k=nMLc-ynOqUGndAQL-va_aMHqv9Y--Q@mail.gmail.com> 
Hi Paul,

Current NVMe-oF target and iSCSI target have used special SGL when reading
from network or writing to network.
When reading from network, leave a metadata space per block by special SGL
and then compute and fill DIF to metadata spaces.
When writing to network, compute and check DIF to metadata spaces, and then
ignore a DIF space per block by special SGL.

Special SGL means that each SGL entry maps a single block and there is a
metadata space between two SGL entries.

The major reason why we used special SGL was to avoid data copy between two
buffers.

So, DIF check and DIF update will be helpful first.

The limitation that SGL is not supported will be acceptable first.
iSCSI target uses only a single contiguous buffer.
NVMe-oF target also can use only a single contiguous buffer by adjusting
maximum IO size.


Another thing I want to share is that SPDK sock layer improved performance
greatly recently, and this may conflict with
the current DIF implementation especially for the performance.

So in future, using DIF insert and DIF strip may be better for NVMe-TCP
target and iSCSI target.
When NVMe-TCP target or iSCSI target use DIF insert and DIF strip, they
prepare two buffers for each read or write, and copy with DIF insertion or
strip.

So all DIF check, update, insert , and strip will be usable anyway.

One difficulty to emulate SGL by DSA is boundary.
To compute DIF for a single block which spans multiple SGL entries, it is
necessary to compute CRC for the first partial block and then use the CRC
as the seed value for the second partial block, and so on.

I don't think my feedback is enough, and so I'm fine for your any further
question or feedback.

Thanks,
Shuhei

On Fri, Jun 12, 2020 at 12:46 AM Luse, Paul E <paul.e.luse(a)intel.com> wrote:

> Hi Everyone,
>
>
>
> This is primarily for Shuhei but please feel free, anyone, to respond J
>
>
>
> Adding support for Intel’s next generation offload engine is going well
> (Note, the feature is not available in HW yet, I’m using a simulator to do
> dev/test). Currently support exists, or is about to land on master, for:
>
>
>
> Copy, fill, dual-cast, CRC32C, compare and the ability to submit batches
> of commands.
>
>
>
> Currently these are only being used by a new tool in /examples/accel/perf
> but once they all land and I’ve added some more tests, we’ll start using
> them in SPDK modules – the most notable uses will be for CRC32C 9iscsi) and
> DIF/DIX throughout the stack.  There will be other uses (compare, fill,
> copy, etc) as well but those are the big ones.
>
>
>
> I’ve just now started looking at DIF/DIX and have determined that using
> these within SPDK won’t be quite as straightforward as some of the others.
> I’ll explain what I’m thinking after briefly summering the DSA DIF/DIX
> functions (more detail is available in the public spec at
> https://software.intel.com/content/www/us/en/develop/download/intel-data-streaming-accelerator-preliminary-architecture-specification.html
> )
>
>
>
> Note: there is no SGL support in any of these, all are single src and/or
> dst:
>
>
>
>    - DIF Check: The DIF Check operation computes the Data Integrity Field
>    (DIF) on the source data and compares the computed DIF to the DIF contained
>    in the source data.
>    - DIF Insert: The DIF Insert operation copies memory from the Source
>    Address to the Destination Address, while computing the Data Integrity
>    Field (DIF) on the source data and inserting the DIF into the output data.
>    - DIF Strip: The DIF Strip operation copies memory from the Source
>    Address to the Destination Address, removing the Data Integrity Field
>    (DIF). It optionally computes the DIF on the source data and compares the
>    computed DIF to the DIF contained in the source data.
>    - DIF Update: The DIF Update operation copies memory from the Source
>    Address to the Destination Address. It optionally computes the Data
>    Integrity Field (DIF) on the source data and compares the computed DIF to
>    the DIF contained in the data. It simultaneously computes the DIF on the
>    source data using Destination DIF fields in the descriptor and inserts the
>    computed DIF into the output data.
>
>
>
> Upon initial review of the relatively complex implementation of DIF?DIX we
> have in SPDK I have the following observations that I’m hoping to get some
> feedback on:
>
>
>
>    - It looks like we require SGL in most if not all cases. I can go
>    through them one by one but wanted to get an initial feel mainly from
>    Shuhei on how lack of SGL support impacts our ability to use DIF?DIX
>    offload w/DSA before I start adding support J
>    - With the exception of DIF Check, all of the DSA functions include a
>    copy (I can only assume they figured a use case where they are moving data
>    from a host buffer into a different memory subsystem in prep for DMA’ing to
>    disk).  It looks like most if not all of our calculations are done on fixed
>    buffers. I see a few copy functions in diff.c but I don’t see them used
>    anywhere
>
>
>
> I’m almost thinking the DSA functions are too “simple” for our current
> implementation but wonder if there’s some refactoring we can do to make use
> of them. I don’t know if the DSA CRC32C engine calculates the same exact
> CRC as the DIF/DIX functions but if so (I can verify) at a minum maybe use
> just accelerate the CRCs called from funcs within diff.c
>
>
>
> Thoughts? We can chat in a community meeting soon too but email might be
> easier to get us all on the amge page first.
>
>
>
> Thanks!!
>
> Paul
>
>
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>
attachment.htm (text/html, 8.3 KB) 
<div dir="ltr"><div dir="ltr">Hi Paul,<div><br></div><div>Current NVMe-oF target and iSCSI target have used special SGL when reading from network or writing to network.</div><div>When reading from network, leave a metadata space per block by special SGL and then compute and fill DIF to metadata spaces.</div><div>When writing to network, compute and check DIF to metadata spaces, and then ignore a DIF space per block by special SGL.</div><div><br></div><div>Special SGL means that each SGL entry maps a single block and there is a metadata space between two SGL entries.</div><div><br></div><div>The major reason why we used special SGL was to avoid data copy between two buffers.</div><div><br></div><div>So, DIF check and DIF update will be helpful first.</div><div><br></div><div>The limitation that SGL is not supported will be acceptable first.</div><div>iSCSI target uses only a single contiguous buffer.</div><div>NVMe-oF target also can use only a single contiguous buffer by adjusting maximum IO size.</div><div><br></div><div><br></div><div>Another thing I want to share is that SPDK sock layer improved performance greatly recently, and this may conflict with</div><div>the current DIF implementation especially for the performance.</div><div><br></div><div>So in future, using DIF insert and DIF strip may be better for NVMe-TCP target and iSCSI target.</div><div>When NVMe-TCP target or iSCSI target use DIF insert and DIF strip, they prepare two buffers for each read or write, and copy with DIF insertion or strip.</div><div><br></div><div>So all DIF check, update, insert , and strip will be usable anyway.</div><div><br></div><div>One difficulty to emulate SGL by DSA is boundary. </div><div>To compute DIF for a single block which spans multiple SGL entries, it is necessary to compute CRC for the first partial block and then use the CRC as the seed value for the second partial block, and so on.</div></div><div><br></div><div>I don&#39;t think my feedback is enough, and so I&#39;m fine for your any further question or feedback.</div><div><br></div><div>Thanks,</div><div>Shuhei</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 12, 2020 at 12:46 AM Luse, Paul E &lt;<a href="mailto:[email protected]">[email protected]</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">






<div bgcolor="white" lang="EN-US">
<img src="cid:172a8ac43c84cdccc1" width="0" height="0" style="display: none; width: 0px; height: 0px;">
<div class="gmail-m_610875093099702040WordSection1">
<p class="MsoNormal">Hi Everyone,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">This is primarily for Shuhei but please feel free, anyone, to respond
<span style="font-family:Wingdings">J</span><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Adding support for Intel’s next generation offload engine is going well (Note, the feature is not available in HW yet, I’m using a simulator to do dev/test). Currently support exists, or is about to land on master, for:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Copy, fill, dual-cast, CRC32C, compare and the ability to submit batches of commands.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Currently these are only being used by a new tool in /examples/accel/perf but once they all land and I’ve added some more tests, we’ll start using them in SPDK modules – the most notable uses will be for CRC32C 9iscsi) and DIF/DIX throughout
 the stack.  There will be other uses (compare, fill, copy, etc) as well but those are the big ones.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’ve just now started looking at DIF/DIX and have determined that using these within SPDK won’t be quite as straightforward as some of the others. I’ll explain what I’m thinking after briefly summering the DSA DIF/DIX functions (more detail
 is available in the public spec at <a href="https://software.intel.com/content/www/us/en/develop/download/intel-data-streaming-accelerator-preliminary-architecture-specification.html" target="_blank">
https://software.intel.com/content/www/us/en/develop/download/intel-data-streaming-accelerator-preliminary-architecture-specification.html</a>)<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Note: there is no SGL support in any of these, all are single src and/or dst:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<ul style="margin-top:0in" type="disc">
<li class="gmail-m_610875093099702040MsoListParagraph" style="margin-left:0in">DIF Check: The DIF Check operation computes the Data Integrity Field (DIF) on the source data and compares the computed DIF to the DIF contained in the source data.<u></u><u></u></li><li class="gmail-m_610875093099702040MsoListParagraph" style="margin-left:0in">DIF Insert: The DIF Insert operation copies memory from the Source Address to the Destination Address, while computing the Data Integrity Field (DIF) on the source data and inserting
 the DIF into the output data.<u></u><u></u></li><li class="gmail-m_610875093099702040MsoListParagraph" style="margin-left:0in">DIF Strip: The DIF Strip operation copies memory from the Source Address to the Destination Address, removing the Data Integrity Field (DIF). It optionally computes the DIF on the
 source data and compares the computed DIF to the DIF contained in the source data.<u></u><u></u></li><li class="gmail-m_610875093099702040MsoListParagraph" style="margin-left:0in">DIF Update: The DIF Update operation copies memory from the Source Address to the Destination Address. It optionally computes the Data Integrity Field (DIF) on the source data and
 compares the computed DIF to the DIF contained in the data. It simultaneously computes the DIF on the source data using Destination DIF fields in the descriptor and inserts the computed DIF into the output data.<u></u><u></u></li></ul>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Upon initial review of the relatively complex implementation of DIF?DIX we have in SPDK I have the following observations that I’m hoping to get some feedback on:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<ul style="margin-top:0in" type="disc">
<li class="gmail-m_610875093099702040MsoListParagraph" style="margin-left:0in">It looks like we require SGL in most if not all cases. I can go through them one by one but wanted to get an initial feel mainly from Shuhei on how lack of SGL support impacts our
 ability to use DIF?DIX offload w/DSA before I start adding support <span style="font-family:Wingdings">
J</span><u></u><u></u></li><li class="gmail-m_610875093099702040MsoListParagraph" style="margin-left:0in">With the exception of DIF Check, all of the DSA functions include a copy (I can only assume they figured a use case where they are moving data from a host buffer into a different memory
 subsystem in prep for DMA’ing to disk).  It looks like most if not all of our calculations are done on fixed buffers. I see a few copy functions in diff.c but I don’t see them used anywhere<u></u><u></u></li></ul>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’m almost thinking the DSA functions are too “simple” for our current implementation but wonder if there’s some refactoring we can do to make use of them. I don’t know if the DSA CRC32C engine calculates the same exact CRC as the DIF/DIX
 functions but if so (I can verify) at a minum maybe use just accelerate the CRCs called from funcs within diff.c<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thoughts? We can chat in a community meeting soon too but email might be easier to get us all on the amge page first.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks!!<u></u><u></u></p>
<p class="MsoNormal">Paul<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>

_______________________________________________<br>
SPDK mailing list -- <a href="mailto:[email protected]" target="_blank">[email protected]</a><br>
To unsubscribe send an email to <a href="mailto:[email protected]" target="_blank">[email protected]</a><br>
</blockquote></div></div>
image001.gif (image/gif, 92 B) - not displayed
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.