[SPDK] Re: SPDK socket abstraction layer

Hello everyone,

Summary:

VPP socket abstraction will be deprecated starting with SPDK 20.07. Later to be removed in SPDK 20.10.

Details:

Soon a full year of SPDK releases will pass since the previous message. During this time the VPP socket abstraction has not seen any community contributions.

In the meantime POSIX socket abstraction has seen major improvements. This includes addition of asynchronous writev, that batches writes to reduce syscalls. Added support for MSG_ZEROCOPY. Batching receives in NVMe-oF TCP.
The enhancements are best demonstrated by comparing SPDK 19.07 and 20.01 NVMe-oF TCP performance reports:
https://spdk.io/doc/performance_reports.html
POSIX/kernel stack saw considerable gains. Yet performance and efficiency of VPP abstraction was left unchanged.
Implementing those changes and keeping compatibility with VPP socket abstraction got more complex.

To not stifle further work in this area, the VPP socket abstraction will be removed.
First step will be deprecation with SPDK 20.07 release - CI will not build, nor test the VPP component.
SPDK 20.10 release will remove the VPP socket abstraction implementation from SPDK.

Thanks,
Tomek

> -----Original Message-----
> From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com>
> Sent: Monday, October 21, 2019 2:01 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] SPDK socket abstraction layer
> 
> Hello everyone,
> 
> Summary:
> 
> With this message I wanted to update SPDK community on state of VPP
> socket abstraction as of SPDK 19.07 release.
> At this time there does not seem to be a clear efficiency improvements with
> VPP. There is no further work planned on SPDK and VPP integration.
> 
> Details:
> 
> As some of you may remember, SPDK 18.04 release introduced support for
> alternative socket types. Along with that release, Vector Packet Processing
> (VPP)<https://wiki.fd.io/view/VPP> 18.01 was integrated with SPDK, by
> expanding socket abstraction to use VPP Communications Library (VCL).
> TCP/IP stack in VPP<https://wiki.fd.io/view/VPP/HostStack> was in early
> stages back then and has seen improvements throughout the last year.
> 
> To better use VPP capabilities, following fruitful collaboration with VPP team,
> in SPDK 19.07, this implementation was changed from VCL to VPP Session API
> from VPP 19.04.2.
> 
> VPP socket abstraction has met some challenges due to inherent design of
> both projects, in particular related to running separate processes and
> memory copies.
> Seeing improvements from original implementation was encouraging, yet
> measuring against posix socket abstraction (taking into consideration entire
> system, i.e. both processes), results are comparable. In other words,  at this
> time there does not seem to be a clear benefit of either socket abstraction
> from standpoint of CPU efficiency or IOPS.
> 
> With this message I just wanted to update SPDK community on state of
> socket abstraction layers as of SPDK 19.07 release. Each SPDK release always
> brings improvements to the abstraction and its implementations, with
> exciting work on more efficient use of kernel TCP stack - changes in SPDK
> 19.10 and SPDK 20.01.
> 
> However there is no active involvement at this point around VPP
> implementation of socket abstraction in SPDK. Contributions in this area are
> always welcome. In case you're interested in implementing further
> enhancements of VPP and SPDK integration feel free to reply, or to use one
> of the many SPDK community communications
> channels<https://spdk.io/community/>.
> 
> Thanks,
> Tomek
> 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org