Re: Rgerading RDMA/SDP and traditional socket apps

Some things to keep in mind:

- Synchronous sockets calls can be completed without any data actually 
being sent in a traditional network stack.  Kernel buffers the data to 
transmit / re-transmit and the application proceeds normally.  This model 
does not reduce total memory consumption.  As link speeds increase, 
synchronous call will either fail (if non-block set) or stall until kernel 
memory becomes available as a function of acknowledgement rate.

- Asynchronous sockets calls can be completed without any data being 
sent.  This is no different than synchronous sockets calls.  However, 
asynchronous can use the application buffer for retransmission (assuming 
operation is not completed to the consumer) thus reducing memory pressure 
in the kernel.

- Asynchronous calls (e.g. sockets) map best to RDMA infrastructures and 
provide optimal performance as well as memory resource management.

- Synchronous sockets calls can be supported with SDP and can use the 
one-copy or zcopy modes to mitigate local as well as end-to-end application 
latency.

- RDMA infrastructure enables OS bypass operation and the ability to 
mitigate interrupts within the system.  This can yield a much more 
responsive solution as well as more predictable behavior as system load 
increases.

- If applications use synchronous sockets, then they should be designed to 
deal with the inevitable stall due to a blocked system call.  Use of SDP is 
orthogonal to this and therefore its presence should not have an impact on 
application design.  Yes, programmers often make assumptions about how fast 
a call will return but if they are creating a robust design, then they 
should have accounted for the stall..


Note: The OpenGroup Interconnect Software Consortium is defining a set of 
Sockets extensions that provide asynchronous interfaces and explicit memory 
management.  These extensions will enable Sockets applications to fully 
exploit the performance benefits of RDMA while enabling the OS to take the 
guess work out of memory management.  These extensions are to be part of 
the Unix branding so should be available across all Unix and hopefully 
Linux implementations in the next couple of years.  The specification is 
nearly complete and should be final by the end of this year.

Also note that Winsocks has been shipping async sockets support for quite 
some time and that Winsocks Direct, which is the basis for SDP, is a good 
proof of concept of the performance benefits.  There are a couple of white 
papers that provide a good overview of what one might expect when running SDP.

Mike


At 07:52 AM 10/4/2004, Caitlin Bestler wrote:

>Sukanta ganguly said:
> > Hi,
> >   I do not think the issue that Mohan is raising is
> > about more traffic across the wire. In essence the
> > traffic on the wire will most probably be less that
> > than the none SDP protocol is concerned.
> >    The question of whether the application with
> > synchronous socket approch will have to wait longer
> > for the RNIC to do the DMA onto its local buffer
> > before the app on the other side get controll of its
> > buffer is concerned is a valid one.
> >    So the zero copy effect is one aspect that will
> > reduce the amount of real time the synchronous socket
> > I/O app will face latency issues. As far as the reuse
> > of the buffer is concerned, it will depend on a lot of
> > elements, the LAN/WAN link speed will be one, the
> > performance of the remote node will be another one.
> >    So o clearly answer this the question, the
> > application should be designed with more memory to be
> > used as during such communication process it may not
> > have the reusebility of its buffers at a very high
> > rate. I think higher memroy consumption will be an
> > element to wonder rather than latency, though latency
> > is also an after effect.
> >
> >
> > Thanks
> > SG
> >
>
>
>Good point.
>
>But it is important to remember that this can be more of an
>accounting issue than an actual savings of memory.
>
>Use of SDP, or for that matter an RDMA-aware interface,
>will have the effect of replacing usage of system buffers
>with usage of application buffers.
>
>The benefit of zero-copying may result in resulting in
>more total buffering, or it might merely be transferring
>the same bytes from column A (system buffering) to
>column B (application buffering).
>
>The applicability statement covers these issues, though
>mostly concentrating on receive buffering.  There are
>applications that truly need the benefits of shared buffering
>across multiple applications -- and they might be poor
>candidates for RDMA.  For many other applications, and
>I personally believe most, the system buffer is a bottleneck
>rather than an enabler.  It is much easier to double the
>amount of RAM in your system than to double its bandwidth.
>
>
>_______________________________________________
>rddp mailing list
>[email protected]
>https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/rddp