RE: RE: MPA Responses - Connection Startup Ordering Requirements

Embedded...

Paul R. Culley
HP Fellow
281-514-5543


> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On 
> Behalf Of Caitlin Bestler
> Sent: Thursday, February 05, 2004 3:00 PM
> To: 'RDDP'
> Subject: [rddp] RE: MPA Responses - Connection Startup 
> Ordering Requirements
> 
> 
> The response draft does clarify the rationale on when the 
> respondent can send the first DDP Segment, but I am still not 
> convinced that there is a need for the restriction.
> 
> To recap:
> 
> Prior to this response, there was an assertion that the 
> responding side must wait for the first message from the 
> initiator before it could send its first message because 
> without the response there was no way to know that there was 
> an RDMA endpoint enabled yet.
> 
> That is not the case, which is why the SCTP adaptation is not 
> required. Basically, unless the ULP has a specific reason to 
> postpone the assignment
> of an endpoint, an endpoint will have been assigned as soon 
> as the initiating side requests the connection. This is how 
> both the DAT and IT-API interfaces work. The initiating side 
> (typically the client) knows all of its endpoint requirements 
> in advance, it only needs to know if they are acceptable to 
> the responding side.
[prc] The initiating side DOES NOT know if markers are to be used until
the MPA Response Frame is received.  As such it cannot begin processing
FPDUs until after the frame has been processed.
In addition, it may not know the size of Send messages, the RDMA Read
resources required and some application values.  Of course these can be
"known" by design, or the responder application can voluntarily delay
the first message until the initiator sends an FPDU.

> 
> But there is a difference between SCTP and TCP that *might* 
> justify this restriction: SCTP Data Chunks containing DDP 
> Segments received out-of-order can still be placed, because 
> not only is the RDMA endpoint already enabled but the DDP 
> Segments cannot be identified.
> 
> With TCP, however, DDP Segments cannot be extracted from the 
> TCP stream until after the MPA Reply Frame is received (or 
> technically the first segment, since that would include the 
> length, but realistically the MPA Reply Frame is almost 
> certain to fit within the first segment).
[prc] at the moment, the MPA Reply frame length can be up to 64k long,
potentially quite a bit longer than a segment.
> 
> That means that once the ULP has signaled the intent to enter 
> MPA mode, the receiving stack knows how to process any TCP 
> segment received. If TCP segments are received out of order, 
> it can hold them until they can be processed, or it can drop 
> them and let them retransmit.
[prc]If we implement this way, the MPA/DDP stack must now implement the
following functionality that was not required previously (it could have
been done with legacy stack and ULP code):
1: accept TCP segments, re-ordering if necessary to find the beginning
of the MPA Response frame.
2: accept TCP segments, re-ordering if necessary to get the entire MPA
Response frame,
3: Decode the Marker, CRC, and other bit fields, setting up the MPA/DDP
"engine" to process FPDUs.
4: Deliver the Private data to the ULP.
5: Hold off processing FPDUs until the ULP has indicated that it is
ready to do so (posted receive buffers of appropriate size, set RDMA
Read resources etc.).
6: Process any already delivered or out of order FPDUs that were
received prior to the MPA Reply frame completion.

Alternately, we could not require the last two steps, and just let FPDUs
arrive, potentially not matching the buffer sizes etc.; thus requiring
the initiator ULP to insert a response FPDU anyway, in at least some
ULPs.

> 
> The response talks about "transferring data", as though prior 
> to fully enabling RDMA over MPA the connection were still 
> being managed as a conventional TCP connection.
[prc]The idea of managing the MPA startup over a conventional TCP
connection was more easily enabled if we do not allow "early" FPDU
sending by the responder.  Admittedly, the extension of this concept
when allowing the "early" FPDUs was the idea we analyzed, and there are
other possible implementations.
> 
> That is one possible decision, but it is not a protocol 
> requirement. The stack, however it is structured, has been 
> clearly instructed that it will be receiving a single MPA 
> Reply Frame followed by a series of MPA framed DDP Segments.
> 
> The existence of this intermediate state make it feasible for 
> the receiving stack to either hold or drop out-of-order 
> segments in a way that would not normally be done for a 
> normal TCP connection.
> 
> So the receiving stack already has the information required 
> to handle these out-of-order segments properly. There is no 
> reason to mistake them for conventional TCP stream payload. 
> This makes deferred processing (by holding the buffers or 
> waiting for the retransmit) simple.
> 
> And given that the probability of delivery for any one TCP 
> Segment is extremely high in any environment where RDMA 
> should be used, the cost of properly handling these out of 
> order segments is very low.
> 
> Keep in mind that during this intermediate state the receiver 
> is NOT required to process these out-of-order segments, the 
> only thing it MUST do is that it MUST NOT process them as 
> part of a conventional TCP stream -- but it already knows not 
> to do that.
> 
> By contrast, the benefit of removing this restriction can be 
> very high for a class of application that tends to have short 
> lived sessions.
> 
> Such applications could piggy-back the first request in the 
> MPA Request Frame's private data. This would allow the first 
> DDP Segment in the response to be piggy-backed in the same 
> TCP segment as the MPA Reply Frame. This can eliminate an 
> entire round-trip from the exchange.
> 
> This is a potential major benefit to at least one class of 
> applications, and it only requires that the receiver properly 
> use information that it already possesses. No additional 
> context data is required.
> 
> Also consider that any ULP that wishes to require the 
> initiator to send the first message is still able to do so. 
> The fact that the respondent's stack is not enforcing this 
> restriction will in no way impact any ULP. If a non-compliant 
> respondent sent its message early, *and* they were received 
> out of order, it *might* have an impact of causing an 
> artificial
> packet drop. This is an *extremely* minor cost compared to 
> the benefit that "quick response" applications could gain.
> 
> Also, it should be noted that  if a ULP dictated the size of 
> the private data in the response then the initiating size 
> would actually be able to process out-of-order segments immediately.
> 
[prc] I think we agree that it is possible to implement your "early"
response FPDU rule in any number of ways.  I think we are primarily
disagreeing on the cost, not of run time processing, but in
implementation design/debug/verification time.  It seems that you
believe the cost is low (or perhaps you think the benefits outweigh any
possible costs) while some of us believe that the cost is high and
benefits are limited enough not to be worth it.

As for short lived sessions; I suspect that if the session is
sufficiently short lived that the time savings you describe becomes an
issue, that RDMA/DDP over TCP may not be the best choice.  It is a bit
early in product life cycles to tell, but I suspect that various setup
overheads may make short lived sessions perform no better than straight
TCP, SCTP, or even datagrams.  The real win for RDMA/DDP is when LOTS of
data must be moved directly to application buffers; that is when the
copy, interrupt and kernel processing overheads of legacy stacks is
eliminated or reduced enough to make the use of RDMA/DDP worth it.

I think it is time for some other voices to say something about this
issue; the current discussion (several months now) has been limited to
Caitlin and myself.  I think I am proxying for a significant list of
other authors and implementers, but it is not clear to me that anyone
other than Caitlin cares much about this issue.

Thanks; 

Paul C.
> 
> -- 
> Caitlin Bestler - [email protected] - http://asomi.com/
> 
> 
> _______________________________________________
> rddp mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/rddp
>