Re: RE: MPA Responses - Connection Startup Ordering Requirements

On Feb 6, 2004, at 9:32 AM, Culley, Paul wrote:

> Embedded...
>
> Paul R. Culley
> HP Fellow
> 281-514-5543
>
>
>> -----Original Message-----
>> From: [email protected] [mailto:[email protected]] On
>> Behalf Of Caitlin Bestler
>> Sent: Thursday, February 05, 2004 3:00 PM
>> To: 'RDDP'
>> Subject: [rddp] RE: MPA Responses - Connection Startup
>> Ordering Requirements
>>
>>
>> The response draft does clarify the rationale on when the
>> respondent can send the first DDP Segment, but I am still not
>> convinced that there is a need for the restriction.
>>
>> To recap:
>>
>> Prior to this response, there was an assertion that the
>> responding side must wait for the first message from the
>> initiator before it could send its first message because
>> without the response there was no way to know that there was
>> an RDMA endpoint enabled yet.
>>
>> That is not the case, which is why the SCTP adaptation is not
>> required. Basically, unless the ULP has a specific reason to
>> postpone the assignment
>> of an endpoint, an endpoint will have been assigned as soon
>> as the initiating side requests the connection. This is how
>> both the DAT and IT-API interfaces work. The initiating side
>> (typically the client) knows all of its endpoint requirements
>> in advance, it only needs to know if they are acceptable to
>> the responding side.
> [prc] The initiating side DOES NOT know if markers are to be used until
> the MPA Response Frame is received.  As such it cannot begin processing
> FPDUs until after the frame has been processed.

Any receiver knows whether or not *it* is insisting on markers, and 
hence
can process it's own incoming DDP Segments for placement even before
completing the MPA Response Frame.

Obviously, you cannot *send* a DDP Segment until you receive the
peer's MPA Request/Response Frame. That is the essential minimal rule.

> In addition, it may not know the size of Send messages, the RDMA Read
> resources required and some application values.  Of course these can be
> "known" by design, or the responder application can voluntarily delay
> the first message until the initiator sends an FPDU.
>
These factors are agreed via ULP specific methods. If these are included
in the private data, then for *that* application protocol you would 
indeed
have to wait for the first FPDU.

The question is whether the standard will *forbid* any application from
sending a first message even when all information needed to support that
message is already present at both ends -- and in doing so adding an 
additional
round-trip delay.

The ULP remains free to dictate which end will talk first. But by 
having the
respondent talk first an application could eliminate a round-trip delay 
for
simple fetch-oriented protocols. Why should the MPA specification block 
this
optimization? The information required to support this feature is 
already
available to the receiver, it merely has to use it.

>>
>> But there is a difference between SCTP and TCP that *might*
>> justify this restriction: SCTP Data Chunks containing DDP
>> Segments received out-of-order can still be placed, because
>> not only is the RDMA endpoint already enabled but the DDP
>> Segments cannot be identified.
>>
>> With TCP, however, DDP Segments cannot be extracted from the
>> TCP stream until after the MPA Reply Frame is received (or
>> technically the first segment, since that would include the
>> length, but realistically the MPA Reply Frame is almost
>> certain to fit within the first segment).
> [prc] at the moment, the MPA Reply frame length can be up to 64k long,
> potentially quite a bit longer than a segment.

Even if it is 64K long, the length is in the very first segment.

There are two reasons why it will tend to fit in one segment:

a) most applications will only send the data that is required before
    the RDMA connection is established. Credits and session identifying
    information will almost certainly fit within a single segment. Even
    an immediate command ("get /xyz/abc.html") will probably still fit.
b) Transport-neutral applications will be motivated to keep the private
    data length compatible with the InfiniBand CM protocol -- which would
	almost always keep it in a single segment.
c) ULPs are quite likely to place a limit on the private data size for
    their application, because doing so limits the pre-committed 
buffering
    required to accept connections. Just as applications do not want to
	commit 64KB for each possible untagged message, they do not want to
	lock down 64KB for each possible connection request.

>>
>> That means that once the ULP has signaled the intent to enter
>> MPA mode, the receiving stack knows how to process any TCP
>> segment received. If TCP segments are received out of order,
>> it can hold them until they can be processed, or it can drop
>> them and let them retransmit.
> [prc]If we implement this way, the MPA/DDP stack must now implement the
> following functionality that was not required previously (it could have
> been done with legacy stack and ULP code):
> 1: accept TCP segments, re-ordering if necessary to find the beginning
> of the MPA Response frame.
> 2: accept TCP segments, re-ordering if necessary to get the entire MPA
> Response frame,
> 3: Decode the Marker, CRC, and other bit fields, setting up the MPA/DDP
> "engine" to process FPDUs.
> 4: Deliver the Private data to the ULP.
> 5: Hold off processing FPDUs until the ULP has indicated that it is
> ready to do so (posted receive buffers of appropriate size, set RDMA
> Read resources etc.).
> 6: Process any already delivered or out of order FPDUs that were
> received prior to the MPA Reply frame completion.
>

You are taking the weaknesses of implementing MPA Start Frame processing
via the conventional stack and making them requirements.

The Start Frame allows the ULP to transition to MPA mode without 
revocation
at a known point. When this request is made, it is known that all 
packets
further in the stream will be part of the MPA Start Frame or FPDUs.

You do indeed have to place and deliver the Private Data to the ULP, but
the data can be placed out of order once the first segment is received
and the buffer length is known.

You are also assuming that the ULP is not assigning the RDMA Endpoint
on the initiating side until after receiving the MPA Response Frame.
That is not a requirement, in fact that option is not even supported
in the DAT or IT-API protocols. So at the minimum we can say that there
will be numerous applications where the initiating-side's RDMA Endpoint
is selected *before* the TCP connection is initiated.

In that case, the only requirement is that out-of-order TCP Segments
received before the private data size is known must be held for later
delivery or dropped.



Just as importantly, any implementation built on top of a TCP
stack will have *zero* problem with supporting quick responses.

Again, the same issue comes up. The benefit of allowing quick
responses can be very substantial for some applications. A receiver
can process this by either re-delivering the payload or by dropping
segments received out-of-order before the private data size is known.
If DDP Segments are received before an RDMA Endpoint is assigned,
then the respondent side is not in compliance with the ULP and
there is no harm in dropping the connection.

There are distinct scenarios where this type of packet COULD be
processed with  MAJOR benefits. Why should this practice be forbidden?





> Alternately, we could not require the last two steps, and just let 
> FPDUs
> arrive, potentially not matching the buffer sizes etc.; thus requiring
> the initiator ULP to insert a response FPDU anyway, in at least some
> ULPs.
>
>>
>> The response talks about "transferring data", as though prior
>> to fully enabling RDMA over MPA the connection were still
>> being managed as a conventional TCP connection.
> [prc]The idea of managing the MPA startup over a conventional TCP
> connection was more easily enabled if we do not allow "early" FPDU
> sending by the responder.  Admittedly, the extension of this concept
> when allowing the "early" FPDUs was the idea we analyzed, and there are
> other possible implementations.
>>
>> That is one possible decision, but it is not a protocol
>> requirement. The stack, however it is structured, has been
>> clearly instructed that it will be receiving a single MPA
>> Reply Frame followed by a series of MPA framed DDP Segments.
>>
>> The existence of this intermediate state make it feasible for
>> the receiving stack to either hold or drop out-of-order
>> segments in a way that would not normally be done for a
>> normal TCP connection.
>>
>> So the receiving stack already has the information required
>> to handle these out-of-order segments properly. There is no
>> reason to mistake them for conventional TCP stream payload.
>> This makes deferred processing (by holding the buffers or
>> waiting for the retransmit) simple.
>>
>> And given that the probability of delivery for any one TCP
>> Segment is extremely high in any environment where RDMA
>> should be used, the cost of properly handling these out of
>> order segments is very low.
>>
>> Keep in mind that during this intermediate state the receiver
>> is NOT required to process these out-of-order segments, the
>> only thing it MUST do is that it MUST NOT process them as
>> part of a conventional TCP stream -- but it already knows not
>> to do that.
>>
>> By contrast, the benefit of removing this restriction can be
>> very high for a class of application that tends to have short
>> lived sessions.
>>
>> Such applications could piggy-back the first request in the
>> MPA Request Frame's private data. This would allow the first
>> DDP Segment in the response to be piggy-backed in the same
>> TCP segment as the MPA Reply Frame. This can eliminate an
>> entire round-trip from the exchange.
>>
>> This is a potential major benefit to at least one class of
>> applications, and it only requires that the receiver properly
>> use information that it already possesses. No additional
>> context data is required.
>>
>> Also consider that any ULP that wishes to require the
>> initiator to send the first message is still able to do so.
>> The fact that the respondent's stack is not enforcing this
>> restriction will in no way impact any ULP. If a non-compliant
>> respondent sent its message early, *and* they were received
>> out of order, it *might* have an impact of causing an
>> artificial
>> packet drop. This is an *extremely* minor cost compared to
>> the benefit that "quick response" applications could gain.
>>
>> Also, it should be noted that  if a ULP dictated the size of
>> the private data in the response then the initiating size
>> would actually be able to process out-of-order segments immediately.
>>
> [prc] I think we agree that it is possible to implement your "early"
> response FPDU rule in any number of ways.  I think we are primarily
> disagreeing on the cost, not of run time processing, but in
> implementation design/debug/verification time.  It seems that you
> believe the cost is low (or perhaps you think the benefits outweigh any
> possible costs) while some of us believe that the cost is high and
> benefits are limited enough not to be worth it.
>
You are proposing a permanent protocol restriction that is not
justified by complexity for the problem space, only of certain
early implementation strategies.


> As for short lived sessions; I suspect that if the session is
> sufficiently short lived that the time savings you describe becomes an
> issue, that RDMA/DDP over TCP may not be the best choice.  It is a bit
> early in product life cycles to tell, but I suspect that various setup
> overheads may make short lived sessions perform no better than straight
> TCP, SCTP, or even datagrams.  The real win for RDMA/DDP is when LOTS 
> of
> data must be moved directly to application buffers; that is when the
> copy, interrupt and kernel processing overheads of legacy stacks is
> eliminated or reduced enough to make the use of RDMA/DDP worth it.

A single large file would benefit from RDMA, and adding an extra 
round-trip
over a long-haul connection is still an unnecessary delay.

>
> I think it is time for some other voices to say something about this
> issue; the current discussion (several months now) has been limited to
> Caitlin and myself.  I think I am proxying for a significant list of
> other authors and implementers, but it is not clear to me that anyone
> other than Caitlin cares much about this issue.
>

David was the one who raised the issue about the extra round-trip delay.

This issue was also discussed within the DAT Collaborative, and the
consensus there was that the requirement was without benefit.

When viewed from the point of an end-to-end application design, any
additional restriction is an added cost in validation/qa/etc.