RE: MPA Responses - Connection Startup Ordering Requirements

The response draft does clarify the rationale on when the respondent
can send the first DDP Segment, but I am still not convinced that
there is a need for the restriction.

To recap:

Prior to this response, there was an assertion that the responding
side must wait for the first message from the initiator before it
could send its first message because without the response there was
no way to know that there was an RDMA endpoint enabled yet.

That is not the case, which is why the SCTP adaptation is not required.
Basically, unless the ULP has a specific reason to postpone the 
assignment
of an endpoint, an endpoint will have been assigned as soon as the
initiating side requests the connection. This is how both the DAT
and IT-API interfaces work. The initiating side (typically the client)
knows all of its endpoint requirements in advance, it only needs to know
if they are acceptable to the responding side.

But there is a difference between SCTP and TCP that *might* justify
this restriction: SCTP Data Chunks containing DDP Segments received
out-of-order can still be placed, because not only is the RDMA endpoint
already enabled but the DDP Segments cannot be identified.

With TCP, however, DDP Segments cannot be extracted from the TCP stream
until after the MPA Reply Frame is received (or technically the first
segment, since that would include the length, but realistically the
MPA Reply Frame is almost certain to fit within the first segment).

That means that once the ULP has signaled the intent to enter MPA mode,
the receiving stack knows how to process any TCP segment received. If
TCP segments are received out of order, it can hold them until they can
be processed, or it can drop them and let them retransmit.

The response talks about "transferring data", as though prior to fully
enabling RDMA over MPA the connection were still being managed as a
conventional TCP connection.

That is one possible decision, but it is not a protocol requirement.
The stack, however it is structured, has been clearly instructed that
it will be receiving a single MPA Reply Frame followed by a series
of MPA framed DDP Segments.

The existence of this intermediate state make it feasible for the
receiving stack to either hold or drop out-of-order segments in a
way that would not normally be done for a normal TCP connection.

So the receiving stack already has the information required to handle
these out-of-order segments properly. There is no reason to mistake
them for conventional TCP stream payload. This makes deferred processing
(by holding the buffers or waiting for the retransmit) simple.

And given that the probability of delivery for any one TCP Segment is
extremely high in any environment where RDMA should be used, the cost
of properly handling these out of order segments is very low.

Keep in mind that during this intermediate state the receiver is NOT
required to process these out-of-order segments, the only thing it
MUST do is that it MUST NOT process them as part of a conventional
TCP stream -- but it already knows not to do that.

By contrast, the benefit of removing this restriction can be very
high for a class of application that tends to have short lived sessions.

Such applications could piggy-back the first request in the MPA
Request Frame's private data. This would allow the first DDP Segment
in the response to be piggy-backed in the same TCP segment as the
MPA Reply Frame. This can eliminate an entire round-trip from the
exchange.

This is a potential major benefit to at least one class of applications,
and it only requires that the receiver properly use information that it
already possesses. No additional context data is required.

Also consider that any ULP that wishes to require the initiator to send
the first message is still able to do so. The fact that the respondent's
stack is not enforcing this restriction will in no way impact any ULP.
If a non-compliant respondent sent its message early, *and* they were
received out of order, it *might* have an impact of causing an 
artificial
packet drop. This is an *extremely* minor cost compared to the benefit
that "quick response" applications could gain.

Also, it should be noted that  if a ULP dictated the size of the
private data in the response then the initiating size would actually
be able to process out-of-order segments immediately.


-- 
Caitlin Bestler - [email protected] - http://asomi.com/