RE: DDP 7.3 - Data Source MUST transmit DDP Messages in order

I agree that state explosion on the receiver is a big concern. 
It's really a question of requiring a scoreboarding receiver verses a
single counter. Pretty radically different complexity, when you throw in
that a message can be 2 gigabytes in length and segmented on any
arbitrary boundary. Look at SACK, for example - they understood that
state explosion is a huge deal, and at some point you have to punt.
Their form of punting was capping the number and then allowing even that
amount to be discarded if things really hit the fan. Note I'm _not_
suggesting that we implement SACK style limits on out-of-order
processing. If we were to change, I'd personally prefer going the other
direction - require in-order DDP segments on an inorder TCP stream.

I think the issue you have is section 7.3, with the SHOULD verses having
a MUST, specifically:

At the Data Source, DDP ...
*	SHOULD transmit DDP Segments within a DDP Message in increasing
MO order for Untagged DDP Messages and in increasing TO order for Tagged
DDP Messages. 

So hopefully the SHOULD means that the normal/fast path is in-order DDP
Segments within an in-order TCP stream, but it also means that the
receiver must be able to handle the out-of-order DDP segments within a
DDP message on an in-order TCP stream case. So it doesn't get you out of
the state explosion - it just means that you don't have to necessarily
handle it in the fast path.



Jim






> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of
Serag
> GadelRab
> Sent: Thursday, November 27, 2003 11:53 AM
> To: 'Caitlin Bestler'; [email protected]
> Cc: 'Michael Krause'
> Subject: RE: [rddp] DDP 7.3 - Data Source MUST transmit DDP Messages
in
> order
> 
> 
> From an implementation point of view, handling out-of-order DDP
segments
> within  in-order TCP flow is not going to come for free.  Special HW
> will have to be added to ensure that this case is handled properly
since
> it does not correspond exactly to the case where the TCP segments
arrive
> out of order.
> 
> The question is whether this added complexity in receivers is
warranted.
> I doubt that any transmitters will try to engage in such optimizations
> due to the complexity of scheduling OOO DDP segments.  Moreover, one
> could argue that this type of optimization is not necessary since the
> data that needs to be transmitted will almost certainly be resident in
> host memory when the transmit WQE is generated to the RNIC; so why
> burden receivers with such a functionality when it will probably be
> never exhibited by a transmitter?
> 
> Serag
> 
> 
> 
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of
> Caitlin Bestler
> Sent: Thursday, November 06, 2003 11:11 AM
> To: [email protected]
> Cc: Michael Krause
> Subject: RE: [rddp] DDP 7.3 - Data Source MUST transmit DDP Messages
in
> order
> 
> 
> 
> Michael Krause's replied to my post as follows:
> 
>  >> I would request the following:
>  >>
>  >> - Does the WG agree that the current wording is compatible
>  >>   with both interpretations? Or is the latter interpretation
>  >>   locked in by some other language the at I overlooked?
> 
>  >> - If there is agreement that the current language is at
>  >>   least potentially ambiguous, is there any objection to
>  >>   adding the clarification that all DDP Segments for a
>  >>   DDP Message must be submitted for transmission to the
>  >>   LLP before any Segment for a following DDP Message.
> 
>  > Implementations should be able to deal with OOO packet delivery and
> being
>  > able to target multiple receive buffers if they are posted (shared
or
> 
>   exclusive).
>  > Given there is no delta in the wire protocol or the expected
> interoperability in
>  > this respect, I do not favor trying to tighten how the  transmitter
> is implemented
>  > as that is potentially delving into the specific  design options in
> the hardware.
>  >
>  > Mike
> 
> Generally, I am in full agreement. Receivers should be as flexible as
> possible
> and senders should be given as much freedom as possible.
> 
> But the purpose of that is to allow senders to find potential
> optimizations
> in ways that might not have been obvious. For example, mandating that
a
> message MUST be segmented in a certain way would block a sender from
> optimizing
> for a DDP Segment that might have piggy-packed in the same SCTP Packet
> or TCP
> Segment that was already partially filled.
> 
> But I'm not sure I can see any benefit to the senders of working
ahead,
> especially when the last segments must be sent in order and when for
> many
> ULPs almost all untagged messages will fit in a single segment.
> 
> On the receiving side, however, the restriction can be beneficial to
an
> entire *class* of implementations -- those that ride on top of
optimized
> TCP solutions.
> 
> Since we have made it clear that the receiver does not have to
> pre-commit
> receive buffer for untagged messages (as in the class receive queue)
we
> have implicitly allowed for receivers that must fetch buffers "on
> demand".
> 
> Whatever implementation is used, clearly having a limit on the number
of
> fetched buffers that have not completed will result in a simpler
> problem.
> 
> Simplifying the problem is legitimate. Yes it benefits certain
> implementations,
> but that is because it simplified the problem.
> 
> Admittedly, the proposed restriction is of virtually no benefit to
> implementations that process DDP Segments in arrival order. For that
> reason it would be wrong to impose any costs on those implementations
> for this feature. But there is no impact on out-of-order receivers.
> 
> The *only* impact is that it might block hyper-aggressive re-ordering
of
> unsent pending DDP Segments. Even with this proposed additional
> language, such re-ordering is still possible *within* an untagged
> message and with tagged segments. In my opinion that is more than
enough
> slack for potential optimizations (in fact I suspect that very few
> implementations are even considering such optimizations).
> 
> 
> 
> 
> 
> 
> --
> Caitlin Bestler - [email protected] - http://asomi.com/
> http://asomi.com/CaitlinBestlerPublicPgpKey.html
> 
> 
> _______________________________________________
> rddp mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
> 
> _______________________________________________
> rddp mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/rddp