RE: legality of interleaving untagged message segments

I hope you are missing the point; all RDDP/MPA receivers MUST be able to
deal with out of order segments caused by lost or delayed TCP segments.
So there is no need for a receiver to negotiate this.  Out of order RDDP
segments within a single message, as sent by the transmitter (but in
order as far as TCP is concerned) does not appear to be any harder.  Out
of order messages (segments with the last bit) must also be handled when
caused by network issues.

We are talking about allowing some (probably minor) optimizations at the
transmitter to send some segments out of order and whether to allow
them.  I personally believe that actually taking advantage of allowable
out of order segments at the transmitter will be difficult and of very
little actual performance or buffering gain at the transmitter.  But a
receiver that has to handle this routinely might lose some performance
if it tried to use spatial caching for Page table lookups or buffer
stores.

I for one, think that requiring everything to be in order out of the
sender will probably save more at the receiver than would ever be lost
at the sender; as such I would be in favor of MUST on always in order.
Or just leave the spec as it is (with some textual clarification as
suggested by Caitlin.  I would absolutely object to negotiating out of
order activities, just to allow these minor transmitter optimizations.

Paul R. Culley
HP Fellow
281-514-5543
 

> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On 
> Behalf Of Jeff Dwork
> Sent: Monday, May 17, 2004 1:17 PM
> To: [email protected]
> Subject: Re: [rddp] legality of interleaving untagged message segments
> 
> A receiver capable of tracking out-of-order messages is more 
> complex than one which requres in-order submission.  Support 
> of out-of-order message submission must be negotiated prior 
> to stream initialization.  
> 
> A possible implementation: The receiver declares the maximum 
> number of simultaneous messages it will accept, possible 
> values being an integer greater than 0 or the special value 
> "NO_LIMIT" (perhaps represented as 0).
> 
> A value of "1" means that all segments of message N must be 
> submitted before any segments of message N+1.  A value of 
> "NO_LIMIT" means that the receiver will accept any number of 
> out-of-order messages up to the size of the RQ/S_RQ.
> 
> Jeff
> 
> Michael Krause writes:
>  > To: [email protected]
>  > Subject: Re: [rddp] legality of interleaving untagged 
> message segments  > Date: Mon, 17 May 2004 09:47:55 -0700  >  
> >  > Within a given LLP stream, you are correct that the 
> optimization benefits  > may be minimal if I/O device is 
> somewhat "distant" (read as relatively high  > latency) from 
> the memory controller.  However, as we see the industry move  
> > towards PCI Express (PCIe) and with the potential to create 
> more integrated  > memory controller chipsets, there may be 
> greater benefit over time  > (especially with some 
> architectures moving to be more NUMA oriented).
>  >
>  > Given this may take time to occur and validate, would you 
> be willing to  > make this a SHOULD NOT instead of a MUST NOT 
> thus allowing leeway for  > future development while 
> providing strong guidance that such interleaving  > is 
> strongly discouraged?
>  >
>  > Mike
>  >
>  >
>  > At 10:28 PM 5/16/2004, Caitlin Bestler wrote:
>  > >The current RDDP draft requires that messages be 
> submitted  > >to the LLP in order.
>  > >
>  > >It also requires that when a message is segmented, that 
> the  > >final segment MUST be the last one submitted to the LLP.
>  > >
>  > >It does not, however, define what is meant by submitting  
> > >"a message" when in fact the DDP layer submits *segments*  
> > >to the LLP.
>  > >
>  > >I believe this could allow a very strained interpretation 
>  > >that would allow interleaving of untagged segments to the 
>  > >LLP, as long as the *final* segments were submitted in order.
>  > >For example, it might allow the following submission order:
>  > >
>  > >      Untagged MSN 7 segment 0 (not final)
>  > >      Untagged MSN 8 segment 0 (not final)
>  > >      Untagged MSN 7 segment 1 (final)
>  > >      Untagged MSN 8 segment 1 (final)
>  > >
>  > >A typical out-of-order receiver would not even notice  > 
> >anything unusual about this sequence of untagged segments.
>  > >However, it has a major impact on in-order receivers. It  
> > >requires that multiple RQ (or S-RQ) entries be 'current'
>  > >for the packet processor. If the in-order 
> packet-processor  > >is on the RNIC and the RQ/S-RQ is in 
> host memory this is  > >a totally unnecessary complication.
>  > >
>  > >I still believe that the sender should be free to 
> re-order  > >segments within a message (but SHOULD NOT unless 
> it gains  > >some advantage in doing so, such as sending data 
> that is  > >already cached), but I cannot come up with any 
> benefit  > >for the sender to interleave message segments. 
> The complexity  > >of doing so is simply too great for any 
> potential additional  > >caching gains.
>  > >
>  > >The fact that such sequences are so unlikely means that 
> it  > >is highly likely that there will be implementations 
> that  > >will not properly handle this case, whether or not 
> the  > >specification explicitly forbids it. This makes any 
> theoretical  > >advantages even more remote. Customers seldom 
> care that  > >your esoteric optimization is "legal", they 
> just know that  > >*your* RNIC is the one that isn't working 
> with everyone else.
>  > >
>  > >So I believe the draft should be clarified to require 
> that  > >all segments for untagged message N MUST be 
> submitted to  > >the LLP before any segments for untagged 
> message N+1 are  > >submitted to the LLP.
>  > >
>  > >Doing so will have no impact on unordered receivers. I 
> doubt  > >it will have any impact on any existing senders, 
> and it  > >eliminates one corner condition that would require 
> extra  > >resources and testing for in-order receivers.
>  > >
>  > >
>  > > >>
>  > > >> Several months ago I raised a query as to whether DDP 
> could/should  > > >> mandate that the process of segmenting 
> DDP Messages MUST NOT  > > >> result in interleaving of 
> messages. That is *all* segments for DDP  > > >> Message N 
> must be submitted to the transmitting LLP: before any  > > >> 
> segment of DDP Message N+1 is submitted.
>  > > >>
>  > > >> I believe the current draft could, in a very strained 
> malicious way,  > > >> be interpreted to only require:
>  > > >>
>  > > >>     that the segment with the L bit for each message 
> be the last
>  > > >>     segment for that message.
>  > > >>
>  > > >>     that the segments with L bits are themselves 
> submitted in order.
>  > > >>
>  > > >> The only real response on the list, as I recall, was 
> from Mike Krause  > > >> who pointed out that such 
> interleaving would not even be noticed  > > >> by an 
> out-of-order receiver. And that is completely correct.
>  > > >>
>  > > >> It does have an impact on in-order receivers, and 
> particular on  > > >> how many untagged buffers an in-order 
> RNIC has to have looked  > > >> up from the host-side receive 
> queues (or worse SRQs).
>  > > >>
>  > > >> I doubt very much whether anyone will ever produce a 
> sender  > > >> with a segmentation algorithm that will 
> violate this rule, and I  > > >> am one of the strong 
> advocates of allowing senders to use  > > >> creative 
> ordering *within* a message to optimize for sending  > > >> 
> data that is already in the RNICs cache.
>  > > >>
>  > > >> But it would be handy to have something firm to 
> justify coding  > > >> an RNIC based on this assumption.
>  > > >>
>  > > >> Do you feel that the current language already forbids 
> interleaving  > > >> of message segments (in terms of the 
> order they are submitted  > > >> to the LLP, and hence their 
> LLP sequence numbers)?
>  > > >>
>  > > >> If so, it is probably enough to just have that stated 
> on the  > > >> reflector.
>  > > >>
>  > > >> Otherwise, as one of the principal advocates of 
> having the protocols  > > >> be friendly to *both* ordered 
> and unordered implementations, would  > > >> you support such 
> a change?
>  > > >>
>  > > >> In my evaluation, the drafters are firmly in the "out 
> of order" camp  > > >> and hence do not see any need for this 
> clarification.
>  > > >>
>  > > >
>  > >
>  > >
>  > >--
>  > >Caitlin Bestler
>  > >http://asomi.com/
>  > >
>  > >_______________________________________________
>  > >rddp mailing list
>  > >[email protected]
>  > >https://www1.ietf.org/mailman/listinfo/rddp
>  >
>  >
>  > _______________________________________________
>  > rddp mailing list
>  > [email protected]
>  > https://www1.ietf.org/mailman/listinfo/rddp
> 
> -- 
> Jeff Dwork                     | [email protected]  
> 408-307-2496 (cell)
> Advanced Micro Devices, M/S 362| 408-749-5216 (desk) 
> 408-774-7811 (fax)
> PO Box 3453                    
> |----------------------------------------
> Sunnyvale, Ca 94088-3453       | 
> 
> 
> 
> _______________________________________________
> rddp mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/rddp
> 
>