Re: legality of interleaving untagged message segments

A receiver capable of tracking out-of-order messages is more complex
than one which requres in-order submission.  Support of out-of-order
message submission must be negotiated prior to stream initialization.  

A possible implementation: The receiver declares the maximum number of
simultaneous messages it will accept, possible values being an integer
greater than 0 or the special value "NO_LIMIT" (perhaps represented as
0).

A value of "1" means that all segments of message N must be submitted
before any segments of message N+1.  A value of "NO_LIMIT" means that
the receiver will accept any number of out-of-order messages up to the
size of the RQ/S_RQ.

Jeff

Michael Krause writes:
 > To: [email protected]
 > Subject: Re: [rddp] legality of interleaving untagged message segments
 > Date: Mon, 17 May 2004 09:47:55 -0700
 > 
 > 
 > Within a given LLP stream, you are correct that the optimization benefits 
 > may be minimal if I/O device is somewhat "distant" (read as relatively high 
 > latency) from the memory controller.  However, as we see the industry move 
 > towards PCI Express (PCIe) and with the potential to create more integrated 
 > memory controller chipsets, there may be greater benefit over time 
 > (especially with some architectures moving to be more NUMA oriented).
 > 
 > Given this may take time to occur and validate, would you be willing to 
 > make this a SHOULD NOT instead of a MUST NOT thus allowing leeway for 
 > future development while providing strong guidance that such interleaving 
 > is strongly discouraged?
 > 
 > Mike
 > 
 > 
 > At 10:28 PM 5/16/2004, Caitlin Bestler wrote:
 > >The current RDDP draft requires that messages be submitted
 > >to the LLP in order.
 > >
 > >It also requires that when a message is segmented, that the
 > >final segment MUST be the last one submitted to the LLP.
 > >
 > >It does not, however, define what is meant by submitting
 > >"a message" when in fact the DDP layer submits *segments*
 > >to the LLP.
 > >
 > >I believe this could allow a very strained interpretation
 > >that would allow interleaving of untagged segments to the
 > >LLP, as long as the *final* segments were submitted in order.
 > >For example, it might allow the following submission order:
 > >
 > >      Untagged MSN 7 segment 0 (not final)
 > >      Untagged MSN 8 segment 0 (not final)
 > >      Untagged MSN 7 segment 1 (final)
 > >      Untagged MSN 8 segment 1 (final)
 > >
 > >A typical out-of-order receiver would not even notice
 > >anything unusual about this sequence of untagged segments.
 > >However, it has a major impact on in-order receivers. It
 > >requires that multiple RQ (or S-RQ) entries be 'current'
 > >for the packet processor. If the in-order packet-processor
 > >is on the RNIC and the RQ/S-RQ is in host memory this is
 > >a totally unnecessary complication.
 > >
 > >I still believe that the sender should be free to re-order
 > >segments within a message (but SHOULD NOT unless it gains
 > >some advantage in doing so, such as sending data that is
 > >already cached), but I cannot come up with any benefit
 > >for the sender to interleave message segments. The complexity
 > >of doing so is simply too great for any potential additional
 > >caching gains.
 > >
 > >The fact that such sequences are so unlikely means that it
 > >is highly likely that there will be implementations that
 > >will not properly handle this case, whether or not the
 > >specification explicitly forbids it. This makes any theoretical
 > >advantages even more remote. Customers seldom care that
 > >your esoteric optimization is "legal", they just know that
 > >*your* RNIC is the one that isn't working with everyone else.
 > >
 > >So I believe the draft should be clarified to require that
 > >all segments for untagged message N MUST be submitted to
 > >the LLP before any segments for untagged message N+1 are
 > >submitted to the LLP.
 > >
 > >Doing so will have no impact on unordered receivers. I doubt
 > >it will have any impact on any existing senders, and it
 > >eliminates one corner condition that would require extra
 > >resources and testing for in-order receivers.
 > >
 > >
 > > >>
 > > >> Several months ago I raised a query as to whether DDP could/should
 > > >> mandate that the process of segmenting DDP Messages MUST NOT
 > > >> result in interleaving of messages. That is *all* segments for DDP
 > > >> Message N must be submitted to the transmitting LLP: before any
 > > >> segment of DDP Message N+1 is submitted.
 > > >>
 > > >> I believe the current draft could, in a very strained malicious way,
 > > >> be interpreted to only require:
 > > >>
 > > >>     that the segment with the L bit for each message be the last
 > > >>     segment for that message.
 > > >>
 > > >>     that the segments with L bits are themselves submitted in order.
 > > >>
 > > >> The only real response on the list, as I recall, was from Mike Krause
 > > >> who pointed out that such interleaving would not even be noticed
 > > >> by an out-of-order receiver. And that is completely correct.
 > > >>
 > > >> It does have an impact on in-order receivers, and particular on
 > > >> how many untagged buffers an in-order RNIC has to have looked
 > > >> up from the host-side receive queues (or worse SRQs).
 > > >>
 > > >> I doubt very much whether anyone will ever produce a sender
 > > >> with a segmentation algorithm that will violate this rule, and I
 > > >> am one of the strong advocates of allowing senders to use
 > > >> creative ordering *within* a message to optimize for sending
 > > >> data that is already in the RNICs cache.
 > > >>
 > > >> But it would be handy to have something firm to justify coding
 > > >> an RNIC based on this assumption.
 > > >>
 > > >> Do you feel that the current language already forbids interleaving
 > > >> of message segments (in terms of the order they are submitted
 > > >> to the LLP, and hence their LLP sequence numbers)?
 > > >>
 > > >> If so, it is probably enough to just have that stated on the
 > > >> reflector.
 > > >>
 > > >> Otherwise, as one of the principal advocates of having the protocols
 > > >> be friendly to *both* ordered and unordered implementations, would
 > > >> you support such a change?
 > > >>
 > > >> In my evaluation, the drafters are firmly in the "out of order" camp
 > > >> and hence do not see any need for this clarification.
 > > >>
 > > >
 > >
 > >
 > >--
 > >Caitlin Bestler
 > >http://asomi.com/
 > >
 > >_______________________________________________
 > >rddp mailing list
 > >[email protected]
 > >https://www1.ietf.org/mailman/listinfo/rddp
 > 
 > 
 > _______________________________________________
 > rddp mailing list
 > [email protected]
 > https://www1.ietf.org/mailman/listinfo/rddp

-- 
Jeff Dwork                     | [email protected]  408-307-2496 (cell)
Advanced Micro Devices, M/S 362| 408-749-5216 (desk) 408-774-7811 (fax)
PO Box 3453                    |----------------------------------------
Sunnyvale, Ca 94088-3453       |