Re: legality of interleaving untagged message segments

Within a given LLP stream, you are correct that the optimization benefits 
may be minimal if I/O device is somewhat "distant" (read as relatively high 
latency) from the memory controller.  However, as we see the industry move 
towards PCI Express (PCIe) and with the potential to create more integrated 
memory controller chipsets, there may be greater benefit over time 
(especially with some architectures moving to be more NUMA oriented).

Given this may take time to occur and validate, would you be willing to 
make this a SHOULD NOT instead of a MUST NOT thus allowing leeway for 
future development while providing strong guidance that such interleaving 
is strongly discouraged?

Mike


At 10:28 PM 5/16/2004, Caitlin Bestler wrote:
>The current RDDP draft requires that messages be submitted
>to the LLP in order.
>
>It also requires that when a message is segmented, that the
>final segment MUST be the last one submitted to the LLP.
>
>It does not, however, define what is meant by submitting
>"a message" when in fact the DDP layer submits *segments*
>to the LLP.
>
>I believe this could allow a very strained interpretation
>that would allow interleaving of untagged segments to the
>LLP, as long as the *final* segments were submitted in order.
>For example, it might allow the following submission order:
>
>      Untagged MSN 7 segment 0 (not final)
>      Untagged MSN 8 segment 0 (not final)
>      Untagged MSN 7 segment 1 (final)
>      Untagged MSN 8 segment 1 (final)
>
>A typical out-of-order receiver would not even notice
>anything unusual about this sequence of untagged segments.
>However, it has a major impact on in-order receivers. It
>requires that multiple RQ (or S-RQ) entries be 'current'
>for the packet processor. If the in-order packet-processor
>is on the RNIC and the RQ/S-RQ is in host memory this is
>a totally unnecessary complication.
>
>I still believe that the sender should be free to re-order
>segments within a message (but SHOULD NOT unless it gains
>some advantage in doing so, such as sending data that is
>already cached), but I cannot come up with any benefit
>for the sender to interleave message segments. The complexity
>of doing so is simply too great for any potential additional
>caching gains.
>
>The fact that such sequences are so unlikely means that it
>is highly likely that there will be implementations that
>will not properly handle this case, whether or not the
>specification explicitly forbids it. This makes any theoretical
>advantages even more remote. Customers seldom care that
>your esoteric optimization is "legal", they just know that
>*your* RNIC is the one that isn't working with everyone else.
>
>So I believe the draft should be clarified to require that
>all segments for untagged message N MUST be submitted to
>the LLP before any segments for untagged message N+1 are
>submitted to the LLP.
>
>Doing so will have no impact on unordered receivers. I doubt
>it will have any impact on any existing senders, and it
>eliminates one corner condition that would require extra
>resources and testing for in-order receivers.
>
>
> >>
> >> Several months ago I raised a query as to whether DDP could/should
> >> mandate that the process of segmenting DDP Messages MUST NOT
> >> result in interleaving of messages. That is *all* segments for DDP
> >> Message N must be submitted to the transmitting LLP: before any
> >> segment of DDP Message N+1 is submitted.
> >>
> >> I believe the current draft could, in a very strained malicious way,
> >> be interpreted to only require:
> >>
> >>     that the segment with the L bit for each message be the last
> >>     segment for that message.
> >>
> >>     that the segments with L bits are themselves submitted in order.
> >>
> >> The only real response on the list, as I recall, was from Mike Krause
> >> who pointed out that such interleaving would not even be noticed
> >> by an out-of-order receiver. And that is completely correct.
> >>
> >> It does have an impact on in-order receivers, and particular on
> >> how many untagged buffers an in-order RNIC has to have looked
> >> up from the host-side receive queues (or worse SRQs).
> >>
> >> I doubt very much whether anyone will ever produce a sender
> >> with a segmentation algorithm that will violate this rule, and I
> >> am one of the strong advocates of allowing senders to use
> >> creative ordering *within* a message to optimize for sending
> >> data that is already in the RNICs cache.
> >>
> >> But it would be handy to have something firm to justify coding
> >> an RNIC based on this assumption.
> >>
> >> Do you feel that the current language already forbids interleaving
> >> of message segments (in terms of the order they are submitted
> >> to the LLP, and hence their LLP sequence numbers)?
> >>
> >> If so, it is probably enough to just have that stated on the
> >> reflector.
> >>
> >> Otherwise, as one of the principal advocates of having the protocols
> >> be friendly to *both* ordered and unordered implementations, would
> >> you support such a change?
> >>
> >> In my evaluation, the drafters are firmly in the "out of order" camp
> >> and hence do not see any need for this clarification.
> >>
> >
>
>
>--
>Caitlin Bestler
>http://asomi.com/
>
>_______________________________________________
>rddp mailing list
>[email protected]
>https://www1.ietf.org/mailman/listinfo/rddp