Re: Error code returns for DDP tagged buffer messages

On Apr 27, 2004, at 6:25 PM, Shah, Hemal wrote:

> <hvs>
> I am not assuming a fixed queue size. At any give point in time, a 
> queue
> can have QLEN size which can change dynamically based on the
> modifications to the queue size. For the errors, the queue size seen at
> the message arrival time should be used.
> </hvs>

You are assuming that there is a queue that is specific to the stream,
and that it's size is easily determined.

That is actually an excellent design. Which is why I was opposed
to the definition of SRQs in the RDMAC iWarp verbs.

If the local interface implements an RDMAC style SRQ it is
obligated to allocate over the nominal per-stream limits if
there are buffers available. Such a constraint all but forces
a more complex SRQ design that is not anything as simple
as an adjustable ring buffer.

With that implementation you *might* be able to check the
per-stream quota whenever a buffer is not available. If the
new MSN minus the stream's base MSN would exceed the
stream's quota (that is, if the buffer had been accepted it
would have raised the per-stream over-limit alarm) then
the implementation could return the "Invalid MSN" error.
But if it is within the limit, meaning the remote peer is
probably not the party responsible for buffer exhaustion,
an "No Available Buffer" error is far more accurate.

However, in order to *reliably* distinguish between the
two the per-stream quota must be available at the same
layer that generates the error. It is entirely plausible
to have an implementation where allocations are
made from the SRQ on the RNIC, but the alarms
are detected by the driver.

My preference for deterministic teardown of streams
that exceed their quota has been previously noted.
But given that the consensus favors allowing asynchronous
detection of this condition (with only the caveat that the
application not be forced into that asynchronous loop)
we have to recognize that valid SRQ implementations
may not always be able to distinguish between the two
errors.

I do not believe we need to disrupt implementations
by running around and renumbering the error conditions.
But we should note that when multiple error returns are
all true then the implementation is free to issue any of
them.  There is no "correct" choice among the error
codes that are true.

The only exception is implied by the DDP ordering rules.
If there is a DDP layer error, then it MUST take precedence
over an RDMAP layer error. After all, evaluating the RDMAP
packet before validating the DDP packet would have been
speculative. Technically, if there is a DDP layer error then
the RDMAP layer should never have gotten the packet
for handling.

>
> Also, nothing in the draft makes these distinctions between the errors.
> In each case the distinction is not particularly useful and I think we
> should consolidate to one error for STag not valid on the stream and 
> one
> error for no buffer for the MSN.
>
> <hvs>
> I believe that the distinction can be useful for the debugging.
> </hvs>

I agree.

When the receiver knows that the MSN is invalid it SHOULD provide
that diagnostic information. This should be simple with RQs and with
many SRQ designs. And there is a very big difference between "you
sent too many messages" and "I ran out of buffers".



--
Caitlin Bestler
http://asomi.com/