Security Implications of sharing buffers/credits

I've been working through my object models to determine
exactly what sorts of sharing related to untagged messages
can be supported, and the implications as to
performance/implementation/interoperability/security.


The performance/implementation  implications I'll be
keeping to myself, but I thought I'd share the
interoperability/security notes I generated.

There are two critical issues regarding reception of untagged
messages:

-- How the MSN is validated.
-- When a buffer is associated with an untagged message.

These issues will be dealt with on a per-strategy basis.



Strategy #1: Simple Ordered Receive Queue

This is the classic RDMA interface.

Buffers are pre-posted on a per stream basis by the ULP. As
each is posted it is pre-assigned to the next MSN.

Validity of an MSN is determined by whether or not there is
already an associated buffer when a segment with the MSN is
first received.






Strategy #2: Shared Buffers / Implicit Shared Credits
with soft per-stream limits.

This is the Shared Receive Queue strategy.

Buffers are posted to a Shared Buffer Pool (it is not
really a queue). Credits are not tracked on a per-stream
basis.

When a DDP Segment for an MSN is first received, it is
presumed valid. A buffer is allocated from the pool.

The number of buffers allocated  can be checked against a
soft limit. This allows detection of "out-of-control"
streams. However this is a soft limit, because any response
is left to the ULP.

There is an important corollary issue: what the verbs draft
refers to as sequential-ordering versus arrival-ordering.

With sequential-ordering, allocation of a buffer for MSN x
requires allocation of a buffer for all previous MSNs. This
allows a single message to tie up a large number of
buffers. While this will undoubtedly trigger a soft-limit
alarm, those buffers are tied up until the ULP acts.

With arrival-ordering, the arrival of any new MSN can
consume at most one buffer. However, this can lead to a
state where the itnermediate MSNs cannot be received
because all available buffers are being held by later MSNs.
The later messages cannot be delivered, therefore the
buffers cannot be released. With SAcks these messages may
not be retroactively releasable. This is particularly
relevant when the LLP is SCTP because TSN ordering is not
necessarily related to DDP sequencing. The SCTP stack is
free to re-order to optimize chunk bundling. Therefore
having acked a non-deliverable untagged message is a
distinct possibility.

With the availability of the soft-limit ordering, the
maximum damage from an abusive stream can be made
predictable to the extent that the ULP's response time to
an asynchronous error can be made predictable. In many
environments this will require a substantial increase in
the allocation of "spare" resources.

Care must be taken in scoping and sizing the Completion
Queue when this strategy is employed.

- The CQ may be sized to allow for every buffer in the pool
   to be completed to it. This will generally be wasteful unless
   the CQ has the same scope as the pool.
- The CQ may be scoped to a single DDP Stream. Excessive
    completions triggered by the remote peer will be incapable
    of damaging any other DDP Stream.





Strategy #3: Shared Buffers / Explicit Per-stream Credits

This is the Buffer Pool strategy proposed in prior emails.

Buffers posted are placed in a potentially shared Buffer
Pool, with a credit being issued to the specific DDP
stream. Each credit pre-enables (or validates) the next
MSN. Unlike the Simple Ordered Receive Queue model,
the MSN is validated *without* pre-assigning a buffer to it.

The ULP may also grant credits *without* supplying a
buffer, thereby underprovisioning the pool.

When a DDP Segment for an MSN is first received it is
considered valid if the MSN is within the enabled range for
that stream. At that time a buffer must be associated. The
absence of a buffer for a valid MSN is a fault. Recovery
procedures are a local issue, but ultimately must include
termination of the stream.

As with any under-provisioning strategy, there is a chance
of running dry. However unauthorized actions by the remote
peer cannot provoke such a shortage.

This strategy is consistent with bother Arrival Ordering
and Sequential Ordering as discussed above. However, this 
is less of an issue with this strategy. Sequential ordering
can only allocate buffers for messages that are authorized
anyway. Stranded undeliverable messages can only occur if
there is a severe under-allocation of buffers because the
stranded message had a pre-validated MSN anyway.

The existence of hard per-stream limits on the number of
valid MSNs also means that the number of completions is
predicatable on a per stream basis.  This allows safe
pooling of completion queues across multiple DDP Streams
without having to match the  scope of the Buffer Pool.