RE: IPsec and RDDP as a transport -- one-shot STags

The "hole" being discussed occurs anytime DDP uses an STag to place data into a sink buffer in host memory (ie during RDMA Writes or RDMA Read Responses).  
 
As David & Caitlin have explained, with a 'persistent' STag, the ULP may begin using the buffer while the RNIC is overwriting the buffer with new data.   This new data doesn't have to come from a man-in-the-middle, it could be from the peer ULP too.  The problem is one of synchronization between the RNIC and peer ULP:  STag invalidation is the synchronization mechanism available at the RDDP layer. 

We already have 'send & invalidate' where the invalidated STag is contained in the payload of a message from the remote peer.  The RNIC reads the message header, extracts the STag, then invalidates the STag before providing the receive completion to the ULP.  

Caitlin's point is that the local ULP should be in control of the invalidation -- relying on the remote peer for invalidating the STag is too risky in a client-server environment where the server may have to deal with multiple client vendors & implementations.  Her proposal is to link the STag with a specific receive buffer.  The RNIC is to invalidate the STag when it consumes this specific untagged message buffer.  

Both of these proposals create a 'short-lived' or 'one-shot' STag that guarantees the invalidation of the STag before the ULP consumes the tagged buffer.  The remote peer may use the STag in multiple RDMAP messages, but signals that it is finished with the STag by sending an untagged message to the local peer.  This untagged message triggers the RNIC to invalidate the local STag.  

Neither of these proposals, however, deal with the synchronization between the RNIC and the local peer.  It is still possible for multiple RDMAP messages to target the same offset in the same sink buffer before the ULP sees it.

Another option is to restrict STags to a single RDMAP message and have the RNIC automatically invalidate the STag itself.  The RNIC is responsible for managing the STag state after the ULP registers a region or binds a window to the STag.  By default all 'valid' STags are 'one-shot' unless the ULP explicitly specifies that the STag is 'persistent'.  

  *  For 'one-shot' STags, the RNIC invalidates the local STag after it places the last DDP segment for the RDMAP message in host memory.  If another message from the remote peer requests the same STag, that RDMAP message would be rejected.  

  * For 'persistent' STags,  the RNIC does no STag invalidation unless instructed to do so by the ULP (explicitly by the local ULP or through a 'send & invalidate' message from the remote ULP).

With the RNIC in control of the STag invalidation, a sink buffer can only be written to once.   The question if this new mechanism breaks existing ULP usage models.  

--jc