Re: [Ips] I-D ACTION:draft-ietf-ips-iser-00.txt

Caitlin,

>From section 5.2 of draft-ietf-rddp-rdmap-02.txt on "RDMAP Interactions 
with the ULP", it states that:

"For RDMA Read Operations, the following are the interactions
between the RDMAP Layer and the ULP:

*  At the Data Sink:

    *   The ULP passes to the RDMAP Layer the following:

        *   ULP Message Length

        *   Data Source STag

        *   Data Sink STag

        *   Data Source Tagged Offset

        *   Data Sink Tagged Offset"

The iSER layer, as the ULP to RDMAP, passes the local STag as the Data 
Sink STag to the RDMAP layer, in accordance with the interface 
interactions as specified in the RDMAP draft.  The language in the iSER 
draft was written to convey this information.  Verbs is cited as an 
informational reference only in the iSER draft.  Nowhere in the iSER draft 
was Verbs ever mentioned.  The term "data descriptor" does not appear 
anywhere in the RDMAP draft.  If you are proposing a change to the RDMAP 
draft to provide this extra flexibility in the interface interactions for 
the RDMA Read operation that you have outlined in your note, perhaps the 
question should be directed to the authors of the RDMAP draft instead.

Mike
To:     Mike Ko <[email protected]>
cc:     [email protected], [email protected] 
Subject:        Re: [Ips] I-D ACTION:draft-ietf-ips-iser-00.txt




On Sep 15, 2004, at 8:00 PM, Mike Ko wrote:

> Section 7.3.6 clause 1 states that:
>
> 1.  It MUST ensure a valid local STag for the I/O Buffer and a valid
>     local mapping that associates the Initiator Task Tag (ITT) to
>     the local STag.  This may involve allocating a valid local STag
>     and establishing a local mapping.
>
> The iSER layer has already allocated a valid local STag for the I/O
> buffer
> which will be used as the data sink buffer for the RDMA Read operation
> based on the DataDescriptorOut which qualifies the Get_Data invocation
> between the iSER layer and the iSCSI layer.  From a layering viewpoint,
> DataDescriptorOut is not visible to the RDMA layer.  Can you clarify
> how
> the RDMA layer can "flexibly map a target Data Descriptor to a local
> STag"
> without violating the layering purity?

The RDMAP layer itself receives a "data descriptor" that specifies the
Data Sink for the RDMA Read operation.  If the local interface has
been designed to be transport neutral this "data descriptor" may
be a *local* descriptor (i.e., subject to the same rules as a the
local target of a receive operation) rather than a previously
advertised MW STag.

The RDMAC verbs call for this buffer to be specified just as
the target of a tagged message would be specified. Using
the RDMAC verbs the iSER implementation would indeed
create an MR STag, pass that MR STag to the RDMAP layer,
and the RDMAP layer would use that STag in the wire
packet.

But the RDMAP layer MAY accept specifications of the
local target in other formats (especially to promote
transport neutrality) and then generate an STag
on its own from a different pool.

Applications using DAT or IT-API will typically end
up submitting a local buffer specification for an
RDMA Read operation that has a single element
and uses an MR STag (LMR Context).  They
cannot accept an MW STag (RMR Context) because
InfiniBand requires an L-Key for an RDMA Read
local buffer, not an R-Key.

With the RDMAC verbs you can use an MR STag
or an MW STag for an RDMA Read -- but doing so
may be undesirable for security reasons. Instead
an implementation could choose to use the RDMA
Read Data Sink STag as the equivalent of an
untagged messages MSN.  The wire protocol
is fully compatible with treating an RDMA Read
Reply virtually identically as an RDMA Write
or as virtually identical to an RDMA Send.

In the latter case the "Data Sink STag' on the
wire is *not* any value that was submitted
by the iSER layer -- even though the data
does end up in the buffer that the isER
layer specified.

The RDMAC verbs are explicitly NOT an IETF
document,, therefore iSER cannot make
assumptions on the local interface based
upon them.