comments on draft-kashyap-ipoib-connected-mode-02.txt

In the last IETF61 IPoIB meeting I made several comments on the
connected mode draft. I'm sending them to the list for a general
discussion. (Yes I saw some disucssion on the connected mode
draft already. I'll try to catch up with the thread after this
mail.)

1. The draft makes a distinction between IPoIB-CM interfaces
and IPoIB-UD interfaces, and portrays IPoIB-UC or IPoIB-RC as
separate subnets superimposed on top of an IPoIB-UD subnet.

For the above to work, due to a lack of multicast support, a fully
connected network by itself can't meet the requirement of an IP
link unless multicast is fully emulated through the use of
multiple unicasts. The latter is complex and cumbersome.

A much simpler model, which I think was presented in earlier
drafts, is to fold the use of IB connections fully into a
regular IPoIB-UD subnet, allowing any two IPoIB nodes to
optionally negotiate the use of IB connection between themselves.

This much simplified model is not without its drawback. Some
nice IP link attributes are no longer unique within a link.
E.g., the link MTU now becomes per-node-pair MTU. Moreover,
the MTU size for multicast will be different from the MTU size
for unicast if IB connections are used. IB UC/RC may exhibit
different RAS, flow control, QoS or other link characteristics
than UD. But I consider these problems a reasonable price to
pay for a seamless support of UC/RC mode in an IPoIB link
defined by UD.

2. The negotiation of the per-connection MTU seems more
complicated than necessary. I think all is needed is for a
node to advertise its own "receive MTU". That is, the MTU
size its peer should never go over when sending packets
to the local interface. Yes this may break the traditional
concept of "symmetric" MTUs. But we're already breaking the
notion of per-link MTU, requring a lot of changes in the host
stack anyway. This additonal breakage doesn't seem much.

I haven't verified if this asymmetric MTU matches well with
IBA connections though.

3. Regarding allowing multiple IB connections between a node
pair, since given an IP address there is only one link-address
for it implying one QPN, hence one service-ID, if a single
service-ID can be used to create multiple IB connections
then this can happen transparently. Otherwise we've got a
problem.

Jerry