Re: comments on draft-kashyap-ipoib-connected-mode-02.txt

At 10:20 AM 11/19/2004, H.K. Jerry Chu wrote:
><snip>
>
> >> A much simpler model, which I think was presented in earlier
> >> drafts, is to fold the use of IB connections fully into a
> >> regular IPoIB-UD subnet, allowing any two IPoIB nodes to
> >> optionally negotiate the use of IB connection between themselves.
> >
> >The difference in the earlier draft and this one is that
> >I modified the requirement on the UD QP. That is, it need not be that
> >IPoIB-CM and IPoIB-UD share a QP but that any UD QP will do for IPoIB-CM.
> >In effect an implementation can still share the UD QP.
> >
> >The only issue is whether the same IP subnet can contain pure
> >IPoIB-UD mixed in with IPoIB-CM nodes or, all nodes must be of the same 
> type.
> >       - all IPoIB-UD
> >or
> >       - all IPoIB-RC
> >
> >or     -- all IPoIB-UC
> >
> >I beleive all of the same type is a good option to choose.
>
>I don't see a clear benefit for this restriction. E.g., even in all 
>IPoIB-RC or
>IPoIB-UC, the nice per-link MTU property is no longer there due to multicast
>supported through UD. Also this restriction will require those implementations
>that don't support IPoIB over UC or RC to form a different subnet in order to
>talk IPoIB, hence forcing the adminstrator to maintain at least two IP subnets
>with one fully contained within another. I don't see why this is needed.

I maintain that *C and UD can co-exist in the same IP subnet and there is 
no reason to restrict this.  Endnode pairs will establish their 
communication paths and take the appropriate QP to reach a given 
destination.  This is all a local issue in the end sans the all unicast 
debate in an earlier string.


> >
> >>
> >> This much simplified model is not without its drawback. Some
> >> nice IP link attributes are no longer unique within a link.
> >> E.g., the link MTU now becomes per-node-pair MTU. Moreover,
> >> the MTU size for multicast will be different from the MTU size
> >> for unicast if IB connections are used. IB UC/RC may exhibit
> >> different RAS, flow control, QoS or other link characteristics
> >> than UD. But I consider these problems a reasonable price to
> >> pay for a seamless support of UC/RC mode in an IPoIB link
> >> defined by UD.
> >>
> >> 2. The negotiation of the per-connection MTU seems more
> >> complicated than necessary. I think all is needed is for a
> >> node to advertise its own "receive MTU". That is, the MTU
> >> size its peer should never go over when sending packets
> >> to the local interface. Yes this may break the traditional
> >> concept of "symmetric" MTUs. But we're already breaking the
> >> notion of per-link MTU, requring a lot of changes in the host
> >> stack anyway. This additonal breakage doesn't seem much.
> >>
> >> I haven't verified if this asymmetric MTU matches well with
> >> IBA connections though.
> >
> >How about:
> >
> >The MTU I would think is exchanged at the IB level during the
> >IPoIB-CM connection setup. The IP layer at both ends keeps a per connection
> >MTU if the implementation permits it. At the link layer the connection will
> >not send messages larger than that requested by the peer.
>
>Not quite understand the above. I'm suggesting to simplify the MTU negotiation
>at the IPoIB-CM connection setup time by each side advertising the "receive
>MTU" it can take. The peer must not send more than that size in each 
>post_send().
>E.g., if node A advertizes 32KB as its receive MTU and node B advertizes 64KB
>as its receive MTU, node B must not send any IP pkt through IPoIB-CM to node
>A that is larger than 32KB. Node A is free to send IP pkts of up to 64KB in
>size to node B. (But if node A decides to restrict its outbound MTU to 32KB,
>that's fine too. Node B doesn't need to know about it.)
>
>I'm not sure what you mean by the last two sentences above. MTU value must
>be made known to the IP layer so that latter won't send anything larger
>than that. Otherwise the pkt will get dropped by the IB layer (unless the
>latter performs SAR, which is a bad idea).

One might argue that *C is focused on an equivalence to TSO (large send) 
thus the logical MTU is not required.  One might argue that the logical MTU 
represents an asymmetric maximum receive buffer that will be posted thus 
messages must be sent that do not exceed this maximum.  One might argue 
that having a single buffer size independent of the *C / UD being used 
maximizes KISS.  I'm open to exploring these options but do not believe all 
must be supported.

> >> 3. Regarding allowing multiple IB connections between a node
> >> pair, since given an IP address there is only one link-address
> >> for it implying one QPN, hence one service-ID, if a single
> >> service-ID can be used to create multiple IB connections
> >> then this can happen transparently. Otherwise we've got a
> >> problem.

A service ID can be used to establish multiple connections thus the 
creation process should be left as an implementation detail in terms of how 
many, etc. as I've noted in a previous response.  The local endnodes will 
determine what is allowed per endnode pair and there are no 
interoperability issues that arise as a result.

Mike

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib