Re: comments on draft-kashyap-ipoib-connected-mode-02.txt

On Fri, 19 Nov 2004, H.K. Jerry Chu wrote:

> <snip>
> 
> >> A much simpler model, which I think was presented in earlier
> >> drafts, is to fold the use of IB connections fully into a
> >> regular IPoIB-UD subnet, allowing any two IPoIB nodes to
> >> optionally negotiate the use of IB connection between themselves.
> >
> >The difference in the earlier draft and this one is that
> >I modified the requirement on the UD QP. That is, it need not be that
> >IPoIB-CM and IPoIB-UD share a QP but that any UD QP will do for IPoIB-CM. 
> >In effect an implementation can still share the UD QP.
> >
> >The only issue is whether the same IP subnet can contain pure 
> >IPoIB-UD mixed in with IPoIB-CM nodes or, all nodes must be of the same type.
> >	- all IPoIB-UD
> >or
> >	- all IPoIB-RC
> >
> >or 	-- all IPoIB-UC
> >
> >I beleive all of the same type is a good option to choose.
> 
> I don't see a clear benefit for this restriction. E.g., even in all IPoIB-RC or
> IPoIB-UC, the nice per-link MTU property is no longer there due to multicast
> supported through UD. Also this restriction will require those implementations
> that don't support IPoIB over UC or RC to form a different subnet in order to
> talk IPoIB, hence forcing the adminstrator to maintain at least two IP subnets
> with one fully contained within another. I don't see why this is needed.

ok..let me posit what seems to be the summary to me (looking for more comments
from WG members here). I'm more or less reverting to the earlier version of 
the draft.

In an IPoIB subnet:

	- Every interface MUST support IPoIB-UD

	- An interface MAY optionally also support IPoIB-CM (one or both)
		i.e. removing the mutually exclusive restriction on rc/uc
		Note: IIRC, the same serviceID can be used for both RC/UC.
		If not then they have to stay mutually exclusive.

	- Interoperability is maintained by all nodes supporting IPoIB-UD. 
	  Any two interfaces that do not have a connection mode in common will
	  fall back to IPoIB-UD. 

	- The support of any particular IB mode is indicated by the flags
 	  in the link layer address. Note: IPoIB-UD is always supported and
	  hence there are no flags to indicate UD support.

	- An interface completes the IPoIB-UD address resolution and then
          optionally MAY set up RC/UC connections based on the local support
	  and received flags.

	- A pure IPoIB-UD implementation ignores the RC/UC flags in link layer
	  address in received packets. It zeroes them on transmit.

	- Every implementation MUST accept all unicast transmissions received
	  over any of the IPoIB modes it supports. Multicast/Broadcast by 
	  their nature will be transmitted and received over the IPoIB-UD only.

	***This implies that an interface MAY transmit/receive a packet
		over any of RC or UC or UD depending on the modes supported
	        between the peer IP and itself.***
		
	- It is an implementation's decision to connect or retry a connect on
  	  failure on the CM modes. This decision is independently made per
	  transmission or reception of a connection request.

	- An implementation MAY make multiple connections to a peer. This
          is a local decision. So is the decision of the peer to refuse
          such a connection. 
	
	The serviceID, link setup, the link address flags, MTU negotiation etc.
	are covered in the draft. 

	- MTU -- we need to discuss more as below.

> 
> >
> >> 
> >> This much simplified model is not without its drawback. Some
> >> nice IP link attributes are no longer unique within a link.
> >> E.g., the link MTU now becomes per-node-pair MTU. Moreover,
> >> the MTU size for multicast will be different from the MTU size
> >> for unicast if IB connections are used. IB UC/RC may exhibit
> >> different RAS, flow control, QoS or other link characteristics
> >> than UD. But I consider these problems a reasonable price to
> >> pay for a seamless support of UC/RC mode in an IPoIB link
> >> defined by UD.
> >> 
> >> 2. The negotiation of the per-connection MTU seems more
> >> complicated than necessary. I think all is needed is for a
> >> node to advertise its own "receive MTU". That is, the MTU
> >> size its peer should never go over when sending packets
> >> to the local interface. Yes this may break the traditional
> >> concept of "symmetric" MTUs. But we're already breaking the
> >> notion of per-link MTU, requring a lot of changes in the host
> >> stack anyway. This additonal breakage doesn't seem much.
> >> 
> >> I haven't verified if this asymmetric MTU matches well with
> >> IBA connections though.
> >
> >How about:
> >
> >The MTU I would think is exchanged at the IB level during the
> >IPoIB-CM connection setup. The IP layer at both ends keeps a per connection
> >MTU if the implementation permits it. At the link layer the connection will
> >not send messages larger than that requested by the peer.
> 
> Not quite understand the above. I'm suggesting to simplify the MTU negotiation
> at the IPoIB-CM connection setup time by each side advertising the "receive
> MTU" it can take. The peer must not send more than that size in each post_send().
> E.g., if node A advertizes 32KB as its receive MTU and node B advertizes 64KB
> as its receive MTU, node B must not send any IP pkt through IPoIB-CM to node
> A that is larger than 32KB. Node A is free to send IP pkts of up to 64KB in
> size to node B. (But if node A decides to restrict its outbound MTU to 32KB,
> that's fine too. Node B doesn't need to know about it.)
> 
> I'm not sure what you mean by the last two sentences above. MTU value must
> be made known to the IP layer so that latter won't send anything larger
> than that. Otherwise the pkt will get dropped by the IB layer (unless the
> latter performs SAR, which is a bad idea).

A model for an implementation can be that the IP MTU is set large but the 
actual packet size or segment size is determined by an internal mechanism 
that determines the MTU for the relevant connection/UD path. An implementation
is also allowed to just drop down to the IPoIB-UD MTU and use that always.

The interface MTUs at the peers need not be the same at IP or IB layers.

I agree with the concept of just exchanging the max receive MTU at the IB 
connection setup. 

Vivek


> 
> Jerry
> 
> >
> >
> >> 
> >> 3. Regarding allowing multiple IB connections between a node
> >> pair, since given an IP address there is only one link-address
> >> for it implying one QPN, hence one service-ID, if a single
> >> service-ID can be used to create multiple IB connections
> >> then this can happen transparently. Otherwise we've got a
> >> problem.
> >> 
> >> Jerry
> >> 
> >> 
> >> _______________________________________________
> >> IPoverIB mailing list
> >> [email protected]
> >> https://www1.ietf.org/mailman/listinfo/ipoverib
> >> 
> >> 
> >
> >
> >
> 
> 
>