Re: A Couple of IPoIB Questions

Mike the format is really off in the last mail from you - making it difficult
to follow.


Other than that let us discuss in the context of the draft. The draft is
built upon the following:

1. IPoIB-RC and IPoIB-UC are optional.
2. IPoIB connected mode depends on a UD QP for address resolution and multicast.

As far as I know, there has been an agreement since the earliest connected mode
draft I posted.


I'd like the WG to give input on the following issues:

3. Where does the UD QP come from?  Choose one of:

a. It is a UD QP that is associated with the interface at startup.

b. It is a UD QP that is shared with IPoIB-UD.


3a is more generic. It can be considered to include the case 3b.  The original
proposal was limited to 3b.



4. Link characteristics

The broadcast domain for IPoIB-RC/UC is determined exactly as the
IPoIB-UD case i.e. through the broadcast-GID. A UD as per 3 is used in this
step.

Do all interfaces in the IPoIB-conneced mode(CM) have the same link
characteristics? i.e.

a. all are either IPoIB-RC or IPoIB-UC.

	-- There is also a UD QP associated. The UD QP will be either 3a or 3b
	   based on WG concensus.

	-- All unicast transmission is on the IPoIB mode i.e. RC or UC.

b. all are IPoIB-UD. Additionally they can be one of IPoIB-RC or IPoIB-UC
or both.

	-- The presence of the flags indicate the type of communication possible.
	-- The decision of communicating using a specific mode is determined by
	   the supported modes and the local policy. Note that incompatible
	   policies imply that the fallback is communication over UD.
	-- fallback mode of communication is UD


4b adds a lot of flexibility at the expense of a simple decision. 4a. by
contrast is straightforward.


5. MTU negotiation

	In the private data field of the CM message the desired MTU is
	included.

	It was suggested during the IPoIB meeting at IETF that it need not be
	symmetric. That is a good idea. Thus each peer declares the max MTU it
	prefers


	REQ: <my desired MTU>
	REP: <my desired MTU>
	RTU:


6. Multiple connections for the same IP address

	Local decision. Note that the peer might choose to not honour multiple
	connections.



Vivek





On Wed, 17 Nov 2004, Michael Krause wrote:

> At 11:38 PM 11/16/2004, Vivek Kashyap wrote:
>
>
>
> >Hi,  I have a couple of questions relative to IPoIB:  1.
> >draft-ietf-ipoib-ip-over-infiniband-07.txt states: "Every IPoIB interface
> >MUST "FullMember" join the IB multicast group defined by the
> >broadcast-GID."  Isn't the broadcast group for IPv4 ? When the IPoIB
> >interface is IPv6 only, does this group still need be joined ?  If not,
> >where do the parameters for any IPv6 groups come from ? I am presuming
> >that this group needs to be joined in  the IPv6 only case. I just want to
> >be sure.
> ><VK> Yes, the broadcast-GID is at the InfiniBand layer and MUST be joined
> >whether you are running at v4 or v6 layer. <VK>  2. ALso, what is the
> >latest status of the Vivek's connected mode draft ? Will it be moving
> >forward ?  <VK> I'll be submitting it as
> >draft-ietf-ipoib-connected-mode-00.txt by the end of the month. There were
> >some interesting suggestions that were made during the IETF WG meeting.
> >Two of the suggestions of consequence are given below. The others we can
> >discuss when the minutes are published (they include some additional
> >requests on clarification on the transmission draft too).  a. The current
> >draft makes the various modes mutually exclusive i.e. RC, UC and UD are
> >not allowed simultaneously in the same IP subnet. The thought is that it
> >is a link characteristic and hence different per connection mode. It was
> >suggested that one be allowed to mix up RC/UC. This goes back to the
> >original suggestion in the first draft which was:  IPoIB-UD must always be
> >supported. Additionally, the interface can also support either both of RC
> >and UC, or one of them. Or neither of them.
> >
> >UD MUST always be supported.
> >
> ><VK> That is and has always been the requirement right from the first
> >draft. <VK>
> >
> >I personally don't care whether one does RC or UC but I don't think both
> >are required as a MAY option. The advantage of RC is the send credit
> >algorithm. The advantage of UC is the lack of ACK packets. ACK is noise in
> >the fabric while send credits provide a simple method to maintain
> >bandwidth / injection control on a per flow basis.
> >
> >I see no problems with supporting both UD and *C on the same subnet; it is
> >rather foolish to attempt to mandate these be on separate subnets.b
> ><VK> As per the connected-mode draft the UD mechanism is *always*
> >required; address resolutoin depends on it.
> >
> >The only point of discussion is whether all nodes must support the same
> >link characteristics in the subnet i.e. all are RC (and UD), or all or UC
> >(and UD), or all are UD only.
>
> Obviously I would oppose such a solution as it creates artificial
> constraints with little benefit.
>
> >The alternative is to allow all the nodes to be mixed up with some nodes
> >being RC/UD, others UC/UD and a third set UD only and yet others probably
> >supporting all. within the same IP subnet. [Can the same serviceID be used
> >by both RC and UC ?]
> >
> >The third alternative is to associating UD only or UD + one of RC or UC on
> >the same interface. In such a case if mismatched/unsupported connected
> >modes are supported by two nodes then the fall back to UD. This option is
> >not too different from UD QP + RC or UC mechanism.
>
> KISS:
>
> - UD universal
> - *C opportunistic
>          - Local management issue to control what is sent on the *C
> interface.  No need to specify
>          - Advertise whether one or more ports are supported by UD or *C
>          - Advertise whether one or more QP are supported by UD or *C
>          - Let local management determine policy for what services are
> mapped where - no need to specify
>
> This is both an interoperable approach and simple to implement.  There may
> be some desire to add a policy interface to state preference for specific
> types of traffic over a given QP.  I would not oppose this but would view
> this as a separate draft once the basics are worked out.
>
>
>
> ><VK>
> >b. Another suggestion was to allow multiple connected mode links (i.e. at
> >IB UC/RC level) between peers.  One thought can be 'yes, but user beware':
> >The IB connections are made using the service ID that is derived from the
> >QPN as described in the draft. If a second attempt succeeds then there are
> >two links. It is up to the implementation to either allow or disallow
> >multiple links.
> >
> >Again, this has been suggested in the past (though most who were involved
> >in the original discussions years gone by are likely gone since much of
> >this discussion occurred before the IETF workgroup was established).
> >
> ><VK> I'm one of the vestiges of those early times along with you and a few
> >others...so we have hope :). <VK>
> >
> >There is obvious benefit to supporting multiple RC per endnode pair. I do
> >not see any technical reason to oppose nor any issue from an
> >interoperability perspective. There is no reason for a "user beware".
> >
> ><VK> It is not opposed. The 'user beware' is only underscoring that the
> >the peer interface might not support multiple links- it might enforce a
> >limited number of connections (maybe only one) between a pair of GIDs.
> >Similarly, an implementation not wanting to support multiple links MUST
> >take steps to deny multiple requests.
>
> *C requires CM to operate thus it is a local issue whether additional CM
> operations are accepted or not.  A given requester node may issue N and a
> given responder may state 0-N as an implementation may limit the number of
> *C available for IP traffic.
>
>
> ><VK>
> >
> >The work is rather straight to do and implement and the benefit to
> >customers, is again, rather obvious when one considers what the IB fabric
> >offers and how connections can be enable flows through multipath as well
> >as transparent fail-over, flow scheduling, mapping of DiffServ to
> >different arbitration / paths, etc.
> >
> ><VK> In addition Large MTU and APM are two of the main reasons why I've
> >been proposing IPoIB-connected mode for so long. In terms of IPoIB itself,
> >except for the Large MTU, the parameters are hidden from it.<VK>
>
> Mike

__

Vivek Kashyap
Linux Technology Center, IBM