Re: A Couple of IPoIB Questions

On Thu, 18 Nov 2004, Michael Krause wrote:

> At 11:33 AM 11/18/2004, Vivek Kashyap wrote:
> >On Thu, 18 Nov 2004, Michael Krause wrote:
> >
> > > At 10:46 PM 11/17/2004, Vivek Kashyap wrote:
> > > >Mike the format is really off in the last mail from you - making it 
> > difficult
> > > >to follow.
> > > >
> > > >
> > > >Other than that let us discuss in the context of the draft. The draft is
> > > >built upon the following:
> > > >
> > > >1. IPoIB-RC and IPoIB-UC are optional.
> > >
> > > I would prefer only one be used - either RC or UC.  I've provided some
> > > logic for either one as a preference but don't see a reason to have
> > > both.  Both just leads to options which leads to interoperability problems.
> >
> >ok.
> >See section 3.1 of the draft draft-kashyap-ipoib-connected-mode-02.txt.
> >It states that the RC and UC are mutually exclusive flags.
> 
> My preference is to only support one of the two in a spec not to have flags 
> to indicate what is implemented.  The benefits of connected mode operation 
> should be done with only one form of communication not two.

A given subnet will support only one of the two. Not both simultaneously. The 
flag only indicates which type it is. RC and UC are both useful to different 
people and implementations so both are allowed. I suggest that both not be 
allowed in the same IPoIB subnet though.

> 
> 
> > >
> > > >2. IPoIB connected mode depends on a UD QP for address resolution and
> > > >multicast.
> > > >
> > > >As far as I know, there has been an agreement since the earliest 
> > connected
> > > >mode
> > > >draft I posted.
> > > >
> > > >
> > > >I'd like the WG to give input on the following issues:
> > > >
> > > >3. Where does the UD QP come from?  Choose one of:
> > > >
> > > >a. It is a UD QP that is associated with the interface at startup.
> > > >
> > > >b. It is a UD QP that is shared with IPoIB-UD.
> > > >
> > > >
> > > >3a is more generic. It can be considered to include the case 3b.  The 
> > original
> > > >proposal was limited to 3b.
> > >
> > >  From an implementation point of view, all of this will be hidden within
> > > the driver below IP.  As such, the driver will maintain the
> > > associations.  Currently, each driver "instance" (may be multiple per IB
> > > port) will have at least 1 UD QP.  Given the existing protocol already
> > > defines how to share this QP with other nodes, why not just re-use it and
> > > avoid doing more work?  The driver can then map on a per endnode pair 
> > basis
> > > what *C QP go with what the UD QP and the spec remains largely silent on
> > > how this is accomplished.
> >
> >The draft at present states that 'IPoIB-CM implementation MAY use the same UD
> >QP as used by IPoIB-UD...'. See section 3.0. I believe it covers what you
> >are stating.
> >
> > > >4. Link characteristics
> > > >
> > > >The broadcast domain for IPoIB-RC/UC is determined exactly as the
> > > >IPoIB-UD case i.e. through the broadcast-GID. A UD as per 3 is used in 
> > this
> > > >step.
> > > >
> > > >Do all interfaces in the IPoIB-conneced mode(CM) have the same link
> > > >characteristics? i.e.
> > >
> > >  From an implementation perspective, this is generally simplest.
> > >
> > > >a. all are either IPoIB-RC or IPoIB-UC.
> > >
> > > Preference is only 1 to be defined.
> > >
> > >
> > > >         -- There is also a UD QP associated. The UD QP will be either 3a
> > > > or 3b
> > > >            based on WG concensus.
> > > >
> > > >         -- All unicast transmission is on the IPoIB mode i.e. RC or UC.
> > >
> > > For a given endnode pair, the policy of which QP is used for a given
> > > unicast IP datagram is really a local issue.  I see some merit in the
> >
> >Not if an implementation chooses to only receive unicast on the CM modes in
> >  an IPoIB-CM subnet. I think the WG must either mandate that between two
> >IP address all unicast communication can be over either UD or the 
> >supported CM,
> >or state that all unicast communication must be over IPoIB-CM. Hence my
> >attempt at a detailed discussion on these issues.
> >
> >Issues such as in order delivery need to be considered: e.g. if RC and UD are
> >used to mix up the traffic, say of TCP segments of the same connection, they
> >may no longer be received in order.
> 
> If a designer is stupid, they may do this. However, one would expect some 
> intelligence here and one may prefer to have specific data flows or 
> DiffServ code points or whatever used to determine which connection or 
> which UD QP and that one would again apply an intelligent and predictable 
> algorithm such that mix-n-match for a given TCP connection does not 
> occur.  Given multiple *C QP can be supported, it is not tenable to state 
> that all unicast must go over a given QP or that no unicast can occur on a 
> UD QP.
> 

You mised my point which was that the specification cannot be silent on this 
and say it is a local issue. That can lead to interoperability failure. The 
specification must support or disallow  unicast communication over UD QP in an 
IPoIB-CM.  

You prefer that such communication be supported. That works. Any other thoughts?

> 
> > > attempt to bifurcate this to multicast / broadcast to the UD QP and 
> > unicast
> > > to the *C QP.  However, if the datagram fits in the PMTU of the UD QP, 
> > then
> > > either could be used.  The driver would work either case.  Please keep in
> > > mind that multiple *C QP can be used and their usage needs to be a local
> > > issue and not defined within the spec.
> > >
> > > >b. all are IPoIB-UD. Additionally they can be one of IPoIB-RC or IPoIB-UC
> > > >or both.
> > > >
> > > >         -- The presence of the flags indicate the type of communication
> > > > possible.
> > > >         -- The decision of communicating using a specific mode is
> > > > determined by
> > > >            the supported modes and the local policy. Note that 
> > incompatible
> > > >            policies imply that the fallback is communication over UD.
> > > >         -- fallback mode of communication is UD
> > > >
> > > >
> > > >4b adds a lot of flexibility at the expense of a simple decision. 4a. by
> > > >contrast is straightforward.
> > > >
> > > >
> > > >5. MTU negotiation
> > > >
> > > >         In the private data field of the CM message the desired MTU is
> > > >         included.
> > > >
> > > >         It was suggested during the IPoIB meeting at IETF that it 
> > need not be
> > > >         symmetric. That is a good idea. Thus each peer declares the max
> > > > MTU it
> > > >         prefers
> > > >
> > > >
> > > >         REQ: <my desired MTU>
> > > >         REP: <my desired MTU>
> > > >         RTU:
> > >
> > > Rephrase this as maximum logical MTU to avoid confusion with the IB link
> >
> >It is covered in section 5.1 of the draft.
> >
> > > MTU.  If you start down this path, then you may need to also consider an
> > > exchange of what range of DiffServ code points to use as well.  Not clear
> > > that anyone needs to deal with any latency or bandwidth guarantees but the
> > > "camel's nose is starting to enter the tent" as the saying goes.
> >
> >The camel comes along if  Diffserv  etc. as listed above are
> >included. Hence they are not in the draft.
> >
> > >
> > >
> > > >6. Multiple connections for the same IP address
> > > >
> > > >         Local decision. Note that the peer might choose to not honour
> > > > multiple
> > > >         connections.
> > >
> > > Agreed.
> > >
> > > Mike
> > >
> > >
> > >
> > >
> > > >Vivek
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >On Wed, 17 Nov 2004, Michael Krause wrote:
> > > >
> > > > > At 11:38 PM 11/16/2004, Vivek Kashyap wrote:
> > > > >
> > > > >
> > > > >
> > > > > >Hi,  I have a couple of questions relative to IPoIB:  1.
> > > > > >draft-ietf-ipoib-ip-over-infiniband-07.txt states: "Every IPoIB 
> > interface
> > > > > >MUST "FullMember" join the IB multicast group defined by the
> > > > > >broadcast-GID."  Isn't the broadcast group for IPv4 ? When the IPoIB
> > > > > >interface is IPv6 only, does this group still need be joined ?  If 
> > not,
> > > > > >where do the parameters for any IPv6 groups come from ? I am presuming
> > > > > >that this group needs to be joined in  the IPv6 only case. I just 
> > want to
> > > > > >be sure.
> > > > > ><VK> Yes, the broadcast-GID is at the InfiniBand layer and MUST be 
> > joined
> > > > > >whether you are running at v4 or v6 layer. <VK>  2. ALso, what is the
> > > > > >latest status of the Vivek's connected mode draft ? Will it be moving
> > > > > >forward ?  <VK> I'll be submitting it as
> > > > > >draft-ietf-ipoib-connected-mode-00.txt by the end of the month. 
> > There were
> > > > > >some interesting suggestions that were made during the IETF WG 
> > meeting.
> > > > > >Two of the suggestions of consequence are given below. The others 
> > we can
> > > > > >discuss when the minutes are published (they include some additional
> > > > > >requests on clarification on the transmission draft too).  a. The 
> > current
> > > > > >draft makes the various modes mutually exclusive i.e. RC, UC and 
> > UD are
> > > > > >not allowed simultaneously in the same IP subnet. The thought is 
> > that it
> > > > > >is a link characteristic and hence different per connection mode. 
> > It was
> > > > > >suggested that one be allowed to mix up RC/UC. This goes back to the
> > > > > >original suggestion in the first draft which was:  IPoIB-UD must 
> > always be
> > > > > >supported. Additionally, the interface can also support either 
> > both of RC
> > > > > >and UC, or one of them. Or neither of them.
> > > > > >
> > > > > >UD MUST always be supported.
> > > > > >
> > > > > ><VK> That is and has always been the requirement right from the first
> > > > > >draft. <VK>
> > > > > >
> > > > > >I personally don't care whether one does RC or UC but I don't 
> > think both
> > > > > >are required as a MAY option. The advantage of RC is the send credit
> > > > > >algorithm. The advantage of UC is the lack of ACK packets. ACK is 
> > noise in
> > > > > >the fabric while send credits provide a simple method to maintain
> > > > > >bandwidth / injection control on a per flow basis.
> > > > > >
> > > > > >I see no problems with supporting both UD and *C on the same 
> > subnet; it is
> > > > > >rather foolish to attempt to mandate these be on separate subnets.b
> > > > > ><VK> As per the connected-mode draft the UD mechanism is *always*
> > > > > >required; address resolutoin depends on it.
> > > > > >
> > > > > >The only point of discussion is whether all nodes must support the 
> > same
> > > > > >link characteristics in the subnet i.e. all are RC (and UD), or 
> > all or UC
> > > > > >(and UD), or all are UD only.
> > > > >
> > > > > Obviously I would oppose such a solution as it creates artificial
> > > > > constraints with little benefit.
> > > > >
> > > > > >The alternative is to allow all the nodes to be mixed up with some 
> > nodes
> > > > > >being RC/UD, others UC/UD and a third set UD only and yet others 
> > probably
> > > > > >supporting all. within the same IP subnet. [Can the same serviceID 
> > be used
> > > > > >by both RC and UC ?]
> > > > > >
> > > > > >The third alternative is to associating UD only or UD + one of RC 
> > or UC on
> > > > > >the same interface. In such a case if mismatched/unsupported connected
> > > > > >modes are supported by two nodes then the fall back to UD. This 
> > option is
> > > > > >not too different from UD QP + RC or UC mechanism.
> > > > >
> > > > > KISS:
> > > > >
> > > > > - UD universal
> > > > > - *C opportunistic
> > > > >          - Local management issue to control what is sent on the *C
> > > > > interface.  No need to specify
> > > > >          - Advertise whether one or more ports are supported by UD 
> > or *C
> > > > >          - Advertise whether one or more QP are supported by UD or *C
> > > > >          - Let local management determine policy for what services are
> > > > > mapped where - no need to specify
> > > > >
> > > > > This is both an interoperable approach and simple to 
> > implement.  There may
> > > > > be some desire to add a policy interface to state preference for 
> > specific
> > > > > types of traffic over a given QP.  I would not oppose this but 
> > would view
> > > > > this as a separate draft once the basics are worked out.
> > > > >
> > > > >
> > > > >
> > > > > ><VK>
> > > > > >b. Another suggestion was to allow multiple connected mode links 
> > (i.e. at
> > > > > >IB UC/RC level) between peers.  One thought can be 'yes, but user 
> > beware':
> > > > > >The IB connections are made using the service ID that is derived 
> > from the
> > > > > >QPN as described in the draft. If a second attempt succeeds then 
> > there are
> > > > > >two links. It is up to the implementation to either allow or disallow
> > > > > >multiple links.
> > > > > >
> > > > > >Again, this has been suggested in the past (though most who were 
> > involved
> > > > > >in the original discussions years gone by are likely gone since 
> > much of
> > > > > >this discussion occurred before the IETF workgroup was established).
> > > > > >
> > > > > ><VK> I'm one of the vestiges of those early times along with you 
> > and a few
> > > > > >others...so we have hope :). <VK>
> > > > > >
> > > > > >There is obvious benefit to supporting multiple RC per endnode 
> > pair. I do
> > > > > >not see any technical reason to oppose nor any issue from an
> > > > > >interoperability perspective. There is no reason for a "user beware".
> > > > > >
> > > > > ><VK> It is not opposed. The 'user beware' is only underscoring 
> > that the
> > > > > >the peer interface might not support multiple links- it might 
> > enforce a
> > > > > >limited number of connections (maybe only one) between a pair of GIDs.
> > > > > >Similarly, an implementation not wanting to support multiple links 
> > MUST
> > > > > >take steps to deny multiple requests.
> > > > >
> > > > > *C requires CM to operate thus it is a local issue whether 
> > additional CM
> > > > > operations are accepted or not.  A given requester node may issue N 
> > and a
> > > > > given responder may state 0-N as an implementation may limit the 
> > number of
> > > > > *C available for IP traffic.
> > > > >
> > > > >
> > > > > ><VK>
> > > > > >
> > > > > >The work is rather straight to do and implement and the benefit to
> > > > > >customers, is again, rather obvious when one considers what the IB 
> > fabric
> > > > > >offers and how connections can be enable flows through multipath 
> > as well
> > > > > >as transparent fail-over, flow scheduling, mapping of DiffServ to
> > > > > >different arbitration / paths, etc.
> > > > > >
> > > > > ><VK> In addition Large MTU and APM are two of the main reasons why 
> > I've
> > > > > >been proposing IPoIB-connected mode for so long. In terms of IPoIB 
> > itself,
> > > > > >except for the Large MTU, the parameters are hidden from it.<VK>
> > > > >
> > > > > Mike
> > > >
> > > >__
> > > >
> > > >Vivek Kashyap
> > > >Linux Technology Center, IBM
> > > >
> > > >
> > > >_______________________________________________
> > > >IPoverIB mailing list
> > > >[email protected]
> > > >https://www1.ietf.org/mailman/listinfo/ipoverib
> > >
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email protected]
> >https://www1.ietf.org/mailman/listinfo/ipoverib
>