Re: A Couple of IPoIB Questions

At 10:46 PM 11/17/2004, Vivek Kashyap wrote:
>Mike the format is really off in the last mail from you - making it difficult
>to follow.
>
>
>Other than that let us discuss in the context of the draft. The draft is
>built upon the following:
>
>1. IPoIB-RC and IPoIB-UC are optional.

I would prefer only one be used - either RC or UC.  I've provided some 
logic for either one as a preference but don't see a reason to have 
both.  Both just leads to options which leads to interoperability problems.

>2. IPoIB connected mode depends on a UD QP for address resolution and 
>multicast.
>
>As far as I know, there has been an agreement since the earliest connected 
>mode
>draft I posted.
>
>
>I'd like the WG to give input on the following issues:
>
>3. Where does the UD QP come from?  Choose one of:
>
>a. It is a UD QP that is associated with the interface at startup.
>
>b. It is a UD QP that is shared with IPoIB-UD.
>
>
>3a is more generic. It can be considered to include the case 3b.  The original
>proposal was limited to 3b.

 From an implementation point of view, all of this will be hidden within 
the driver below IP.  As such, the driver will maintain the 
associations.  Currently, each driver "instance" (may be multiple per IB 
port) will have at least 1 UD QP.  Given the existing protocol already 
defines how to share this QP with other nodes, why not just re-use it and 
avoid doing more work?  The driver can then map on a per endnode pair basis 
what *C QP go with what the UD QP and the spec remains largely silent on 
how this is accomplished.

>4. Link characteristics
>
>The broadcast domain for IPoIB-RC/UC is determined exactly as the
>IPoIB-UD case i.e. through the broadcast-GID. A UD as per 3 is used in this
>step.
>
>Do all interfaces in the IPoIB-conneced mode(CM) have the same link 
>characteristics? i.e.

 From an implementation perspective, this is generally simplest.

>a. all are either IPoIB-RC or IPoIB-UC.

Preference is only 1 to be defined.


>         -- There is also a UD QP associated. The UD QP will be either 3a 
> or 3b
>            based on WG concensus.
>
>         -- All unicast transmission is on the IPoIB mode i.e. RC or UC.

For a given endnode pair, the policy of which QP is used for a given 
unicast IP datagram is really a local issue.  I see some merit in the 
attempt to bifurcate this to multicast / broadcast to the UD QP and unicast 
to the *C QP.  However, if the datagram fits in the PMTU of the UD QP, then 
either could be used.  The driver would work either case.  Please keep in 
mind that multiple *C QP can be used and their usage needs to be a local 
issue and not defined within the spec.

>b. all are IPoIB-UD. Additionally they can be one of IPoIB-RC or IPoIB-UC
>or both.
>
>         -- The presence of the flags indicate the type of communication 
> possible.
>         -- The decision of communicating using a specific mode is 
> determined by
>            the supported modes and the local policy. Note that incompatible
>            policies imply that the fallback is communication over UD.
>         -- fallback mode of communication is UD
>
>
>4b adds a lot of flexibility at the expense of a simple decision. 4a. by
>contrast is straightforward.
>
>
>5. MTU negotiation
>
>         In the private data field of the CM message the desired MTU is
>         included.
>
>         It was suggested during the IPoIB meeting at IETF that it need not be
>         symmetric. That is a good idea. Thus each peer declares the max 
> MTU it
>         prefers
>
>
>         REQ: <my desired MTU>
>         REP: <my desired MTU>
>         RTU:

Rephrase this as maximum logical MTU to avoid confusion with the IB link 
MTU.  If you start down this path, then you may need to also consider an 
exchange of what range of DiffServ code points to use as well.  Not clear 
that anyone needs to deal with any latency or bandwidth guarantees but the 
"camel's nose is starting to enter the tent" as the saying goes.


>6. Multiple connections for the same IP address
>
>         Local decision. Note that the peer might choose to not honour 
> multiple
>         connections.

Agreed.

Mike




>Vivek
>
>
>
>
>
>On Wed, 17 Nov 2004, Michael Krause wrote:
>
> > At 11:38 PM 11/16/2004, Vivek Kashyap wrote:
> >
> >
> >
> > >Hi,  I have a couple of questions relative to IPoIB:  1.
> > >draft-ietf-ipoib-ip-over-infiniband-07.txt states: "Every IPoIB interface
> > >MUST "FullMember" join the IB multicast group defined by the
> > >broadcast-GID."  Isn't the broadcast group for IPv4 ? When the IPoIB
> > >interface is IPv6 only, does this group still need be joined ?  If not,
> > >where do the parameters for any IPv6 groups come from ? I am presuming
> > >that this group needs to be joined in  the IPv6 only case. I just want to
> > >be sure.
> > ><VK> Yes, the broadcast-GID is at the InfiniBand layer and MUST be joined
> > >whether you are running at v4 or v6 layer. <VK>  2. ALso, what is the
> > >latest status of the Vivek's connected mode draft ? Will it be moving
> > >forward ?  <VK> I'll be submitting it as
> > >draft-ietf-ipoib-connected-mode-00.txt by the end of the month. There were
> > >some interesting suggestions that were made during the IETF WG meeting.
> > >Two of the suggestions of consequence are given below. The others we can
> > >discuss when the minutes are published (they include some additional
> > >requests on clarification on the transmission draft too).  a. The current
> > >draft makes the various modes mutually exclusive i.e. RC, UC and UD are
> > >not allowed simultaneously in the same IP subnet. The thought is that it
> > >is a link characteristic and hence different per connection mode. It was
> > >suggested that one be allowed to mix up RC/UC. This goes back to the
> > >original suggestion in the first draft which was:  IPoIB-UD must always be
> > >supported. Additionally, the interface can also support either both of RC
> > >and UC, or one of them. Or neither of them.
> > >
> > >UD MUST always be supported.
> > >
> > ><VK> That is and has always been the requirement right from the first
> > >draft. <VK>
> > >
> > >I personally don't care whether one does RC or UC but I don't think both
> > >are required as a MAY option. The advantage of RC is the send credit
> > >algorithm. The advantage of UC is the lack of ACK packets. ACK is noise in
> > >the fabric while send credits provide a simple method to maintain
> > >bandwidth / injection control on a per flow basis.
> > >
> > >I see no problems with supporting both UD and *C on the same subnet; it is
> > >rather foolish to attempt to mandate these be on separate subnets.b
> > ><VK> As per the connected-mode draft the UD mechanism is *always*
> > >required; address resolutoin depends on it.
> > >
> > >The only point of discussion is whether all nodes must support the same
> > >link characteristics in the subnet i.e. all are RC (and UD), or all or UC
> > >(and UD), or all are UD only.
> >
> > Obviously I would oppose such a solution as it creates artificial
> > constraints with little benefit.
> >
> > >The alternative is to allow all the nodes to be mixed up with some nodes
> > >being RC/UD, others UC/UD and a third set UD only and yet others probably
> > >supporting all. within the same IP subnet. [Can the same serviceID be used
> > >by both RC and UC ?]
> > >
> > >The third alternative is to associating UD only or UD + one of RC or UC on
> > >the same interface. In such a case if mismatched/unsupported connected
> > >modes are supported by two nodes then the fall back to UD. This option is
> > >not too different from UD QP + RC or UC mechanism.
> >
> > KISS:
> >
> > - UD universal
> > - *C opportunistic
> >          - Local management issue to control what is sent on the *C
> > interface.  No need to specify
> >          - Advertise whether one or more ports are supported by UD or *C
> >          - Advertise whether one or more QP are supported by UD or *C
> >          - Let local management determine policy for what services are
> > mapped where - no need to specify
> >
> > This is both an interoperable approach and simple to implement.  There may
> > be some desire to add a policy interface to state preference for specific
> > types of traffic over a given QP.  I would not oppose this but would view
> > this as a separate draft once the basics are worked out.
> >
> >
> >
> > ><VK>
> > >b. Another suggestion was to allow multiple connected mode links (i.e. at
> > >IB UC/RC level) between peers.  One thought can be 'yes, but user beware':
> > >The IB connections are made using the service ID that is derived from the
> > >QPN as described in the draft. If a second attempt succeeds then there are
> > >two links. It is up to the implementation to either allow or disallow
> > >multiple links.
> > >
> > >Again, this has been suggested in the past (though most who were involved
> > >in the original discussions years gone by are likely gone since much of
> > >this discussion occurred before the IETF workgroup was established).
> > >
> > ><VK> I'm one of the vestiges of those early times along with you and a few
> > >others...so we have hope :). <VK>
> > >
> > >There is obvious benefit to supporting multiple RC per endnode pair. I do
> > >not see any technical reason to oppose nor any issue from an
> > >interoperability perspective. There is no reason for a "user beware".
> > >
> > ><VK> It is not opposed. The 'user beware' is only underscoring that the
> > >the peer interface might not support multiple links- it might enforce a
> > >limited number of connections (maybe only one) between a pair of GIDs.
> > >Similarly, an implementation not wanting to support multiple links MUST
> > >take steps to deny multiple requests.
> >
> > *C requires CM to operate thus it is a local issue whether additional CM
> > operations are accepted or not.  A given requester node may issue N and a
> > given responder may state 0-N as an implementation may limit the number of
> > *C available for IP traffic.
> >
> >
> > ><VK>
> > >
> > >The work is rather straight to do and implement and the benefit to
> > >customers, is again, rather obvious when one considers what the IB fabric
> > >offers and how connections can be enable flows through multipath as well
> > >as transparent fail-over, flow scheduling, mapping of DiffServ to
> > >different arbitration / paths, etc.
> > >
> > ><VK> In addition Large MTU and APM are two of the main reasons why I've
> > >been proposing IPoIB-connected mode for so long. In terms of IPoIB itself,
> > >except for the Large MTU, the parameters are hidden from it.<VK>
> >
> > Mike
>
>__
>
>Vivek Kashyap
>Linux Technology Center, IBM
>
>
>_______________________________________________
>IPoverIB mailing list
>[email protected]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib