Re: GID clarifications

Michael Krause wrote:
> 
> At 08:22 PM 12/7/2003 -0800, H.K. Jerry Chu wrote:
> >Mike,

Hi,

Michael, thanks for providing the IBTA context about multiple GIDs and path selection.

> >
> > >At 06:04 PM 12/4/2003 -0800, bill wrote:
> > >>Ok, let me ask a few questions to lead us toward concensus
> > >
> > >As the author of the IB spec for chapter 4, I'll attempt to provide some
> > >clarifications.
> > >
> > >>1) Does anyone give a compelling reason to use anything but GID0 for IPoIB
> > >>- what is the benefit of doing this ?
> > >
> > >Multiple GIDs were provided as a method to:
> > >
> > >- Provide traffic segregation between subnets conceptually similar to
> > >multi-LID usage within a single subnet.
> >
> >According to IBA 1.0a spec (i don't have 1.1 spec at hand), "A subnet shall
> >have only one assigned GID prefix at any given time (at GID index 0)."
> >It seems to me unless a CA port can be simultaneously connected to more
> >than one IB subnet, the unique subnet prefix plus the default GID prefix
> >(the link-local 0xFF80 prefix) are the only two possible values to have
> >for the upper 64 bits of a GID.
> 
> Correct.  One can have either the default or an assigned thus all GID share
> one or the other.
> 
> >  I wonder what the traffic seggregation between subnets you have in mind.
> 
> The basic idea was one could encode in the lower 64 bits (these are admin
> assigned if not the port EUI64 value) or through a combination of other GRH
> attributes (e.g. TClass, Flow Label, etc.) one could segregate traffic to
> take different paths when passing through a router. The path selection
> would be implemented via selecting an egress port and assigning an
> appropriate next router hop / endnode LID.

It seems to me that for endports to influence which path to
use for each UD packet, the SM and the endnodes need to exchange 
some information. I was under the impression that SL/VL information 
within a IBA subnet, and additionally Tclass etc between IBA 
subnets, would be sufficient; ie, segregation could be achieved
with just these parameters, and would not need to extend to source
GID. I am not sure whether your use of the word "or" above means
that this is still a viable option.

> 
> > >- Provide a method to support software / OS virtualization and device
> > >sharing via using a unique global identifier independent of the number of
> > >LIDs assigned (though one can take a similar approach if the LMC value is
> > >greater than 0.
> > >
> > >I will note that there are many OSVs offering or starting to offer
> > >virtualization services and while some initial implementations may be
> > >focused on GID 0, this may not hold true in the coming few years.
> >
> >Not sure what flavor of "virtualization" you're alluding to. If the
> >virtualization is achieved through the use of multiple IB partitions,
> >different IB partitions can safely share the same GID much like
> >different Ethernet VLAN sharing the same MAC address. I don't see the
> >need for multiple GIDs.
> 
> Virtualization is nominally a function of address while partitions are a
> function of segregation enforcement ala 802.1Q.  While one can use
> partitions, they are not seen by the underlying software while a GRH is
> seen thus depending upon how it is implemented, how the remote peer "sees"
> each target peer, etc. a GID is more flexible and may align better with
> application usage.  Please keep in mind I'm explaining how the motivation
> for multiple GID from an IBA perspective.  Now, can one use a single GID
> with multiple P_Key and then have a unique IP-to-QP mapping per resulting
> partition to achieve something similar?  The answer is yes, it is possible
> but that was not the intent of the IBTA when this was first being debated.
> 
> >Perhaps you meant virtualization like the hosting of multiple IP
> >addressses with the same subnet prefix but different host numbers
> >on the same NIC, this can be achieved through the use of multiple
> >locally-assigned EUI-64s in IB, and I agree it is a useful feature
> >to support.

At least from the ipoib perspective, it is also possible to 
virtualize based on the QPN (using same Pkey), since that is 
part of the mac address. I do agree that virtualization based on GIDs
is attractive.

> >
> > >
> > >- Provide a method to constrain out some data flows are routed - local vs.
> > >non-local subnet access given a GID can use either the default or the SM
> > >assigned subnet prefix.
> >
> >Note that the draft currently recommends the use of the IB subnet
> >prefix for the GID in IPoIB. Although the real requirement is that the
> >GID must be reachable (i.e. routable) at the IB layer from any node
> >within the scope of the IP link it belongs to, recommending the IB
> >subnet prefix is less error prone as it continues to work even after
> >the underlying IP link has been expanded to cover multiple IB subnets.
> 
> Agreed.
> 
> > >
> > >>2) Does anyone have an implementation that uses anything but GID0 for
> > IPoIB ?
> > >
> > >I believe it would be a mistake to limit support to just GID 0.  IB
> > >technology was designed to support a diverse set of usage models that will
> > >evolve over time.  The only potential problem was a DLPI application raw
> > >level usage model which is (a) somewhat outside of the scope of the IP over
> > >IB specification and (b) rather limited in terms of usage within the
> > >industry.  I don't really want to debate the merits of raw DLPI but do
> > >acknowledge it is in use today on Ethernet and other layer 2 interconnects.
> > >

Note that as has been pointed out, it is also possible for non DLPI apps to
want to know the sender's L2 address (trivially possible on ethernet by looking
at the received packet and picking out the source ethernet address). As you point
out, this might be slightly out of scope, but here's an example that is within
scope: imagine a node A with mac address M1 wants to know what IP address the 
RARP server will assign for a different mac address M2; it can fabricate a RARP 
message and broadcast it out; the RARP server must send the RARP response to M1
and not to M2. For this, the RARP server must be able to determine that the
sender's L2 address is M1. Similarly, an ARP implementation might want to 
determine the sender of an ARP request and validate against the "sender hardware
address" in the ARP message. I can't make a call on how extensive this type
of usage is though.

Thanks.

Kanoj


> > >>If I do not hear any counter arguments by next Friday - I would like to
> > >>ask for concensus on MUST use GID0
> > >
> > >Please consider this a counter argument and a vote against such a movement.
> >
> >You are opposing requiring GID0, but are you ok with the current wording
> >of RECOMMENDING the GID formed by the IB subnet prefix and the portGUID
> >(the one assigned by the manufacturer)?
> 
> I can live with RECOMMENDING not with MUST.
> 
> >BTW, I'm confused by the use of GID0 as I thought in IBA spec. it means
> >the one with the link-local prefix but I thought people wanted to use
> >the one with the IB subnet prefix.
> 
> GID 0 is just the guaranteed GID to be created via the combination of the
> port GUID (manufacturer assigned GUID) and the subnet prefix.
> 
> Mike
> 
> >Jerry
> >
> > >
> > >Mike
> > >
> > >
> > >_______________________________________________
> > >IPoverIB mailing list
> > >[email protected]
> > >https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email protected]
> >https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> _______________________________________________
> IPoverIB mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/ipoverib