Re: IPoIB link address

On Mon, 29 Sep 2003, H.K. Jerry Chu wrote:

> Vivek & Vandana,
>
> My comments are embedded in the text below (with co-chair hat OFF).
>
> >> >         1. Define a broadcast-GID defined as follows (one of - please
> >> >         comment):
> >> >
> >> >         a) FF1x:401B::255.255.255.255
> >> >         b) FF1x:001B::255.255.255.255
> >> >         c) FF1x:0000::255.255.255.255
> >> >         d) FF1x:401B::1
> >> >
> >> >     My preference is b) or c) since it removes the IP version
> >> >     completely from the mapping. a) is what we have today and requires
> >> >     no change and so could be used. d) is derived from the multicast
> >> >     mapping and so I'd like to leave it separate.
> >>
> >>I would go for a) since it is already there; if we do change, I would
> >>go for d, but can live with b or c.
> >>
> >><VK> OK. I find (a), (b) and (c) acceptable. My order of preference (c),
> >>then (b) then (a). No (d). <VK>
> >
> >
> >My order of preference is (a) then (b). No (c) or (d). The reason I do not
> >think (c) is a good idea
> >is because having a signature defined is useful to mark these MGIDs as
> >IPoIB MGIDs. There is less possibility of MGID collision with non-IPoIB
> >applications
> >Also, (a) makes the most sense as a broadcast group applies only to v4, so
> >why not leave things as is.
>
> I'd suggest "191B" (for "IPIB") as the signature (FF1x:191B::255.255.255.255)
> to reflect the fact that the broadcast group is for both v4 and v6.
>
> But if people prefer any of the above it's fine with me too, as long as
> we can get a closure on this sooner.

:) - that is close 19 == IP.  For the discussion (b) therefore becomes
FF1x:191B::255.255.255.255.

OK  by me. Between a signature of 191B and 401B I believe Vandana and Kanoj
have preferred the latter. If they are still in favour of 401B then let us
choose that.



>
> >
> >> >
> >> >         2. The multicast mappings stay as currently specified.
> >> >
> >> >         3. The broadcast-GID (one of)
> >> >
> >> >         a) MUST be FullMember IB_joined
> >> >
> >> >         b) MUST be used to derive the attributes such as the MTU, Q_Key.
> >> >         The attributes MAY be obtained by FullMember IB_Join of the
> >> >         broadcast-GID or by some other operation supported by IBA. The
> >> >         node MUST fail all IP operations on the interface if the
> >> >         Broadcast-GID ceases to exist since that will correspond to a
> >> >         downed link. It is upto the implementation to track the existence
> >> >         fo the Broadcast-GID in a suitable way. Note: If the
> >>broadcast-GID
> >> >         is FullMember IB_joined then its existence/deletion is
> >> >         automatically tracked.
> >>
> >>I would go with option b, but without the requirement "MUST fail all IP
> >>operations ..." for multiple reasons; firstly, it requires at least a
> >>memory check in the send/receive path, and secondly, if other L2's
> >>(ethernet)
> >>aren't requring this, why should IB? (Seems like mingling an L2 issue with
> >>L3,
> >>the same logic that was used against seperate groups).
> >>
> >><VK> On further thinking tracking with a requirement to re-acquire modified
> >>attributes is needed. 'must fail' is not needed though for reasons below
> >>(analogous to why we want to track other multicast GIDs):
> >>
> >>Assume that the broadcast-GID was not FullMember joined and so ends up
> >>getting deleted. There are 3 possibilities (not counting a recreate with
> >>the same attributes and MLID):
> >>a) Broadcast-GID is not recreated.
> >>b) Broadcast-GID is recreated with different attributes but same MLID
> >>c) Broadcast-GID is recreated with different/same attributes but different
> >>MLID
> >>
> >>
> >>Now, if (a) occurs - the nodes still have agreed upon MTU, Q_Key etc. and
> >>can continue to communicate with peers. No new nodes can join the link
> >>though. No broadcasts can occur if the MLID was removed from the switch
> >>tables => no ARP. One  could let it all go on till nodes drop off(ARP
> >>timeouts for example) or declare interface is down.
> >>
> >>Now, if (b) occurs the link's attributes have changed and communication
> >>with nodes that were on the link before, and are still using the old
> >>information, can occur however communication with other nodes using new
> >>attributes on the link may fail. Broadcast however will probably succeed
> >>with some set of nodes (dependent on how the fabric is setup for this
> >>MLID).
> >>
> >>if (c) occurs then broadcast with nodes that were part of the link before,
> >>and are still using the old information, cannot occur (don't know the new
> >>MLID) though the node can still communicate with the nodes using older
> >>attributes.
> >>
> >>Therefore, the requriement really needs to be:
> >>       - track broadcast-GID
> >>       - reacquire attributes if broadcast-GID is recreated
> >>       - (possibly) if no broadcast-GID down the interface
> >>
> >>Thoughts?, especially for the case where the broadcast-GID is deleted and
> >>not brought back.
> >><VK>
> >
> >I agree. If end-nodes do not want to join the broadcast GID, they need to
> >deal with all the requirements that
> >you have detailed above.
>
> Vivek brought up a good point that the empty broadcast group risks being
> deleted by SM. But unfortunately this problem already exists in the current
> draft regardless of one or two broadcast groups.

I don't think that this risk existed .. one had to join the broadcast-GID
permanently -- an IPv4 node couldn't leave the v4 broadcast GID and the v6
couldn't leave the all-nodes multicast GID since those mappings are always
required by the v4/v6 stacks. The above issues crop up if
we allow acquiring the information but not full-joining - as some v6
only implementations are likely to do.

The thing is that if a group has users(full-join) then the deletion of the
group is not very likely -- the SM will complain or disallow the deletion.
Thus there is some admin control and check.  However, if no one is 'joined'
then there is nothing distinguishing it from no members or no one is
interested. The SM will quietly acquise to it being deleted OR even delete
the broadcast-GID by itself since there are no members.


>
> In my mind the broadcast groups for IPoIB are to be treated differently by IB
> fabric adminstrator/management software compared with other IB MC groups.
> They are to be created as part of configuring an IPoIB link, and their
> livelihood should not depend on the condition of nodes running an IP stack.
> (E.g. your Ethernet VLAN doesn't disappear when you shutdown all the nodes.)

I think it is reverse -- a stack might report a link down if  the
switch/NIC is down. In this case the broadcast-GID is down/gone -- which
implies that the link is down since we can't broadcast or new interfaces
can't join it.


> This reasoning is compatible with IBA 1.0 spec when there was "MCGroupRecord"
> to hold MC groups separate from "MCGroupRecord" that tracks membership.
> Unfortunately "MCGroupRecord" got removed in 1.0a spec. Now it seems that one
> must join a MCMemberRecord before an IB MC group can be successfully created.
>
> Instead of adding more complexity to the draft, can we leave this to the
> implementation, e.g., the fabric software to ensure the continuous existence of
> the special broadcast group as part of IPoIB link? I've checked with some people
> from IBTA and this seems to be a reasonable requirement that is very easy to
> meet.
>
> We can simply add a clause requiring implementations to ensure that the special
> broadcast group won't be deleted other than when the IPoIB link is torn down.
> Then we don't need to require v6 node to JOIN the broadcast group. (Note that
> even if we require v6 to JOIN, it doesn't solve the problem as I stated ealier.)


Yes, the requirement is that the link not be deleted. My interest is to
determine the level of specification. We could do one of following:


a) make it a requirement that the broadcast-GID won't be torn down unless
the link is torn down.


This is somewhat circular since we define the link using the broadcast-GID.
Additonally, the admin will have to find some method to know that the link
can be brought down.

b) Always have to full-member join broadcast-GID.

Now, the admin will always know if someone is using the link and can do a
managed shutdown.


c) Always derive the data from the broadcast-GID and track it either by
full-member joining or reports.

Now, the admin can delete/modify as needed. the nodes will respond accordingly.


We can avoid going into the details in the specification since the details
listed above are largely informational. However, I believe that the case
of 'no-broadcast GID' needs to be specified. Is it not similar to the case
of a downed interface?



My preference is for (c).

Vivek



Vivek

>
> Jerry
>
> >
> >-Vandana
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email protected]
> >https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
> _______________________________________________
> IPoverIB mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/ipoverib
>
>

__

Vivek Kashyap
Linux Technology Center, IBM