Re: IPoIB link address

On Thu, 2 Oct 2003, H.K. Jerry Chu wrote:

> <snip>
>
> >> Not sure I understand what exactly c) is. It refers to v6 only so v4 MUST
> >> always join the broadcast group, right?
> >
> >The suggestion is in general for an IPoIB link. The link attributes are
> >derived from the IPoIB broadcast-GID. In the case of v4 it just so happens
> >that IP broadcast maps to this and so a node will surely join it. We have
> >earlier always talked of full-join since the node will send and receive
> >packets on the broadcast GID. However, with the notion that one need not
> >join the broadcast GID I believe we need to, irrespective of the IP layer,
> >state that :
>
> I am confused. What is the notion that "one need not join the broadcast
> GID" for v4? There isn't any leeway for v4 - it must join the broadcast
> group for ARP to work.

Jerry, I think we are all confused :).


Join is of three types: Full member, Nonmember, Sendonlynonmember.


As far as I know one will have to join (implying one of the above three) to
retrieve the multicast information. In the case of v4, for the broadcast-GID
one will have to either full-member join or non-member join. Both allow for
sends and receives except the latter does not count the requester for
deletions(at the SM).

In my previous note I meant 'need not full-join'. Until now we have always
talked in context of full-join, however if we are now talking about not
full-joining I would rather have a common specification whether v4 or v6
and not have an exception for v6 only.


>
> >
> >a) either full-join b) or track the broadcast-GID using traps/reports.
> >(the GIDs could be joined sendonly or nonmember joined or as some have
> >suggested not joined at all).
> >
> >
> >>
> >> I prefer not to require v6 to join the broadcast group. This is compatible
> >> with c). But I prefer to continue to leave the creation/maintainence of
> >
> >Yes, it is in conformance with (c).
> >
> >> the broadcast group out of scope because the broadcast group should really
> >> be managed together with the IPoIB link as part of the fabric management
> >> function. (Note that this doesn't necessarily imply admin involvement.
> >> If implementations can somehow find a way to embed this function in IPoIB
> >> drivers that is fine too.) If we start getting into problems that really
> >> belong to fabric management, we may never find a bullet-proof solution within
> >> IPoIB itself. E.g., if we don't require the continuous existence of the
> >> broadcast group as part of the IPoIB link, but we merely rely on IP nodes
> >> joining so the broadcast group won't go away, what happen if for some reason
> >> all the IP nodes went down, and the broadcast group got deleted as a result
> >> of zero member. On a subsequent boot, what should IPoIB driver do? Create
> >> the broadcast group? But how would the driver know what link attributes
> >> to use? Implementations may supply link attributes to the IPoIB driver
> >> through some implementation-specific mechanism. But we can't make this a
> >> requirement in IPoIB spec, right?
> >
> >It is not with fabric management that I have any concerns.. it is with
> >nodes that are up and running and with the suggestion that they not be
> >full-members of the broadcast-GID. If there are no members then the
> >broadcast-GID can be easily deleted - I'm sure that every impelementation
> >will have safeguards against it. Option (a) merely is stating that - have
> >safeguards and keep it up. But it is not comprehensive and leaves some
> >aspects to individual impelementations which could lead to
> >interoperability issues.
> >
> >What happens if, for some reason, the link needs to be modified. The admin
> >has to bring the link down -- with no members the admin has to resort to
> >other methods (and it is not an IBA issue) to determine if there are IPoIB
> >members on the link. It cannot be determined trivially, one needs to look
> >for other MGIDs and guess or find some other method. If there are members
> >in the broadcast-GID then it is known easily that the link is in use.
> >
> >Now, let us go with (a) and choose to bring the link down. What do we do
> >now? As per (a) - say it is out of scope of our specification which really
> >means there could be interoperability issues since implementations could
> >do things differently.
>
> We shouldn't have to deal with link reconfiguration any more than IP over
> Ethernet dealing with VLAN. IPoE doesn't ever try to deal with any L2
> configuration issue. The admin had better bring down the IP stack on
> nodes before bringing down a VLAN and reassign its id. Any
> interoperability issue resulted from adminstrator misstep IMO is out
> of scope.

Not all correlations hold between the two layers.  As I noted, IPoE also
does not state that the link is defined only by joining the broadcast link
address. IPoIB does and therefore is much more closely intertwined with what
happens to the broadcast-GID.

>
> >Instead why not have it that if the broadcast-GID
> >is deleted or modified then the nodes declare the link down or modfied
> >respectively. It allows for a commonly agreed method of downing the link
> >and all the nodes will respond the same way.
> >
> >If we go with (a) and if there is an error and the GID does get deleted.
> >What happens? As per (a) -- out of scope. Instead we can say that if the
> >link is deleted then the node declares the link down - we therefore have a
> >common agreed response across all implementations.
>
> Ok.
>
> >
> >If all nodes go down and the link/GID stays there are no issues. The node
> >coming up will join the link and all is fine. If the nodes go down and the
> >GID is not there then the nodes will complain. The situation is the same
> >whether we choose (a) or (c). It is equivalent to first bringup of the
> >link.
>
> No, this is not an issue if we require a). If we don't require a), but
> instead relying on non-empty broadcast membership to persist the broadcast
> group as suggested by c), it will be an issue when all nodes go down.

We are also considering possible error cases. Note that (a) is  essentially
asking the layer 2 to ensure that the interface always stays up but due to
some reason it could go down. That is what I've attempted to address in
(A) and (B) below -- (A) as a requirement on the administrator and (B) as
a requirement on the hosts. It also takes care of managed shutdown of the
link.

>
> >
> >In summary it appears to me that we can combine the two. (A) describes the
> >requirement on the administrator to keep the link consistent. (B)
> >describes the requirement on the hosts.
> >
> >A) Broadcast-GID defines the link. It MUST not be deleted/modified if the
> >IPoIB subnet is in use.
> >
> >[Just (A) has the following issues issues:
> >	what about managed shutdown of the link?
> >	what about erroneous deletion/modification of broadcast-GID?]
>
> I'd say both are out-of-scope. See my IPoE analogy above.
>
> >
> >B) Nodes always derive link attributes from the broadcast-GID. If a node
> >discovers that the broadcast-GID is no longer in existence it MUST declare
> >the interface 'down'. If the node discovers that the link-attributes are
> >different than the ones that it had discovered earlier then it SHOULD
> >reconfigure itself with the new attributes. A node may detect the
> >existence/disappearance of the broadcast-GID by using traps/reports or the
> >node might choose to full-member join the broadcast-GID.
>
> Although I still don't see the necessity of including the above paragraph,
> it does look much better and more acceptable.

OK. Is the above (A) and (B) acceptable to all?
Note: read 'down' above as 'no link'.

>
> >
> >>
> >> Similarly link attributes shouldn't be modified without some coordination
> >> between fabric mgmt and IP software and that coordination shouldn't be part
> >> of IPoIB spec just like IP over Ethernet doesn't talk about how to cope
> >> with VLAN reconfiguration or changes in VLAN properties.
> >
> >In Ethernet we also do not talk about the link-broadcast address
> >determining the existence of the link - in IPoIB it is the basic
> >requirement. However, ethernet nodes to respond to a downed link. In IPoIB
> >the disappearance of a broadcast-GID is equivalent to a downed link: no
>
> I guess this is the main difference between our view points. You have
> the notion of a "downed link" while in my mind the broadcast group is
> an integral part of the IPoIB link (because it defines the link attributes
> such as MTU, Q_Key...). Without the braodcast group, we don't have a link
> (not ever a "downed" one).

No, that is not the difference. We are talking the same thing except for the
use of different words and more importantly the reaction to the possibility
of the broadcast-GID disappearing.

I used the words 'downed link' to imply the link is gone(no link since no
broadcast-GID) from the point of view of the IPoIB interface. Therefore since
the link is non-existent it is illegal for the interfaces to continue to be
functioning. You prefer that we make it illegal for the link to
go away  by mandating that the link always be administratively kept up/in
existence.

I actually think that we need both as addressed in (A) and (B) above.

Vivek

>
> Jerry
>
> >one can broadcast and no one new can join. It is a downed link for
> >implementations that do not actively join the broadcast-GID also, since we
> >require that all implementations derive their attributes from the
> >broadcast-GID.
> >
> >Vivek
> >
> >> So my preference remains to be a).
> >>
> >> Jerry
> >>
> >> >-Vandana
> >> >
> >> >>We can avoid going into the details in the specification since the details
> >> >>listed above are largely informational. However, I believe that the case
> >> >>of 'no-broadcast GID' needs to be specified. Is it not similar to the case
> >> >>of a downed interface?
> >> >>
> >> >>
> >> >>
> >> >>My preference is for (c).
> >> >>
> >> >>Vivek
> >> >>
> >> >>
> >> >>
> >> >>Vivek
> >> >>
> >> >> >
> >> >> > Jerry
> >> >> >
> >> >> > >
> >> >> > >-Vandana
> >> >> > >
> >> >> > >
> >> >> > >_______________________________________________
> >> >> > >IPoverIB mailing list
> >> >> > >[email protected]
> >> >> > >https://www1.ietf.org/mailman/listinfo/ipoverib
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > IPoverIB mailing list
> >> >> > [email protected]
> >> >> > https://www1.ietf.org/mailman/listinfo/ipoverib
> >> >> >
> >> >> >
> >> >>
> >> >>__
> >> >>
> >> >>Vivek Kashyap
> >> >>Linux Technology Center, IBM
> >> >
> >>
> >>
> >> _______________________________________________
> >> IPoverIB mailing list
> >> [email protected]
> >> https://www1.ietf.org/mailman/listinfo/ipoverib
> >>
> >>
> >
> >__
> >
> >Vivek Kashyap
> >Linux Technology Center, IBM
> >
> >
>
>
> _______________________________________________
> IPoverIB mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/ipoverib
>
>

__

Vivek Kashyap
Linux Technology Center, IBM