Re: IPoIB link address

Kanoj and I discussed the issues raised in this thread offline. After some
discussion we came to the conclusion that it is better to have the nodes
always full-join the broadcast-GID and not query or non/sendonlynon member
join the broadcast GID. If that is done then the implementations are much
simpler and cleaner.


Therefore, we suggest that the proposal of 'one may receive the
broadcast-group attributes using a query/other join modes' be dropped.
Note this does not mean that every QPN (if multiple are used) on a given
port MUST join the broadcast-GID but that at least one of them MUST join.


I'd like to suggest the following two as the conclusion of this thread.

i) There is a single broadcast-GID associated with any IPoIB link. This
broadcast-GID is -> FF1x:401B:<P_Key>::255.255.255.255.

ii) The broadcast-GID MUST be full-member joined by the IPoIB interface.


Thanks
	Vivek



On Sat, 4 Oct 2003, Vivek Kashyap wrote:

> Kanoj,
>
> In summary,
>
>     - we have had the notion of full-joining (and I don't include query in
>     joining) the broadcast GID.
>
>     - now it has been proposed that one may not join the broadcast-group
>     and get the attributes using a query or may join but not full join.
>
>     - the IPoIB link is defined by creating a broadcast-GID. The
>     broadcast-GID is the source of the link's attributes.
>
>     - if there is no broadcast GID then there is no link.
>
> Now, given the above we have the follwoing suggestion:
>
> A) Broadcast-GID defines the link. It MUST not be deleted/modified if the
> IPoIB subnet is in use.
>
> B) Nodes always derive link attributes from the broadcast-GID. If a node
> discovers that the broadcast-GID is no longer in existence it MUST declare
> the interface 'down'. If the node discovers that the link-attributes are
> different than the ones that it had discovered earlier then it SHOULD
> reconfigure itself with the new attributes. A node may detect the
> existence/disappearance of the broadcast-GID by using traps/reports or the
> node might choose to full-member join the broadcast-GID.
>
>
> <snip..>
> > >
> > > I believe that we all agree that the broadcast-GID defines the link. If
> > > there is no broadcast-GID there is no link.
> > >
> > > The question to look into is: If there is no link then shouldn't the hosts
> > > react by reflecting this condition to their stacks? What should the
> > > reaction be?
> >
> > Two things here: in an ethernet fabric, if the link between two switches in
> > the fabric break in a way that broadcast traffic from one node can not reach
> > some others (maybe causing segregated nodes as discussed previously), do the
> > end node stacks get told about this? In fact, if the ethernet link is completely
> > broken, maybe by pulling out all the wires, do any of the stacks get told
> > about this? Of course, ethernet does not provide a way to detect this condition
>
>
> By the word 'down' above (and we can define the word more or substitute
> another word) I do mean that the packets are no longer transmitted or
> received. Same as the case of cables pulled you mention. Depending on the
> implementation 'telling the stack', can mean various things -- I just mean
> that it has an effect such that no packets can get transmitted or
> received.
>
> > possibly, whereas IB does (via the GID deletion trap), so it might make
> > sense to decide whether IB should do something that ethernet does not.
>
> We have done IB specific things -- such as a more elaborate multicast
> mapping with indications for the protocol being mapped.
>
> >
> > Also, I think we should be clear: I believe link conditions should not be
> > reflected to the IP stack (aren't we mixing L2 and L3 if we do that?), but
> > as you mention at other places, the L2 driver can stop tx/rx of packets. But
> > definitely the driver can not (for example) "inform" the stack that some
> > IP multicast membership that was previously granted has been revoked;
> > although it can silently drop those mcg packets.
>
>
> That is something I've said before -- no link implies cannot send packets.
> That is even if the IB cable is connected and possibly one could transmit
> unicast/mcast packets to peers, but the broadcast-GID is gone, then: do
> not transmit/receive packets. I agree with that. 'Inform' stack as you
> mention above is not what I had in mind.
>
>
>
> >
> > >
> > > >
> > > > If you are trying to solve the issue of how to change link parameters safely
> > > > in a racefree manner, please list the steps that the admin/hosts have to do;
> > > > I don't believe JOIN or traps solve all aspects of the problem, unless the
> > > > admin iterates thru all the groups; contrary to what you are stating above.
> > >
> > > I think we are looking at the issue differently. One way is to say that
> > > the MGIDs have to be looked at by the admin. The other way is to say that
> > > if the broadcast-GID is removed, the host brings the interfaces down. All
> > > packet transfer automatically stops. The other is to also imply that since
> > > there is no broadcast-GID the derived MGIDs do not make any sense any more
> > > and hence the hosts need to purge them. The admin doesn't do anything in
> > > this case -- I believe this has been Vandana's point too.
> >
> > So, are you saying that the hosts MUST drop IBA memberships to other
> > groups too when it receives the broadcast-GID deleted trap? This was
> > what I hinted in my previous mail.  Are you suggesting nodes that
> > JOINed the broadcast GID also have to do this (otherwise, how would
> > you do any recovery actions on erroneous deletions of the bcast-GID?) I don't
> > believe Vandana mentioned this, I would like to hear her comment specifically
> > if its okay to put more requirements for nodes that do JOIN.
>
> I was, probably not very clearly, giving options: if the link is being downed
> then the rules that can help are -
>
> one way -->
>     Admin must look up the related MGIDs and nodes and down/delete them
>
> 2nd way -->
>     if no broadcast-GID then the nodes stop transmission/reception.
>
> 3rd way (or extension to 2nd way) -->
>
>     if no broadcast-GID should we also drop all MGIDs. If this was done
> then the admin doesn't have to manually or through some other means have
> to get the nodes to leave the membership.
>
> The MUST clause in (B) above is the '2nd way'.
>
> >
> > As I mentioned, if you believe a certain handling of bcast-GID
> > deleted trap will help solve some problem(s), (when a node JOINs
> > or doesn't JOIN the broadcast GID) please list the problem(s) and how
> > using the traps help, specifically by listing the steps the admin
> > and hosts must take. I have tried to portray how it doesn't help, at
> > least for the problem(s) we have discussed.
> >
> > On the other hand, if you believe trap usage requirement is due to
> > some different view point, or some architecture or cleanliness issue,
> > and really does not contribute to solving or creating any more problems,
> > please state that. I would be very disinclined to putting more requirements
> > based on this argument, but at least I (and maybe others) will clearly
> > understand the basis of the requirement.
>
> The above is not clear to me -- the traps/reports are suggested in (B)
> above. The suggestion is to track the disappearance of broadcast-GID so
> that the interface may be 'downed' (we can define downed suitably).
>
>
> >
> > >
> > >  >
> > > > If you are trying to solve the problem of how to track all IP nodes on the
> > > > fabric, thats a different problem, and yes, I agree that requring all nodes
> > > > to JOIN will give you an instantaneous snapshot. I don't know though how this
> > > > information can be used meaningfully.
> > >
> > > At the begining of the this thread the issues with broadcast-GID
> > > removal/modification were listed. Those issues are adequately addressed by
> >
> > I disagree; I don't believe having nodes EITHER join bcast-GID OR track
> > bcast-GID solves any problems for the host or the fabric; specially if
> > I don't see the rules of what a host that subscribes to the trap must do
> > (and secondarily, how those actions might help the host itself or the
> > fabric administrator).
>
> The host (in the driver or anywhere) must stop transmission or reception
> of packets on the IPoIB interface(s) associated with that broadcast-GID.
> That is the MUST clause in (B).
>
>
> >
> > > using both (A) and (B). (B) appears to be simple enough to implement --
> > > just a report; same as the reports used with other MGIDs.
> >
> > I am not sure what you mean by "same as reports used with other MGIDs".
>
>
> I mean implmenting reception of reports for broadcast-GID is not difficult
> afterall the reports are probably requestd for MGIDs to which one might
> joing send-only or non-member or 'create MGID' events.
>
> > I thought we are discussing what the host SHOULD/MUST do when it discovers
> > the bcast-GID deleted erroneously or recreated.
> >
> > >
> > >  > > > > >
>
> <snip...>
>
>
> > > > But to what good though? If the MGID gets deleted, how does it matter how
> > > > the hosts behave? We agree that the MGID getting deleted when hosts are
> > > > active is erroneous; assuring the admin that the end nodes will "do the
> > > > right thing" (declaring link down et al) is dangerous; because he might
> > > > turn around and recreate the MGID with new properties, and face all the race
> > > > problems with other existing IP multicast groups still using old properties.
> > >
> > > When the link is down -- it follows that no packets leave or are received
> > > on the interface. So the existing MGIDs cannot be used either.
> >
> > So, do the nodes LEAVE these other IP groups (and if so, how many seconds
> > does the admin wait to make sure all nodes have done the LEAVEs and thus
> > all old groups are gone, before he can safely recreate)?
> >
> > Or should the admin just delete these other groups? I think this is what
> > you were trying to prevent in the first place, ie having the admin iterate
> > over the groups.
>
> IP layer is not affected by the MGIDs going away.
>
>
> <snip..>
>
> > >
> > > What if one views it as - no broadcast-GID is same as no link. In IPoE it
> > > could be cable pulled or a downed switch. It atleast means that the
> > > interface is down.
> >
> > The distinction is that IP still sends packets to the driver, the driver might
> > not send it out (or the transmit fails). IP's memberships to groups are not
> > affected. The ipoib driver can drop IBA memberships without telling IP, but
> > are you suggesting that?
>
> OK, the crux of the matter...
>
>
> Yes, by 'down' the interface I meant that the link is down. Whether the
> down indication causes the IP interface to be marked down or whether the
> driver just queues packets/drops packets is upto the implementation.
>
> IP's membership is not affected, that is correct. That is certainly not
> what I suggested.
>
> I did ask if we could, because all MGIDs are derived from broadcast-GID
> and if the broadcast-GID is down drop the MGID membership? Yes, that
> amounts to dropping IBA membership without telling IP.
>
> >
> > To conclude, I still can not see that option B is required; I will refrain
> > from reiterating that opinion, unless I see more logic to change it.
>
> Well, reading the rest of the discussion above I was getting the
> impression that you are not exactly disagreeing.....
>
>
> Vivek
>
> >
> > Thanks.
> >
> > Kanoj
> >
> > >
> > > Modification of broadcast-GID is same as moving the NIC to a different
> > > ethernet segment -- things might not work any more.
> > >
> > > These are the issues covered by (B).
> > >
> > > Vivek
> > >
> > > > Thanks.
> > > >
> > > > Kanoj
> > > >
> > > >
> > > > > Note: read 'down' above as 'no link'.
> > > > >
> > > > > >
> > > > > > >
> > > > > > >>
> > > > > > >> Similarly link attributes shouldn't be modified without some coordination
> > > > > > >> between fabric mgmt and IP software and that coordination shouldn't be part
> > > > > > >> of IPoIB spec just like IP over Ethernet doesn't talk about how to cope
> > > > > > >> with VLAN reconfiguration or changes in VLAN properties.
> > > > > > >
> > > > > > >In Ethernet we also do not talk about the link-broadcast address
> > > > > > >determining the existence of the link - in IPoIB it is the basic
> > > > > > >requirement. However, ethernet nodes to respond to a downed link. In IPoIB
> > > > > > >the disappearance of a broadcast-GID is equivalent to a downed link: no
> > > > > >
> > > > > > I guess this is the main difference between our view points. You have
> > > > > > the notion of a "downed link" while in my mind the broadcast group is
> > > > > > an integral part of the IPoIB link (because it defines the link attributes
> > > > > > such as MTU, Q_Key...). Without the braodcast group, we don't have a link
> > > > > > (not ever a "downed" one).
> > > > >
> > > > > No, that is not the difference. We are talking the same thing except for the
> > > > > use of different words and more importantly the reaction to the possibility
> > > > > of the broadcast-GID disappearing.
> > > > >
> > > > > I used the words 'downed link' to imply the link is gone(no link since no
> > > > > broadcast-GID) from the point of view of the IPoIB interface. Therefore since
> > > > > the link is non-existent it is illegal for the interfaces to continue to be
> > > > > functioning. You prefer that we make it illegal for the link to
> > > > > go away  by mandating that the link always be administratively kept up/in
> > > > > existence.
> > > > >
> > > > > I actually think that we need both as addressed in (A) and (B) above.
> > > > >
> > > > > Vivek
> > > > >
> > > > > >
> > > > > > Jerry
> > > > > >
> > > > > > >one can broadcast and no one new can join. It is a downed link for
> > > > > > >implementations that do not actively join the broadcast-GID also, since we
> > > > > > >require that all implementations derive their attributes from the
> > > > > > >broadcast-GID.
> > > > > > >
> > > > > > >Vivek
> > > > > > >
> > > > > > >> So my preference remains to be a).
> > > > > > >>
> > > > > > >> Jerry
> > > > > > >>
> > > > > > >> >-Vandana
> > > > > > >> >
> > > > > > >> >>We can avoid going into the details in the specification since the details
> > > > > > >> >>listed above are largely informational. However, I believe that the case
> > > > > > >> >>of 'no-broadcast GID' needs to be specified. Is it not similar to the case
> > > > > > >> >>of a downed interface?
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>My preference is for (c).
> > > > > > >> >>
> > > > > > >> >>Vivek
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>Vivek
> > > > > > >> >>
> > > > > > >> >> >
> > > > > > >> >> > Jerry
> > > > > > >> >> >
> > > > > > >> >> > >
> > > > > > >> >> > >-Vandana
> > > > > > >> >> > >
> > > > > > >> >> > >
> > > > > > >> >> > >_______________________________________________
> > > > > > >> >> > >IPoverIB mailing list
> > > > > > >> >> > >[email protected]
> > > > > > >> >> > >https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > _______________________________________________
> > > > > > >> >> > IPoverIB mailing list
> > > > > > >> >> > [email protected]
> > > > > > >> >> > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >>__
> > > > > > >> >>
> > > > > > >> >>Vivek Kashyap
> > > > > > >> >>Linux Technology Center, IBM
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> _______________________________________________
> > > > > > >> IPoverIB mailing list
> > > > > > >> [email protected]
> > > > > > >> https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > >__
> > > > > > >
> > > > > > >Vivek Kashyap
> > > > > > >Linux Technology Center, IBM
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > IPoverIB mailing list
> > > > > > [email protected]
> > > > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > >
> > > > > >
> > > > >
> > > > > __
> > > > >
> > > > > Vivek Kashyap
> > > > > Linux Technology Center, IBM
> > > > >
> > > > > _______________________________________________
> > > > > IPoverIB mailing list
> > > > > [email protected]
> > > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > [email protected]
> > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > >
> > >
> > > __
> > >
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > [email protected]
> > > https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> > _______________________________________________
> > IPoverIB mailing list
> > [email protected]
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> >
>
> __
>
> Vivek Kashyap
> Linux Technology Center, IBM
>
>
> _______________________________________________
> IPoverIB mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/ipoverib
>
>

__

Vivek Kashyap
Linux Technology Center, IBM