Re: IPoIB link address

At 10:24 PM 10/6/2003 -0700, Vivek Kashyap wrote:
>Kanoj and I discussed the issues raised in this thread offline. After some
>discussion we came to the conclusion that it is better to have the nodes
>always full-join the broadcast-GID and not query or non/sendonlynon member
>join the broadcast GID. If that is done then the implementations are much
>simpler and cleaner.
>
>
>Therefore, we suggest that the proposal of 'one may receive the
>broadcast-group attributes using a query/other join modes' be dropped.
>Note this does not mean that every QPN (if multiple are used) on a given
>port MUST join the broadcast-GID but that at least one of them MUST join.
>
>
>I'd like to suggest the following two as the conclusion of this thread.
>
>i) There is a single broadcast-GID associated with any IPoIB link. This
>broadcast-GID is -> FF1x:401B:<P_Key>::255.255.255.255.
>
>ii) The broadcast-GID MUST be full-member joined by the IPoIB interface.
>

This is good news from the perspective of keeping things simple.  I agree 
with the above conclusions completely.
-Vandana

>Thanks
>         Vivek
>
>
>
>On Sat, 4 Oct 2003, Vivek Kashyap wrote:
>
> > Kanoj,
> >
> > In summary,
> >
> >     - we have had the notion of full-joining (and I don't include query in
> >     joining) the broadcast GID.
> >
> >     - now it has been proposed that one may not join the broadcast-group
> >     and get the attributes using a query or may join but not full join.
> >
> >     - the IPoIB link is defined by creating a broadcast-GID. The
> >     broadcast-GID is the source of the link's attributes.
> >
> >     - if there is no broadcast GID then there is no link.
> >
> > Now, given the above we have the follwoing suggestion:
> >
> > A) Broadcast-GID defines the link. It MUST not be deleted/modified if the
> > IPoIB subnet is in use.
> >
> > B) Nodes always derive link attributes from the broadcast-GID. If a node
> > discovers that the broadcast-GID is no longer in existence it MUST declare
> > the interface 'down'. If the node discovers that the link-attributes are
> > different than the ones that it had discovered earlier then it SHOULD
> > reconfigure itself with the new attributes. A node may detect the
> > existence/disappearance of the broadcast-GID by using traps/reports or the
> > node might choose to full-member join the broadcast-GID.
> >
> >
> > <snip..>
> > > >
> > > > I believe that we all agree that the broadcast-GID defines the link. If
> > > > there is no broadcast-GID there is no link.
> > > >
> > > > The question to look into is: If there is no link then shouldn't 
> the hosts
> > > > react by reflecting this condition to their stacks? What should the
> > > > reaction be?
> > >
> > > Two things here: in an ethernet fabric, if the link between two 
> switches in
> > > the fabric break in a way that broadcast traffic from one node can 
> not reach
> > > some others (maybe causing segregated nodes as discussed previously), 
> do the
> > > end node stacks get told about this? In fact, if the ethernet link is 
> completely
> > > broken, maybe by pulling out all the wires, do any of the stacks get told
> > > about this? Of course, ethernet does not provide a way to detect this 
> condition
> >
> >
> > By the word 'down' above (and we can define the word more or substitute
> > another word) I do mean that the packets are no longer transmitted or
> > received. Same as the case of cables pulled you mention. Depending on the
> > implementation 'telling the stack', can mean various things -- I just mean
> > that it has an effect such that no packets can get transmitted or
> > received.
> >
> > > possibly, whereas IB does (via the GID deletion trap), so it might make
> > > sense to decide whether IB should do something that ethernet does not.
> >
> > We have done IB specific things -- such as a more elaborate multicast
> > mapping with indications for the protocol being mapped.
> >
> > >
> > > Also, I think we should be clear: I believe link conditions should not be
> > > reflected to the IP stack (aren't we mixing L2 and L3 if we do 
> that?), but
> > > as you mention at other places, the L2 driver can stop tx/rx of 
> packets. But
> > > definitely the driver can not (for example) "inform" the stack that some
> > > IP multicast membership that was previously granted has been revoked;
> > > although it can silently drop those mcg packets.
> >
> >
> > That is something I've said before -- no link implies cannot send packets.
> > That is even if the IB cable is connected and possibly one could transmit
> > unicast/mcast packets to peers, but the broadcast-GID is gone, then: do
> > not transmit/receive packets. I agree with that. 'Inform' stack as you
> > mention above is not what I had in mind.
> >
> >
> >
> > >
> > > >
> > > > >
> > > > > If you are trying to solve the issue of how to change link 
> parameters safely
> > > > > in a racefree manner, please list the steps that the admin/hosts 
> have to do;
> > > > > I don't believe JOIN or traps solve all aspects of the problem, 
> unless the
> > > > > admin iterates thru all the groups; contrary to what you are 
> stating above.
> > > >
> > > > I think we are looking at the issue differently. One way is to say that
> > > > the MGIDs have to be looked at by the admin. The other way is to 
> say that
> > > > if the broadcast-GID is removed, the host brings the interfaces 
> down. All
> > > > packet transfer automatically stops. The other is to also imply 
> that since
> > > > there is no broadcast-GID the derived MGIDs do not make any sense 
> any more
> > > > and hence the hosts need to purge them. The admin doesn't do 
> anything in
> > > > this case -- I believe this has been Vandana's point too.
> > >
> > > So, are you saying that the hosts MUST drop IBA memberships to other
> > > groups too when it receives the broadcast-GID deleted trap? This was
> > > what I hinted in my previous mail.  Are you suggesting nodes that
> > > JOINed the broadcast GID also have to do this (otherwise, how would
> > > you do any recovery actions on erroneous deletions of the bcast-GID?) 
> I don't
> > > believe Vandana mentioned this, I would like to hear her comment 
> specifically
> > > if its okay to put more requirements for nodes that do JOIN.
> >
> > I was, probably not very clearly, giving options: if the link is being 
> downed
> > then the rules that can help are -
> >
> > one way -->
> >     Admin must look up the related MGIDs and nodes and down/delete them
> >
> > 2nd way -->
> >     if no broadcast-GID then the nodes stop transmission/reception.
> >
> > 3rd way (or extension to 2nd way) -->
> >
> >     if no broadcast-GID should we also drop all MGIDs. If this was done
> > then the admin doesn't have to manually or through some other means have
> > to get the nodes to leave the membership.
> >
> > The MUST clause in (B) above is the '2nd way'.
> >
> > >
> > > As I mentioned, if you believe a certain handling of bcast-GID
> > > deleted trap will help solve some problem(s), (when a node JOINs
> > > or doesn't JOIN the broadcast GID) please list the problem(s) and how
> > > using the traps help, specifically by listing the steps the admin
> > > and hosts must take. I have tried to portray how it doesn't help, at
> > > least for the problem(s) we have discussed.
> > >
> > > On the other hand, if you believe trap usage requirement is due to
> > > some different view point, or some architecture or cleanliness issue,
> > > and really does not contribute to solving or creating any more problems,
> > > please state that. I would be very disinclined to putting more 
> requirements
> > > based on this argument, but at least I (and maybe others) will clearly
> > > understand the basis of the requirement.
> >
> > The above is not clear to me -- the traps/reports are suggested in (B)
> > above. The suggestion is to track the disappearance of broadcast-GID so
> > that the interface may be 'downed' (we can define downed suitably).
> >
> >
> > >
> > > >
> > > >  >
> > > > > If you are trying to solve the problem of how to track all IP 
> nodes on the
> > > > > fabric, thats a different problem, and yes, I agree that requring 
> all nodes
> > > > > to JOIN will give you an instantaneous snapshot. I don't know 
> though how this
> > > > > information can be used meaningfully.
> > > >
> > > > At the begining of the this thread the issues with broadcast-GID
> > > > removal/modification were listed. Those issues are adequately 
> addressed by
> > >
> > > I disagree; I don't believe having nodes EITHER join bcast-GID OR track
> > > bcast-GID solves any problems for the host or the fabric; specially if
> > > I don't see the rules of what a host that subscribes to the trap must do
> > > (and secondarily, how those actions might help the host itself or the
> > > fabric administrator).
> >
> > The host (in the driver or anywhere) must stop transmission or reception
> > of packets on the IPoIB interface(s) associated with that broadcast-GID.
> > That is the MUST clause in (B).
> >
> >
> > >
> > > > using both (A) and (B). (B) appears to be simple enough to implement --
> > > > just a report; same as the reports used with other MGIDs.
> > >
> > > I am not sure what you mean by "same as reports used with other MGIDs".
> >
> >
> > I mean implmenting reception of reports for broadcast-GID is not difficult
> > afterall the reports are probably requestd for MGIDs to which one might
> > joing send-only or non-member or 'create MGID' events.
> >
> > > I thought we are discussing what the host SHOULD/MUST do when it 
> discovers
> > > the bcast-GID deleted erroneously or recreated.
> > >
> > > >
> > > >  > > > > >
> >
> > <snip...>
> >
> >
> > > > > But to what good though? If the MGID gets deleted, how does it 
> matter how
> > > > > the hosts behave? We agree that the MGID getting deleted when 
> hosts are
> > > > > active is erroneous; assuring the admin that the end nodes will 
> "do the
> > > > > right thing" (declaring link down et al) is dangerous; because he 
> might
> > > > > turn around and recreate the MGID with new properties, and face 
> all the race
> > > > > problems with other existing IP multicast groups still using old 
> properties.
> > > >
> > > > When the link is down -- it follows that no packets leave or are 
> received
> > > > on the interface. So the existing MGIDs cannot be used either.
> > >
> > > So, do the nodes LEAVE these other IP groups (and if so, how many seconds
> > > does the admin wait to make sure all nodes have done the LEAVEs and thus
> > > all old groups are gone, before he can safely recreate)?
> > >
> > > Or should the admin just delete these other groups? I think this is what
> > > you were trying to prevent in the first place, ie having the admin 
> iterate
> > > over the groups.
> >
> > IP layer is not affected by the MGIDs going away.
> >
> >
> > <snip..>
> >
> > > >
> > > > What if one views it as - no broadcast-GID is same as no link. In 
> IPoE it
> > > > could be cable pulled or a downed switch. It atleast means that the
> > > > interface is down.
> > >
> > > The distinction is that IP still sends packets to the driver, the 
> driver might
> > > not send it out (or the transmit fails). IP's memberships to groups 
> are not
> > > affected. The ipoib driver can drop IBA memberships without telling 
> IP, but
> > > are you suggesting that?
> >
> > OK, the crux of the matter...
> >
> >
> > Yes, by 'down' the interface I meant that the link is down. Whether the
> > down indication causes the IP interface to be marked down or whether the
> > driver just queues packets/drops packets is upto the implementation.
> >
> > IP's membership is not affected, that is correct. That is certainly not
> > what I suggested.
> >
> > I did ask if we could, because all MGIDs are derived from broadcast-GID
> > and if the broadcast-GID is down drop the MGID membership? Yes, that
> > amounts to dropping IBA membership without telling IP.
> >
> > >
> > > To conclude, I still can not see that option B is required; I will 
> refrain
> > > from reiterating that opinion, unless I see more logic to change it.
> >
> > Well, reading the rest of the discussion above I was getting the
> > impression that you are not exactly disagreeing.....
> >
> >
> > Vivek
> >
> > >
> > > Thanks.
> > >
> > > Kanoj
> > >
> > > >
> > > > Modification of broadcast-GID is same as moving the NIC to a different
> > > > ethernet segment -- things might not work any more.
> > > >
> > > > These are the issues covered by (B).
> > > >
> > > > Vivek
> > > >
> > > > > Thanks.
> > > > >
> > > > > Kanoj
> > > > >
> > > > >
> > > > > > Note: read 'down' above as 'no link'.
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > >>
> > > > > > > >> Similarly link attributes shouldn't be modified without 
> some coordination
> > > > > > > >> between fabric mgmt and IP software and that coordination 
> shouldn't be part
> > > > > > > >> of IPoIB spec just like IP over Ethernet doesn't talk 
> about how to cope
> > > > > > > >> with VLAN reconfiguration or changes in VLAN properties.
> > > > > > > >
> > > > > > > >In Ethernet we also do not talk about the link-broadcast address
> > > > > > > >determining the existence of the link - in IPoIB it is the basic
> > > > > > > >requirement. However, ethernet nodes to respond to a downed 
> link. In IPoIB
> > > > > > > >the disappearance of a broadcast-GID is equivalent to a 
> downed link: no
> > > > > > >
> > > > > > > I guess this is the main difference between our view points. 
> You have
> > > > > > > the notion of a "downed link" while in my mind the broadcast 
> group is
> > > > > > > an integral part of the IPoIB link (because it defines the 
> link attributes
> > > > > > > such as MTU, Q_Key...). Without the braodcast group, we don't 
> have a link
> > > > > > > (not ever a "downed" one).
> > > > > >
> > > > > > No, that is not the difference. We are talking the same thing 
> except for the
> > > > > > use of different words and more importantly the reaction to the 
> possibility
> > > > > > of the broadcast-GID disappearing.
> > > > > >
> > > > > > I used the words 'downed link' to imply the link is gone(no 
> link since no
> > > > > > broadcast-GID) from the point of view of the IPoIB interface. 
> Therefore since
> > > > > > the link is non-existent it is illegal for the interfaces to 
> continue to be
> > > > > > functioning. You prefer that we make it illegal for the link to
> > > > > > go away  by mandating that the link always be administratively 
> kept up/in
> > > > > > existence.
> > > > > >
> > > > > > I actually think that we need both as addressed in (A) and (B) 
> above.
> > > > > >
> > > > > > Vivek
> > > > > >
> > > > > > >
> > > > > > > Jerry
> > > > > > >
> > > > > > > >one can broadcast and no one new can join. It is a downed 
> link for
> > > > > > > >implementations that do not actively join the broadcast-GID 
> also, since we
> > > > > > > >require that all implementations derive their attributes 
> from the
> > > > > > > >broadcast-GID.
> > > > > > > >
> > > > > > > >Vivek
> > > > > > > >
> > > > > > > >> So my preference remains to be a).
> > > > > > > >>
> > > > > > > >> Jerry
> > > > > > > >>
> > > > > > > >> >-Vandana
> > > > > > > >> >
> > > > > > > >> >>We can avoid going into the details in the specification 
> since the details
> > > > > > > >> >>listed above are largely informational. However, I 
> believe that the case
> > > > > > > >> >>of 'no-broadcast GID' needs to be specified. Is it not 
> similar to the case
> > > > > > > >> >>of a downed interface?
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>My preference is for (c).
> > > > > > > >> >>
> > > > > > > >> >>Vivek
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >>Vivek
> > > > > > > >> >>
> > > > > > > >> >> >
> > > > > > > >> >> > Jerry
> > > > > > > >> >> >
> > > > > > > >> >> > >
> > > > > > > >> >> > >-Vandana
> > > > > > > >> >> > >
> > > > > > > >> >> > >
> > > > > > > >> >> > >_______________________________________________
> > > > > > > >> >> > >IPoverIB mailing list
> > > > > > > >> >> > >[email protected]
> > > > > > > >> >> > >https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > _______________________________________________
> > > > > > > >> >> > IPoverIB mailing list
> > > > > > > >> >> > [email protected]
> > > > > > > >> >> > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >> >>__
> > > > > > > >> >>
> > > > > > > >> >>Vivek Kashyap
> > > > > > > >> >>Linux Technology Center, IBM
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> _______________________________________________
> > > > > > > >> IPoverIB mailing list
> > > > > > > >> [email protected]
> > > > > > > >> https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > > >__
> > > > > > > >
> > > > > > > >Vivek Kashyap
> > > > > > > >Linux Technology Center, IBM
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > IPoverIB mailing list
> > > > > > > [email protected]
> > > > > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > __
> > > > > >
> > > > > > Vivek Kashyap
> > > > > > Linux Technology Center, IBM
> > > > > >
> > > > > > _______________________________________________
> > > > > > IPoverIB mailing list
> > > > > > [email protected]
> > > > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > >
> > > > > _______________________________________________
> > > > > IPoverIB mailing list
> > > > > [email protected]
> > > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > >
> > > > >
> > > >
> > > > __
> > > >
> > > > Vivek Kashyap
> > > > Linux Technology Center, IBM
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > [email protected]
> > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > [email protected]
> > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> >
> > __
> >
> > Vivek Kashyap
> > Linux Technology Center, IBM
> >
> >
> > _______________________________________________
> > IPoverIB mailing list
> > [email protected]
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> >
>
>__
>
>Vivek Kashyap
>Linux Technology Center, IBM