RE: IPoIB link address

Well this seems to change the concensus I was leaning toward last
week... ( part ii, part i is still correct )

Ok, baring anyone expressing bad things happening stating that the IPoIB
interface MUST full member join the broadcast GID - that will be
concensus Friday.

Speak up now - or forever hold your peace

Bill

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Vivek Kashyap
Sent: Monday, October 06, 2003 10:24 PM
To: Kanoj Sarcar
Cc: H.K. Jerry Chu; [email protected]; [email protected]
Subject: Re: [Ipoverib] IPoIB link address


Kanoj and I discussed the issues raised in this thread offline. After
some discussion we came to the conclusion that it is better to have the
nodes always full-join the broadcast-GID and not query or
non/sendonlynon member join the broadcast GID. If that is done then the
implementations are much simpler and cleaner.


Therefore, we suggest that the proposal of 'one may receive the
broadcast-group attributes using a query/other join modes' be dropped.
Note this does not mean that every QPN (if multiple are used) on a given
port MUST join the broadcast-GID but that at least one of them MUST
join.


I'd like to suggest the following two as the conclusion of this thread.

i) There is a single broadcast-GID associated with any IPoIB link. This
broadcast-GID is -> FF1x:401B:<P_Key>::255.255.255.255.

ii) The broadcast-GID MUST be full-member joined by the IPoIB interface.


Thanks
	Vivek



On Sat, 4 Oct 2003, Vivek Kashyap wrote:

> Kanoj,
>
> In summary,
>
>     - we have had the notion of full-joining (and I don't include
query in
>     joining) the broadcast GID.
>
>     - now it has been proposed that one may not join the
broadcast-group
>     and get the attributes using a query or may join but not full 
> join.
>
>     - the IPoIB link is defined by creating a broadcast-GID. The
>     broadcast-GID is the source of the link's attributes.
>
>     - if there is no broadcast GID then there is no link.
>
> Now, given the above we have the follwoing suggestion:
>
> A) Broadcast-GID defines the link. It MUST not be deleted/modified if 
> the IPoIB subnet is in use.
>
> B) Nodes always derive link attributes from the broadcast-GID. If a 
> node discovers that the broadcast-GID is no longer in existence it 
> MUST declare the interface 'down'. If the node discovers that the 
> link-attributes are different than the ones that it had discovered 
> earlier then it SHOULD reconfigure itself with the new attributes. A 
> node may detect the existence/disappearance of the broadcast-GID by 
> using traps/reports or the node might choose to full-member join the 
> broadcast-GID.
>
>
> <snip..>
> > >
> > > I believe that we all agree that the broadcast-GID defines the 
> > > link. If there is no broadcast-GID there is no link.
> > >
> > > The question to look into is: If there is no link then shouldn't 
> > > the hosts react by reflecting this condition to their stacks? What

> > > should the reaction be?
> >
> > Two things here: in an ethernet fabric, if the link between two 
> > switches in the fabric break in a way that broadcast traffic from 
> > one node can not reach some others (maybe causing segregated nodes 
> > as discussed previously), do the end node stacks get told about 
> > this? In fact, if the ethernet link is completely broken, maybe by 
> > pulling out all the wires, do any of the stacks get told about this?

> > Of course, ethernet does not provide a way to detect this condition
>
>
> By the word 'down' above (and we can define the word more or 
> substitute another word) I do mean that the packets are no longer 
> transmitted or received. Same as the case of cables pulled you 
> mention. Depending on the implementation 'telling the stack', can mean

> various things -- I just mean that it has an effect such that no 
> packets can get transmitted or received.
>
> > possibly, whereas IB does (via the GID deletion trap), so it might 
> > make sense to decide whether IB should do something that ethernet 
> > does not.
>
> We have done IB specific things -- such as a more elaborate multicast 
> mapping with indications for the protocol being mapped.
>
> >
> > Also, I think we should be clear: I believe link conditions should 
> > not be reflected to the IP stack (aren't we mixing L2 and L3 if we 
> > do that?), but as you mention at other places, the L2 driver can 
> > stop tx/rx of packets. But definitely the driver can not (for 
> > example) "inform" the stack that some IP multicast membership that 
> > was previously granted has been revoked; although it can silently 
> > drop those mcg packets.
>
>
> That is something I've said before -- no link implies cannot send 
> packets. That is even if the IB cable is connected and possibly one 
> could transmit unicast/mcast packets to peers, but the broadcast-GID 
> is gone, then: do not transmit/receive packets. I agree with that. 
> 'Inform' stack as you mention above is not what I had in mind.
>
>
>
> >
> > >
> > > >
> > > > If you are trying to solve the issue of how to change link 
> > > > parameters safely in a racefree manner, please list the steps 
> > > > that the admin/hosts have to do; I don't believe JOIN or traps 
> > > > solve all aspects of the problem, unless the admin iterates thru

> > > > all the groups; contrary to what you are stating above.
> > >
> > > I think we are looking at the issue differently. One way is to say

> > > that the MGIDs have to be looked at by the admin. The other way is

> > > to say that if the broadcast-GID is removed, the host brings the 
> > > interfaces down. All packet transfer automatically stops. The 
> > > other is to also imply that since there is no broadcast-GID the 
> > > derived MGIDs do not make any sense any more and hence the hosts 
> > > need to purge them. The admin doesn't do anything in this case -- 
> > > I believe this has been Vandana's point too.
> >
> > So, are you saying that the hosts MUST drop IBA memberships to other

> > groups too when it receives the broadcast-GID deleted trap? This was

> > what I hinted in my previous mail.  Are you suggesting nodes that 
> > JOINed the broadcast GID also have to do this (otherwise, how would 
> > you do any recovery actions on erroneous deletions of the 
> > bcast-GID?) I don't believe Vandana mentioned this, I would like to 
> > hear her comment specifically if its okay to put more requirements 
> > for nodes that do JOIN.
>
> I was, probably not very clearly, giving options: if the link is being

> downed then the rules that can help are -
>
> one way -->
>     Admin must look up the related MGIDs and nodes and down/delete 
> them
>
> 2nd way -->
>     if no broadcast-GID then the nodes stop transmission/reception.
>
> 3rd way (or extension to 2nd way) -->
>
>     if no broadcast-GID should we also drop all MGIDs. If this was 
> done then the admin doesn't have to manually or through some other 
> means have to get the nodes to leave the membership.
>
> The MUST clause in (B) above is the '2nd way'.
>
> >
> > As I mentioned, if you believe a certain handling of bcast-GID 
> > deleted trap will help solve some problem(s), (when a node JOINs or 
> > doesn't JOIN the broadcast GID) please list the problem(s) and how 
> > using the traps help, specifically by listing the steps the admin 
> > and hosts must take. I have tried to portray how it doesn't help, at

> > least for the problem(s) we have discussed.
> >
> > On the other hand, if you believe trap usage requirement is due to 
> > some different view point, or some architecture or cleanliness 
> > issue, and really does not contribute to solving or creating any 
> > more problems, please state that. I would be very disinclined to 
> > putting more requirements based on this argument, but at least I 
> > (and maybe others) will clearly understand the basis of the 
> > requirement.
>
> The above is not clear to me -- the traps/reports are suggested in (B)

> above. The suggestion is to track the disappearance of broadcast-GID 
> so that the interface may be 'downed' (we can define downed suitably).
>
>
> >
> > >
> > >  >
> > > > If you are trying to solve the problem of how to track all IP 
> > > > nodes on the fabric, thats a different problem, and yes, I agree

> > > > that requring all nodes to JOIN will give you an instantaneous 
> > > > snapshot. I don't know though how this information can be used 
> > > > meaningfully.
> > >
> > > At the begining of the this thread the issues with broadcast-GID 
> > > removal/modification were listed. Those issues are adequately 
> > > addressed by
> >
> > I disagree; I don't believe having nodes EITHER join bcast-GID OR 
> > track bcast-GID solves any problems for the host or the fabric; 
> > specially if I don't see the rules of what a host that subscribes to

> > the trap must do (and secondarily, how those actions might help the 
> > host itself or the fabric administrator).
>
> The host (in the driver or anywhere) must stop transmission or 
> reception of packets on the IPoIB interface(s) associated with that 
> broadcast-GID. That is the MUST clause in (B).
>
>
> >
> > > using both (A) and (B). (B) appears to be simple enough to 
> > > implement -- just a report; same as the reports used with other 
> > > MGIDs.
> >
> > I am not sure what you mean by "same as reports used with other 
> > MGIDs".
>
>
> I mean implmenting reception of reports for broadcast-GID is not 
> difficult afterall the reports are probably requestd for MGIDs to 
> which one might joing send-only or non-member or 'create MGID' events.
>
> > I thought we are discussing what the host SHOULD/MUST do when it 
> > discovers the bcast-GID deleted erroneously or recreated.
> >
> > >
> > >  > > > > >
>
> <snip...>
>
>
> > > > But to what good though? If the MGID gets deleted, how does it 
> > > > matter how the hosts behave? We agree that the MGID getting 
> > > > deleted when hosts are active is erroneous; assuring the admin 
> > > > that the end nodes will "do the right thing" (declaring link 
> > > > down et al) is dangerous; because he might turn around and 
> > > > recreate the MGID with new properties, and face all the race 
> > > > problems with other existing IP multicast groups still using old

> > > > properties.
> > >
> > > When the link is down -- it follows that no packets leave or are 
> > > received on the interface. So the existing MGIDs cannot be used 
> > > either.
> >
> > So, do the nodes LEAVE these other IP groups (and if so, how many 
> > seconds does the admin wait to make sure all nodes have done the 
> > LEAVEs and thus all old groups are gone, before he can safely 
> > recreate)?
> >
> > Or should the admin just delete these other groups? I think this is 
> > what you were trying to prevent in the first place, ie having the 
> > admin iterate over the groups.
>
> IP layer is not affected by the MGIDs going away.
>
>
> <snip..>
>
> > >
> > > What if one views it as - no broadcast-GID is same as no link. In 
> > > IPoE it could be cable pulled or a downed switch. It atleast means

> > > that the interface is down.
> >
> > The distinction is that IP still sends packets to the driver, the 
> > driver might not send it out (or the transmit fails). IP's 
> > memberships to groups are not affected. The ipoib driver can drop 
> > IBA memberships without telling IP, but are you suggesting that?
>
> OK, the crux of the matter...
>
>
> Yes, by 'down' the interface I meant that the link is down. Whether 
> the down indication causes the IP interface to be marked down or 
> whether the driver just queues packets/drops packets is upto the 
> implementation.
>
> IP's membership is not affected, that is correct. That is certainly 
> not what I suggested.
>
> I did ask if we could, because all MGIDs are derived from 
> broadcast-GID and if the broadcast-GID is down drop the MGID 
> membership? Yes, that amounts to dropping IBA membership without 
> telling IP.
>
> >
> > To conclude, I still can not see that option B is required; I will 
> > refrain from reiterating that opinion, unless I see more logic to 
> > change it.
>
> Well, reading the rest of the discussion above I was getting the 
> impression that you are not exactly disagreeing.....
>
>
> Vivek
>
> >
> > Thanks.
> >
> > Kanoj
> >
> > >
> > > Modification of broadcast-GID is same as moving the NIC to a 
> > > different ethernet segment -- things might not work any more.
> > >
> > > These are the issues covered by (B).
> > >
> > > Vivek
> > >
> > > > Thanks.
> > > >
> > > > Kanoj
> > > >
> > > >
> > > > > Note: read 'down' above as 'no link'.
> > > > >
> > > > > >
> > > > > > >
> > > > > > >>
> > > > > > >> Similarly link attributes shouldn't be modified without 
> > > > > > >> some coordination between fabric mgmt and IP software and

> > > > > > >> that coordination shouldn't be part of IPoIB spec just 
> > > > > > >> like IP over Ethernet doesn't talk about how to cope with

> > > > > > >> VLAN reconfiguration or changes in VLAN properties.
> > > > > > >
> > > > > > >In Ethernet we also do not talk about the link-broadcast 
> > > > > > >address determining the existence of the link - in IPoIB it

> > > > > > >is the basic requirement. However, ethernet nodes to 
> > > > > > >respond to a downed link. In IPoIB the disappearance of a 
> > > > > > >broadcast-GID is equivalent to a downed link: no
> > > > > >
> > > > > > I guess this is the main difference between our view points.

> > > > > > You have the notion of a "downed link" while in my mind the 
> > > > > > broadcast group is an integral part of the IPoIB link 
> > > > > > (because it defines the link attributes such as MTU, 
> > > > > > Q_Key...). Without the braodcast group, we don't have a link

> > > > > > (not ever a "downed" one).
> > > > >
> > > > > No, that is not the difference. We are talking the same thing 
> > > > > except for the use of different words and more importantly the

> > > > > reaction to the possibility of the broadcast-GID disappearing.
> > > > >
> > > > > I used the words 'downed link' to imply the link is gone(no 
> > > > > link since no
> > > > > broadcast-GID) from the point of view of the IPoIB interface.
Therefore since
> > > > > the link is non-existent it is illegal for the interfaces to
continue to be
> > > > > functioning. You prefer that we make it illegal for the link
to
> > > > > go away  by mandating that the link always be administratively
kept up/in
> > > > > existence.
> > > > >
> > > > > I actually think that we need both as addressed in (A) and (B)

> > > > > above.
> > > > >
> > > > > Vivek
> > > > >
> > > > > >
> > > > > > Jerry
> > > > > >
> > > > > > >one can broadcast and no one new can join. It is a downed 
> > > > > > >link for implementations that do not actively join the 
> > > > > > >broadcast-GID also, since we require that all 
> > > > > > >implementations derive their attributes from the 
> > > > > > >broadcast-GID.
> > > > > > >
> > > > > > >Vivek
> > > > > > >
> > > > > > >> So my preference remains to be a).
> > > > > > >>
> > > > > > >> Jerry
> > > > > > >>
> > > > > > >> >-Vandana
> > > > > > >> >
> > > > > > >> >>We can avoid going into the details in the 
> > > > > > >> >>specification since the details listed above are 
> > > > > > >> >>largely informational. However, I believe that the case

> > > > > > >> >>of 'no-broadcast GID' needs to be specified. Is it not 
> > > > > > >> >>similar to the case of a downed interface?
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>My preference is for (c).
> > > > > > >> >>
> > > > > > >> >>Vivek
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>Vivek
> > > > > > >> >>
> > > > > > >> >> >
> > > > > > >> >> > Jerry
> > > > > > >> >> >
> > > > > > >> >> > >
> > > > > > >> >> > >-Vandana
> > > > > > >> >> > >
> > > > > > >> >> > >
> > > > > > >> >> > >_______________________________________________
> > > > > > >> >> > >IPoverIB mailing list
> > > > > > >> >> > >[email protected] 
> > > > > > >> >> > >https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > _______________________________________________
> > > > > > >> >> > IPoverIB mailing list
> > > > > > >> >> > [email protected] 
> > > > > > >> >> > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >>__
> > > > > > >> >>
> > > > > > >> >>Vivek Kashyap
> > > > > > >> >>Linux Technology Center, IBM
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> _______________________________________________
> > > > > > >> IPoverIB mailing list
> > > > > > >> [email protected] 
> > > > > > >> https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > >__
> > > > > > >
> > > > > > >Vivek Kashyap
> > > > > > >Linux Technology Center, IBM
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > IPoverIB mailing list
> > > > > > [email protected] 
> > > > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > > > >
> > > > > >
> > > > >
> > > > > __
> > > > >
> > > > > Vivek Kashyap
> > > > > Linux Technology Center, IBM
> > > > >
> > > > > _______________________________________________
> > > > > IPoverIB mailing list
> > > > > [email protected] 
> > > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > > _______________________________________________
> > > > IPoverIB mailing list
> > > > [email protected] 
> > > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > > >
> > > >
> > >
> > > __
> > >
> > > Vivek Kashyap
> > > Linux Technology Center, IBM
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> > _______________________________________________
> > IPoverIB mailing list
> > [email protected] https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> >
>
> __
>
> Vivek Kashyap
> Linux Technology Center, IBM
>
>
> _______________________________________________
> IPoverIB mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/ipoverib
>
>

__

Vivek Kashyap
Linux Technology Center, IBM


_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib