Re: IPoIB link address

Kanoj,

See below in <VK> --  with the corrected (a typo) order of preference :).

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email protected]
[email protected]
Ph: 503 578 3422 T/L: 775 3422



|---------+---------------------------->
|         |           Kanoj Sarcar     |
|         |           <Kanoj.Sarcar@Sun|
|         |           .COM>            |
|         |           Sent by:         |
|         |           Kanoj.Sarcar@Sun.|
|         |           COM              |
|         |                            |
|         |                            |
|         |           09/25/03 13:49   |
|         |           Please respond to|
|         |           Kanoj.Sarcar     |
|         |                            |
|---------+---------------------------->

>-------------------------------------------------------------------------------------------------------------------|

  |
|
  |       To:       Vivek Kashyap/Beaverton/IBM@IBMUS
|
  |       cc:       Vandana Rao <[email protected]>, [email protected]
|
  |       Subject:  Re: [Ipoverib] IPoIB link address
|
  |
|

>-------------------------------------------------------------------------------------------------------------------|





Vivek Kashyap wrote:
>
> Kanoj/Vandana,
>
>         Could we say that it is upto the implementation on how the
>         attributes are received? If there is a query attributes call as
>         Kanoj says then it could be used - my concern is that the
>         broadcast-GID cannot be deleted if someone is using it. It can be
>         if no one is -- that is one of the reasons for joining it.
>
>         I'll attempt a summation of the changes. Please comment:

Hi,

>
>         1. Define a broadcast-GID defined as follows (one of - please
>         comment):
>
>         a) FF1x:401B::255.255.255.255
>         b) FF1x:001B::255.255.255.255
>         c) FF1x:0000::255.255.255.255
>         d) FF1x:401B::1
>
>     My preference is b) or c) since it removes the IP version
>     completely from the mapping. a) is what we have today and requires
>     no change and so could be used. d) is derived from the multicast
>     mapping and so I'd like to leave it separate.

I would go for a) since it is already there; if we do change, I would
go for d, but can live with b or c.

<VK> OK. I find (a), (b) and (c) acceptable. My order of preference (c),
then (b) then (a). No (d). <VK>
>
>         2. The multicast mappings stay as currently specified.
>
>         3. The broadcast-GID (one of)
>
>         a) MUST be FullMember IB_joined
>
>         b) MUST be used to derive the attributes such as the MTU, Q_Key.
>         The attributes MAY be obtained by FullMember IB_Join of the
>         broadcast-GID or by some other operation supported by IBA. The
>         node MUST fail all IP operations on the interface if the
>         Broadcast-GID ceases to exist since that will correspond to a
>         downed link. It is upto the implementation to track the existence
>         fo the Broadcast-GID in a suitable way. Note: If the
broadcast-GID
>         is FullMember IB_joined then its existence/deletion is
>         automatically tracked.

I would go with option b, but without the requirement "MUST fail all IP
operations ..." for multiple reasons; firstly, it requires at least a
memory check in the send/receive path, and secondly, if other L2's
(ethernet)
aren't requring this, why should IB? (Seems like mingling an L2 issue with
L3,
the same logic that was used against seperate groups).

<VK> On further thinking tracking with a requirement to re-acquire modified
attributes is needed. 'must fail' is not needed though for reasons below
(analogous to why we want to track other multicast GIDs):

Assume that the broadcast-GID was not FullMember joined and so ends up
getting deleted. There are 3 possibilities (not counting a recreate with
the same attributes and MLID):
a) Broadcast-GID is not recreated.
b) Broadcast-GID is recreated with different attributes but same MLID
c) Broadcast-GID is recreated with different/same attributes but different
MLID


Now, if (a) occurs - the nodes still have agreed upon MTU, Q_Key etc. and
can continue to communicate with peers. No new nodes can join the link
though. No broadcasts can occur if the MLID was removed from the switch
tables => no ARP. One  could let it all go on till nodes drop off(ARP
timeouts for example) or declare interface is down.

Now, if (b) occurs the link's attributes have changed and communication
with nodes that were on the link before, and are still using the old
information, can occur however communication with other nodes using new
attributes on the link may fail. Broadcast however will probably succeed
with some set of nodes (dependent on how the fabric is setup for this
MLID).

if (c) occurs then broadcast with nodes that were part of the link before,
and are still using the old information, cannot occur (don't know the new
MLID) though the node can still communicate with the nodes using older
attributes.

Therefore, the requriement really needs to be:
      - track broadcast-GID
      - reacquire attributes if broadcast-GID is recreated
      - (possibly) if no broadcast-GID down the interface

Thoughts?, especially for the case where the broadcast-GID is deleted and
not brought back.
<VK>


Also, the Note: seems informational, but it does not mention that an
implementation can subscribe to group creation/deletion traps to become
aware of the broadcast-GID coming and going.

<VK> One clarification..the text was only for this discussion and not the
drafts. The text in the drafts will be updated after agreeing on the
solution. Yes, joining the GID or tracking using trap/report are the two
obvious methods <VK>

Thanks.

Kanoj

>
> On Thu, 25 Sep 2003, Kanoj Sarcar wrote:
>
> > Vandana Rao wrote:
> > >
> > > At 04:02 PM 9/24/2003 -0700, Kanoj Sarcar wrote:
> > >
> > > <snip>
> > >
> > > >I think we have already decided on the other solution, but maybe I
can
> > > >clarify my comment here.
> > > >
> > > >Notice that whatever the IP layer does, finally device creation,
> > > >and/or possibly IP configuration, is dependent on an L2 property for
> > > >IPoIB, namely the existance of broadcast group(s). I am not
suggesting
> > > >everyone's implementation should error out V4 and/or V6
configuration
> > > >depending on the group's existance, but pointing out that is a
possibility
> > > >for certain implementations (in lieu of a central administrative
tool
> > > >that can control v4/v6 plumbing on a node basis).
> > > >
> > > >Why I think it might be more important to control v4 and/or v6
protocol
> > > >plumbing in an IBA fabric (vs ethernet) is that each v4 and v6
configuration
> > > >on a node adds routing overhead to the fabric for the various
multicast
> > > >groups. This is absent in ethernet (or at least the switches'
spanning
> > > >trees know how to gracefully handle too many multicast groups).
Section
> > > >11 of the multicast draft discusses some of these limitations.
> > >
> >
> > Hi,
> >
> > > The issue with IB multicast resource limitation has nothing to do
with
> > > allowing v4 and/or v6 configuration.
> > > It has to do with using too many  (v4 or v6) IP multicast groups. I
do not
> > > understand what you mean by each v4/v6 configuration requiring
additional
> > > routing overhead to the fabric for the various multicast groups. I do
not
> > > believe configuring a new IP address will add any extra overhead to
the
> > > fabric for multicast groups.
> >
> > This is academic at this point, but when v4 or v6 is configured on a
device,
> > normally IP tries to JOIN to a few multicast groups (224.0.0.1 et al).
These
> > map to IBA JOINs, which are messages to the SM to arrange the switching
tables
> > in all the IBA switches so that the multicast packets are appropriately
> > forwarded. Thats fabric overhead, and also consumes switching resources
at
> > the IBA switches.
>
> Overloading of MLIDs can mitigate some of it.
>
> >
> > >
> > > > >
> > > > > >
> > > > > > Wrt a single QP supporting both v4/v6, two broadcast groups
with the same
> > > > > > attribute set allow that too.
> > > > > >
> > > > > > >
> > > > > > >> > There is also the issue of broadcast traffic isolation
that has been
> > > > > > >> > pointed out; any port which has a only a single qpn
accepting v6
> > > > traffic
> > > > > > >> > currently does not need to receive broadcast traffic,
saving
> > > > both fabric
> > > > > > >> > resources and host interruption.
> > > > > > >>
> > > > > > >>The v6 qpn need not join the broadcast-GID if option 2) below
is
> > > > chosen. The
> > > > > > >>driver can, on startup, join the broadcast-GID on the qpn
> > > > associated with v4
> > > > > > >>broadcasts. The attributes can then be borrowed for the v6
qpn
> > > > without it
> > > > > > >>having to actually join the broadcast-GID. The isolation can
still be
> > > > > > >>maintained in those implementations that prefer it.
> > > > > > >
> > > > > > >The expectation was that this broadcast GID was primarily
"management"
> > > > > > >focused (e.g. ARP / ND) thus the overhead / interrupt rate /
etc. is
> > > > > > >expected to be relatively low.  If we all agree that the
separation does
> > > > > > >not impact interoperability, then it is a local issue.
> > > > > >
> > > > > > Actually the broadcast traffic is not rare either, not until we
move
> > > > > > all the apps off broadcast to SLP/DHCP or well-known multicast.
> > > > > > IPv6 has got it right by basing ND on multicast addresses
instead of
> > > > > > the broadcast address.
> > > > >
> > > > > Yes, however, as I noted earlier an implementation can always
join the
> > > > > broadcast-GID on one QP and then borrow its attributes for v6
> > > > > multicast related QPs; it does not have to join the broadcast-GID
> > > > > itself. Therefore an implementation can choose to keep the
traffic
> > > > > separate.
> > > >
> > > >Agreed. By extension, I think we should be careful not to require a
> > > >node to JOIN the combined broadcast group if it can determine that
> > > >only v6 will be used on the device, and leave that as an
implementation
> > > >decision.
> > >
> > > Again, could you clarify what you mean here? The node HAS to join the
> > > combined broadcast group
> > > if it has to support either or both protocols. In the example given
above,
> > > the v4 QP  (if different for v4 and v6) does not have to attach to
the
> > > broadcast MGID.
> >
> > If a node only ever intends to support v6 traffic, it might have no
reason
> > to join the combined broadcast group; rather it can just QUERY the
attributes
> > for the combined group, and then just JOIN the appropriate V6 multicast
> > groups as instructed by IP.
> >
> > Thanks.
> >
> > Kanoj
> >
> > > -Vandana
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > [email protected]
> > > https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> > _______________________________________________
> > IPoverIB mailing list
> > [email protected]
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> >
> >
>
> _______________________________________________
> IPoverIB mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/ipoverib





_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib