Re: draft-ietf-ipoib-link-multicast-03.txt

> >> 10.0 IP Multicast Routing
> >>
> >>    IP multicast routing requires multicast routers to receive a copy of
> >>    every link multicast packet on a locally connected link [IPMULT,
> >>    IP6MLD].  For Ethernet this is usually achieved by turning on the
> >>    promiscuous multicast mode on a locally connected Ethernet interface.
> >>
> >>    IBA does not provide any hardware support for promiscuous multicast
> >>    mode.  Fortunately a promiscuous multicast mode can be emulated in
> >>    the software running on a router through the following steps.
> >>
> >>    A) Obtain a list of all active IB multicast groups from the local SA.
> >>
> >>    B) Make a "NonMember" join request to the SA for every group that has
> >>    a signature in its MGID matching the one for either IPv4 or IPv6.
> >>
> >>    C) Subscribe to the IB multicast group creation events using a
> >>    wildcarded MGID so that the router can "NonMember" join all IB
> >>    multicast groups created subsequently for IPv4 or IPv6.
> >>
> >
> >Looking at step C) in detail, the IP multicast router must make sure
> >to attach the QP representing the IPoIB interface to the newly created
> >multicast group which triggered the creation trap. Thus, there are two
> >parts to handling the creation trap; the first is to NonMember join
> >the group, and the second is to attach the QPN to the group. The point
> >to note is that either the join or the attach might fail, depending
> >on characterstics of the switches and the hca involved. The attach is
> >more probable to fail due to multicast related hca limits specified in
> >section 11.2.1.2 in IBA. If the join/attach fails beyond a certain
> >limit, IP multicast routing might not work well beyond that.
> >
> >To get around this problem, one solution I can think of is to make
> >all the nodes on the link aware that the hca running the IP mrouted
> >or the fabric has reached its multicast limits, thus all multicast
> >traffic should now use the v4 broadcast or the v6 all-node group.
> >If the individual nodes degrade to using link level broadcast
> >to achieve multicasting, IP mrouted should be able to receive all
> >packets again then. If we adopt this solution, we also need to
> >standardize the mechanism by which the router will indicate this
> >to the nodes on the link. Section 9 already requires the nodes
> >to subscribe to group creation/deletion traps, thus IP mrouted
> >might indicate the multicast-limit condition by the creation of
> >a special multicast group (given the 28 bit IPv4 multicast group
> >id, there are a lot of IBA multicast groups that will never be
> >used for traffic; pick one of them, eg FF12:401B:PKEY:8000:0:0:0:0).
> >The router can then delete the group (at an implementation specific
> >time), when it can ensure that it will be able to receive all the
> >multicast traffic without requiring the nodes to degrade to using
> >broadcast.
> 
> IBA spec. already contains a number of error conditions where a
> management function can fail, e.g., a FullMember/NonMember/SendOnlyNonMember
> join can fail, attaching a QP to a MC group can fail..., etc. These
> failure conditions either rarely happen, or are highly implementation
> dependent.
> 
> Can the failure conditions related to the availability of the local
> resources be treated as simply implementation limitations, or must we
> address these vulnerability by adding more complexity to IPoIB protocol
> to make multicast routing bullet proof?
> 
> Jerry
> 
> >
> >Thanks.
> >
> >Kanoj


Jerry,

Note that this is not solely an IP multicast routing issue, although 
that is the most thorny case (whose failures are hardest to deal with).

The join/attach has to be done at

1. device/protocol initialization time, and on most systems, a failure
at this point is easily communicated to the sysadmin (via ifconfig 
failing etc). 
2. IP-join time, ie when a process on a host joins a IP multicast
group, and again, the failure at this point is probably communicable
to the process synchronously.
3. IP multicast sender who has not joined the IP group (case B in
section 9.0); failure at this point may or may not be communicable
to writing process (system dependent).
4. Multicast router trying to join/attach, in which case the failure
is not communicable to mrouted (since he has made the system call 
to request receipt of all multicast packets in the past and the
call succeeded).

Now, it seems to me that an hca might well have a limit in the
number of mcg's it supports in hardware, and once that limit is
reached, all the above cases will fail for new groups. Contrast
that with an ethernet fabric (which in legacy mode floods multicast
packets to all nodes) and nics (most allow promiscuous mode
to be set by software when the number of multicast groups reaches 
a limit).

To keep IP multicasting working, an easy way might be to use mcg 
create/delete traps to signal to all members to use the broadcast 
gid for transmitting multicast packets. Any node that determines it 
has reached the limit can then FullMember join the special mcg (with
same scope and attributes as the all-nodes group), and leave it 
when it can again optimally support the required number of groups 
after some of its processes terminate or stop using IP multicasting. 
The mcg traps will thus be clear indications to all subscribing nodes
whether any member requires the broadcast-fallback handling.

Specially formatted broadcast packets to signal these conditions
can be an alternative, but firstly the packets themselves can get
lost due to UD nature, and secondly will require all hosts to 
reference count how many members request broadcast-fallback handling.

Of course, if the typical case is that join/attach will not fail
(ie the number of mcg's are not too great), we can probably punt
on this, but at least with a note that a limit on the number of
IP multicast groups exists depending on member hca's and fabric
(switch/SM) capability.

Thanks.

Kanoj