Re: draft-ietf-ipoib-link-multicast-03.txt

Jerry,

The following is in section 4.0:

>    Unreliable multicast is defined by IBA as an optional functionality
>    for channel adaptors (CAs) and switches. In today's IP technology,
>    link multicast has become an indispensable function for better
>    supporting a modern IP network. For this reason, it is required that
>    an IPoIB fabric supports multicast. This includes all the CAs and
>    switches that are part of an IP network.

I would suggest adding in explicit dependencies on the SM. Something like
"This includes all the CAs and switches that are part of an IP network
alongwith multicast capable SMs that provide multicast group creation
and deletion traps." I believe the IBA specs consider the traps optional,
it is better to clearly state the requirement for proper IPoIB operation.

> 
> 10.0 IP Multicast Routing
> 
>    IP multicast routing requires multicast routers to receive a copy of
>    every link multicast packet on a locally connected link [IPMULT,
>    IP6MLD].  For Ethernet this is usually achieved by turning on the
>    promiscuous multicast mode on a locally connected Ethernet interface.
> 
>    IBA does not provide any hardware support for promiscuous multicast
>    mode.  Fortunately a promiscuous multicast mode can be emulated in
>    the software running on a router through the following steps.
> 
>    A) Obtain a list of all active IB multicast groups from the local SA.
> 
>    B) Make a "NonMember" join request to the SA for every group that has
>    a signature in its MGID matching the one for either IPv4 or IPv6.
> 
>    C) Subscribe to the IB multicast group creation events using a
>    wildcarded MGID so that the router can "NonMember" join all IB
>    multicast groups created subsequently for IPv4 or IPv6.
> 

Looking at step C) in detail, the IP multicast router must make sure
to attach the QP representing the IPoIB interface to the newly created
multicast group which triggered the creation trap. Thus, there are two
parts to handling the creation trap; the first is to NonMember join 
the group, and the second is to attach the QPN to the group. The point
to note is that either the join or the attach might fail, depending
on characterstics of the switches and the hca involved. The attach is
more probable to fail due to multicast related hca limits specified in
section 11.2.1.2 in IBA. If the join/attach fails beyond a certain
limit, IP multicast routing might not work well beyond that.

To get around this problem, one solution I can think of is to make
all the nodes on the link aware that the hca running the IP mrouted
or the fabric has reached its multicast limits, thus all multicast
traffic should now use the v4 broadcast or the v6 all-node group.
If the individual nodes degrade to using link level broadcast
to achieve multicasting, IP mrouted should be able to receive all 
packets again then. If we adopt this solution, we also need to 
standardize the mechanism by which the router will indicate this 
to the nodes on the link. Section 9 already requires the nodes
to subscribe to group creation/deletion traps, thus IP mrouted
might indicate the multicast-limit condition by the creation of
a special multicast group (given the 28 bit IPv4 multicast group
id, there are a lot of IBA multicast groups that will never be
used for traffic; pick one of them, eg FF12:401B:PKEY:8000:0:0:0:0).
The router can then delete the group (at an implementation specific
time), when it can ensure that it will be able to receive all the
multicast traffic without requiring the nodes to degrade to using
broadcast.

Thanks.

Kanoj