Re: Comments on IPoIB Connected Mode Connection Establishment

On Wed, 10 Aug 2005, Dror Goldenberg wrote:

> Reviewing the ipoib-cm draft
> (http://www1.ietf.org/mail-archive/web/ipoverib/current/msg01372.html
> <http://www1.ietf.org/mail-archive/web/ipoverib/current/msg01372.html> )
> I have three comments about the connection establishment of IPoIB-CM.
>
> (1) Service ID Reserved Bits
>
> Section 3.3. defines the Service-ID as having reserved bits which must be
> transmitted
> as zeroes and ignored on receive.
> It doesn't sound right to ignore those bits on the receive side. Ignoring
> those bits on
> receive essentially means that the IPoIB-CM has to listen on a range of
> service IDs.
> I believe that it's better to define a service ID without reserved bits such
> that the
> passive side listens on a single service ID.
> For example, service IDs defined for SDP, don't have any reserved bits.

Is it not possible to mask the bits marked as reserved (or unused)? The draft
requires that the reserved bits must be zeroes on send. If it is a limitation 
of the connection manager that it cannot mask the bits and will have to listen
to the 'range' then I agree that the service-ID will have to be specified 
received as zeroes instead of ignoring them.

>
>
> (2) Active/Active Model Support
>
> 2.3 implies that it is desirable to have one connection between peers, i.e.
> one QP
> on each side. I believe that active-active connection establishment model is
> more
> appropriate here. Otherwise, there may be cases of simultaneous access that
> will
> end up having two connections between the same two nodes (one for each
> direction).
> The current Service ID assignment, makes it impossible to use the
> active-active
> model. The main reason is that the service ID has to be identical at both
> ends. In
> other words, active-active requires that both REQ messages be used with the
> same
> Service ID. The current Service ID is different (it has the QP number in
> it).
>
> A model that enables active-active, for example, is using a Service ID which
> is a derivative
> of the PKey (no QP# in the Service ID). However, it will not support the
> model of more than
> one UD QP on the same HCA on the same PKey. This model is important.
>
> Another model that enables active-active is to have both local and remote QP
> numbers
> in the service ID. However, it will require both ends to listen on many
> Service IDs.
>
> Another alternative is to stay with active-passive connection establishment,
> but add
> an interface through which one of the connection can be quiesced and closed.
> In the
> case where you happened to open two connections between the hosts, then you
> can close one of them afterwards.
>
>
> (3) Identification of Remote Peer in Passive Side
>
> It is unclear to me how can the passive side tell which is the remote peer.
> It must be
> able to tell that, so that it can use the other half of the connection to
> send messages
> back to the active side (instead of creating a new connection).
> The identification of the peer is based on Link-Layer Address, which is
> {GID,QPN}. The
> CM REQ message only includes GID. Maybe the QPN (the UD QP) should be passed
>
> through the REQ private data ?

ok..thinking about the above points, how about:

1. Include the local QPN in the request
2. The receiver of CM REQ will have:
 	- local QPN (by way of ServiceID requested)
 	- remote QPN (in private data as above)
 	- remote GID (in REQ)
3. In case of the active-active connection the request from the numerically 
smaller MAC Address (QPN+GID or probably just the GID) is terminated.
The error code 'state connection' is a possibility here.

The active-active situation is known when it is determined that
there is an outstanding request to the remote peer (as determined by 
GID+QPN) while a request from the same is received on local ServiceID.

4. However, there might be a need to establish multiple connections
between the same two peers. Therefore, if a connection already exits
between the remote GID+QPN on the local ServiceID, it is up to the 
receiver to accept this connection. The implementation accepting
multiple connections and the one requesting multiple connections on
the same ServiceID must manage the load; it is beyond the scope of
this spec.

If two connection requests cross then case 3. above applies.

thoughts?

Vivek

>
> -Dror
>