RE: Comments on IPoIB Connected Mode Connection Establ ishment

On Mon, 12 Sep 2005, Michael Krause wrote:

> At 10:23 PM 9/10/2005, Dror Goldenberg wrote:
>
>> Sounds like we agree that active/active is very difficult and will be 
>> really hard
>> to define given how IB service ID and communication manager work. 
>> Therefore,
>> I think that we should go for the active/passive with the ability to reject 
>> at the
>> application (ipoib) level.
>
> The IBTA can develop what is required to fix the SID issues.

Mike, probably you can answer this the best. Doesn't the CM consult the 
relevant service (service ID) during the connection process? That is when
REQ is receive it probably checks with the service to see if it is willing to
accept it. And so on through the handshake. If so, active/active can be done.

thoughts?

>
>
>> For the case of more than one connection between two peers, we first need 
>> to
>> decide whether we want to allow it (now or in the future). If we disallow 
>> it, then
>> it's very simple, for each crossing request we need to decide which side 
>> should
>> accept and which side should reject it (e.g. GID compare).

I beleive we need to allow more than one connection between two peers. It 
seems possible (assuming I'm not missing something obvious :)) to do it if 
the request is verified against the service (in this case the IPoIB driver 
for the relevant serviceID). We need to determine whether QPN+GID compare 
can be used to fail simultaneous or only temporary fail when the 
implementation allows multiple connections.

>
> A multi-connection model just like Ethernet aggregation solutions deployed 
> today enables aggregate performance to be achieved without burdening the 
> upper layers.  Flows are predictable, i.e. a given heuristic generally 
> insures that IP datagrams flow over consistent paths thus eliminating the 
> re-ordering problem.  These heuristics are locally controlled with Ethernet 
> today and do not need to be specified.  As a general comment, people should 
> review what is already defined and deployed today with aggregation technology 
> and see how this might be leveraged for IB.  While Ethernet currently limits 
> aggregation to a single switch attachment, that restriction will be removed 
> in the future thus matching what IB can do with its multi-path support. 
> Given the desire to minimize how much a customer really sees of IB from a 
> stack integration perspective, it would be good to see both technologies 
> sharing as much of the infrastructure as possible.
>
>
>> If we want to allow  more than one connection in the future, then maybe we
>> should think of embedding a "channel index" into the private data. This 
>> way,
>> the two peers can add the channel index on which they are transmitting.
>> BTW, it'll take more than that to enable more than one channel, for 
>> example,
>> we should try to avoid reordering of IP datagrams that can happen when more
>> than one channel is being used to send them.
>> 
>> What about connection termination ? This should probably be added to the 
>> RFC
>> draft. I can think of two cases where one would want to terminate a 
>> connection.
>> 1) When the ipoib driver goes down; 2) Inactive connections can be taken 
>> down
>> to reduce resource consumption.
>
> Termination should only occur when an unplumb operation occurs.  This is when 
> the network stack is effectively taken down.  The cause of the unplumb can be 
> any number of reasons thus it does not matter what the specific reason is 
> only that an admin has decided to unplumb the stack.  Note: An error, loss of 
> cable, etc. are not unplumb operations so should not impact the connection 
> until the IB fabric says communication is no longer possible and errors it 
> out.

Yes, and also the IB connection is created when there is a need to talk to the
remote node. This will reflect in the IP level address resolution structures
which will be controlled by IP requirements (ARP/ND). The implementation can
choose to terminate the connection when the ND/ARP cache is deemded invalid or 
deleted.

>
> Mike
>
>
>> -Dror
>> 
>> > -----Original Message-----
>> > From: Vivek Kashyap 
>> [<mailto:[email protected]>mailto:[email protected]]
>> > Sent: Tuesday, September 06, 2005 10:40 AM
>> > To: Dror Goldenberg
>> > Cc: IPoverIB
>> > Subject: RE: [Ipoverib] Comments on IPoIB Connected Mode
>> > Connection Establ ishment
>> >
>> >
>> > On Sun, 21 Aug 2005, Dror Goldenberg wrote:
>> >
>> > >
>> > >
>> > >> From: Vivek Kashyap 
>> [<mailto:[email protected]>mailto:[email protected]]
>> > >> Sent: Thursday, August 18, 2005 9:36 AM
>> > >>
>> > >> On Wed, 17 Aug 2005, Dror Goldenberg wrote:
>> > >>
>> > >>>
>> > >>>
>> > >>>> From: Vivek Kashyap 
>> [<mailto:[email protected]>mailto:[email protected]]
>> > >>>> Sent: Monday, August 15, 2005 9:41 AM
>> > >>>>
>> > >>>> On Wed, 10 Aug 2005, Dror Goldenberg wrote:
>> > >
>> > > <snip>
>> > >
>> > >>
>> > >>>>> (2) Active/Active Model Support
>> > >>>>>
>> > >
>> > >
>> > > <snip>
>> > >
>> > >>>> 3. In case of the active-active connection the request from the
>> > >>>> numerically smaller MAC Address (QPN+GID or probably just
>> > >> the GID) is
>> > >>>> terminated. The error code 'state connection' is a
>> > >> possibility here.
>> > >>>>
>> > >>>> The active-active situation is known when it is determined
>> > >> that there
>> > >>>> is an outstanding request to the remote peer (as determined by
>> > >>>> GID+QPN) while a request from the same is received on local
>> > >>>> GID+ServiceID.
>> > >>>
>> > >>> In IB, active-active is only determined by equal Service
>> > IDs. See IB
>> > >>> spec 1.2 vol1 12.9.7.1, the Active CM state machine goes to
>> > >>> Peer-Compare state based solely on the Service ID.
>> > Therefore, if you
>> > >>> want to have active-active, then you must
>> > >> ensure that
>> > >>> both service IDs are identical. In the current draft, the
>> > >> Service IDs are
>> > >>> different, as they include a remote QP number.
>> > >>>
>> > >>
>> > >> Yes, if we are determining it based on the CM. However, I was
>> > >> suggesting the above as what the ipoib component will do
>> > when it is
>> > >> informed by CM. It can return a suitable error to CM when it
>> > >> determines that a particular
>> > >> connection shouldn't be allowed. Is that not possible?
>> > >>
>> > >
>> > > That wouldn't be "active/active" per se. But maybe it's
>> > just as good,
>> > > because it just pushes this model to the ipoib layer. In a case of
>> > > crossing connection
>> > > requests one side is supposed to accept, while the other is
>> > supposed to
>> > > reject. The rules for rejection have to be described in the
>> > RFC. Maybe it
>> > > worth describing the state machine that now needs to be implemented
>> > > in ipoib.
>> >
>> > yes, need to look into it in detail. My initial thought is that the
>> > method of determining if there are connections in flight is
>> > based on receiving a request source GID+QPN  while attempting
>> > a connection to the same.
>> >
>> > >
>> > > BTW, what will be the rejection reason ? Seems like the most
>> > > appropriate is 28 - consumer reject.
>> >
>> > seems ok to me.
>> >
>> > >
>> > > Note that it becomes tricky if you want to allow more than one
>> > > connection on the same service ID. So, is there a way you can tell
>> > > whether your peer is trying to establish the 1st or 2nd connection
>> > > while you are trying to set up the 1st one ?
>> >
>> > hmm...good point, unless we take it that any request that
>> > crosses is deemed active-active. Once a connection is
>> > established further requests can be made on the same
>> > Service-ID (or denied depending on the implementation).
>> > >
>> > > I think it'll be best if we can use the standard active-active
>> > > mechanism.
>> >
>> > We need a common Service-ID for it that must be
>> > known/derivable by both the peers. Since masks are not
>> > supported by CMs using both QPNs in the
>> > service-ID is difficult.
>> >
>> >
>> > > However, if we don't manage to find how, then this approach
>> > (rejection
>> > > at the app level) will do the work. However, it will complicate the
>> > > ipoib implementation and will duplicate the active-active
>> > work that is
>> > > already implemented in the cm.
>> > >
>> > > -Dror
>> > >
>> > > <snip>
>> > >
>> > >
>> >
>> > _______________________________________________
>> > IPoverIB mailing list
>> > [email protected] 
>> <https://www1.ietf.org/mailman/listinfo/ipoverib>https://www1.ietf.org/mailman/listinfo/ipoverib 
>> >
>> _______________________________________________
>> IPoverIB mailing list
>> [email protected]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>