Re: Re: M2PA (SIGTRAN) and multi-homing

Andrew,

Andrew Booth wrote:                        (Tue, 12 Dec 2006 16:08:19)
> Hi Brian,
> 
> The thrust of your reply I think is that, since multiple M2PA links use
> the same underlying IP network, you don't get as much redundancy as you
> might expect from multiple M2PA links.  In some situations it can cause
> problems.
> 
> I think in some network topologies there may be advantages to multiple
> M2PA links, though a single multi-homed link may provide similar
> characteristics.  For instance, suppose that you architect you IP
> network into two distinct subnets that use distinct hardware.  Each end
> of the M2PA link has 2 NICs, one on each subnet.  Having at least one
> M2PA link per subnet seems not unreasonable in this case and shouldn't
> be vulnerable to the failure you mentioned.

Well, yes and no.  Although separate underlying networks might appear to
remove some of the independence between link sets, it only does so at the
hardware level and not necessarily at the system level (i.e. when both
links are still pumped through the same SCTP stack that has a single IP
queue that can act as a common bottleneck).  Many operating systems have
issues with regard to internal IP queue bottlenecks particularly when
high speed interfaces are used such as GigE.  To get independence at the
system level require independent priority queueing through the IP
subsystem, a situation that is not obvious to most.

Also, it does not remove the load related coupling that can lead to
cascade failure (other traffic using the same two subnets).  It is less
costly to over-engineer one network than two.

All in all, if multiple subnets are provided, it is always better to use
the subnets for multi-homing with SCTP (and better still CMT) to provide
a single higher grade M2PA link than two lower grade links.

SCTP multi-homing is designed for the characteristics of the underlying
IP network, whereas MTP3 simply was not.  When SCTP multi-homing is
used, SCTP determines which of the subnets is viable and best suited for
traffic (CMT better than traditional) regardless of any shared
bottleneck that might exist.  If multiple link sets are used, MTP3
requires that no shared bottleneck exists anywhere (network or host),
otherwise cascade failure can result.

Even when using a single link in a link-set, if there are multiple link
sets (C and B/D links in the scenario), a shared IP bottleneck within
the host can lead to cascade failure between link-sets.  To avoid
catastrophic failures in such a scenario would require that it can be
guaranteed that an IP bottleneck internal to the host will never occur,
even when bursting buffers.  (In the SS7 failure that I experienced,
there was a 4 second 1 Erlang burst of traffic over C-links milliseconds
before the second STP went down.)

One way to ensure no shared IP system bottleneck is to provide
independent blades for each M2PA link in the STP: but then they are
separate hosts and there is no more than one M2PA link per host.

So, ideally, one M2PA link per host and if multiple subnets are
available, use SCTP multi-homing and CMT to get better grade of service
on the single link.

--brian


> Having more than one M2PA link per subnet probably doesn't help much in
> most cases, since your robustness and throughput are limited by the
> underlying IP layer.  Having some additional links may help reduce the
> effect of head-of-line blocking.  Having some additional links may help
> throughput during SCTP slow-start (if for some reason a burst of traffic
> suddenly appears).  Neither of these considerations is likely to be
> relevant over a good (managed) IP network.  Additional links increase
> the chance of cascading failure and increase overall bandwidth
> (slightly).  The above considerations are largely contradictory.  There
> are tradeoffs in each case.  Over a good IP network none of them are
> likely to be serious.
> 
> Robby, regarding your original question, here are some considerations
> that weren't mentioned yet.
> 
> If you're working over a good IP network, either single or multi-homed
> M2PA links should work well.  The differences that I see are mainly
> operational.
> 
> Consider a network topology with distinct subnets.
> 
> - in the multi-homed case, you get to choose your SCTP primary path. 
> This is potentially useful if one ISP is cheaper.  Depending on your
> SCTP parameters you may also see better throughput and less added
> latency if a path fails, as compared to relying on MTP changeover.
> 
> - in the single-homed case you get MTP alarms for link failures, link
> congestion, etc if one of your IP routes goes down or becomes less
> performant.  This gives you some extra diagnostics for free, especially
> if you don't get alarms for ISP router failures.
> 
> - in the single-homed case you have the option to deactivate a link to
> redirect traffic away from the impaired IP network.
> 
> Comments welcome,
> Andrew
> 
> Brian F. G. Bidulock wrote:
> > Robby,
> >
> > Robby Pocornie wrote:                           (Tue, 12 Dec 2006 15:23:14)
> >   
> >>    Brian,
> >>
> >>
> >>
> >>
> >>
> >>    Refer  to  the  Sigtran scenario's specification (e.g. ETSI EG 202 360
> >>    v1.1.1(2004-12)), chapter 4.3...
> >>
> >>    In case there is only a single M2PA link in the link set, the messages
> >>    formerly  sent out in parallel over different links are now serialized
> >>    over  one  channel,  as  M2PA  makes  no use of the " streams" concept
> >>    provided by SCTP...
> >>     
> >
> > Yes, I see that (thanks for the reference).  However, I do not think
> > that ETSI gave due consideration to the impacts of multiple M2PA links
> > in a link set.  M2PA runs a slightly modified T7 timer to ensure that no
> > message is delayed beyond limits.  If, as is stated, an IAM becomes
> > delayed, if the delay is within T7 no message misses a deadline.  If it
> > is delayed beyond T7, a changeover will occur to another link in the
> > link-set.  However, in this scenario, MTP3 will changeover to a link that
> > follows the same network path as the original link and can be expected
> > to experience the same problems.  This is radically different from
> > traditional SS7, where the links in a link set are independent and are
> > normally engineered for diversity to ensure independence.
> >
> > Let me give you an example of the bad thing we were trying to avoid with
> > the recommendation to use only one link in a link-set.  Take 16 M2PA
> > links in one link set (each using the same network path).
> >
> > On link 1, an IAM gets delayed T7+.  The link fails.  MTP3 starts
> > changeover (time controlled for argument's sake).  The traffic through
> > the network path drops by 1/16th allowing intermediate bottleneck
> > routers to pass messages now without delay.  MTP3 tries to restore link
> > 1, but proving adds additional burst traffic and link 1 fails alignment
> > and goes into lockout.  After T2, MTP3 adds the buffered traffic to
> > link 8.  Because buffered traffic is bursted by MTP3 on link 8, link 8
> > subsequently fails.  The situation is bad already because the original
> > IAM (and all the message behind it) has been delayed T7 + T2.  Going
> > around again, after T2, link 8 adds traffic to link 2 and 9, both of
> > which will fail because the burst traffic again exceeds the handling
> > capacity of the intermediate bottleneck.  The original IAM and all
> > messages behind it are now delay T7 + T2 + T2.  Two more iterations and
> > all 16 links will fail.  The IAM and all messages behind it are now
> > delayed T7 + 4 * T2.  MTP3 will then changeover to the other link-set
> > (after another T2).  The IAM and all messages behind it have been
> > delayed T7 + 5 * T2 and there could now be about 5 times the normal
> > changeover burst traffic pent up.
> >
> > The other link-set, even when it uses a different network path, might not
> > successfully handle the burst traffic which can lead to another another
> > 16 link failures, and then C-links, and then another cascade failure in
> > the associated STP in the STP pair, and then network isolation.  Large
> > transmit buffers and M2PA's extremely large window (maximum RTB size)
> > contribute to such a cascade failure.  SS7 is vulnerable to this
> > whenever link failure is load related.
> >
> > At no point does MTP3 understand that these messages have been delayed
> > long past their useful lifetime.  With a T7 of 1 second and a T2 of 2
> > seconds, the initial IAM has been delayed 11 seconds and messages behind
> > it in the buffers have been delayed up to 12 seconds.  Now, although
> > ISUP T7 (awaiting ACM timeout) is 20-30 seconds, every other timer in
> > ISUP is less than 12 seconds.  After the second link-set failure, there
> > will be 24 seconds delay and after C-link failure, 36 seconds delay
> > (given 16 C-links).
> >
> > Consider that if there was only one M2PA link in the link-set, the
> > initial T7 timer expiry would have resulted in a changeover to the
> > alternate link set restoring traffic within T7 + T2 and only buffering 3
> > seconds worth of traffic.
> >
> > These types of failure scenarios are only possible because MTP3 assumes
> > that the failure of one link in a link-set is completely independent of
> > the failure of another link in the link-set and that the failure of a
> > link is not load related.  That is, that the probability that link 8
> > will fail given than link 1 has failed is the same as the probability
> > that link 8 will fail given that link 1 has not failed.  Also, that the
> > probability that link 8 will fail is independent of the load offered to
> > link 8.  With multiple M2PA links in a link-set, the first condition no
> > longer holds true.  Unfortunately, for M2PA the second condition (load
> > related failure) never holds true.  Therefore, the recommendation as
> > laid out in the RFC is that there only be one M2PA link in a link-set.
> >
> > There are two problems with the ETSI passage that you reference.  First,
> > placing multiple M2PA links in a link-set does not decrease inter-link
> > changeovers as is stated.  As show above, it increases inter-link
> > changeovers which is bad, and, worse, can lead to catastrophic cascade
> > failures.  Second, the messages behind the initial IAM do not get
> > delayed as implied by the passage.  The messages behind the initial IAM
> > will be immediately available in the buffers at the receiving SCTP once
> > the IAM is filled in by Fast Retransmission and will not suffer another
> > network queueing delay (that is, it is not the same as a Q.703 NACK).
> > Providing redundant network paths and using SCTP's multi-homing feature,
> > or CMT (concurrent multi-path transfer) can further reduce delay.
> >
> >   
> >>    Advise  is: To avoid that messages are not delivered (message loss) in
> >>    case  of  retransmission, more than one link using M2PA should be used
> >>    to form a linkset.
> >>     
> >
> > Bad advise.  You are better to use multi-homing over separate network
> > paths and CMT to reduce delay in a single M2PA link in a link-set.
> >
> >   
> >>    That  is  why  I  design  more than 1 M2PA link between two STP's (for
> >>    telecommunication SG, not internet host)
> >>     
> >
> > It is your network, but I would not in mine.
> >
> > BTW, the cascade failure above is not conjecture on my part.  In a
> > previous life I diagnosed just such a cascade failure that resulted
> > simply from introducing delay into SS7 B/D links between STPs and also
> > introducing the characteristic of a load-related link failure.  The
> > cascade failure (that occurred during the Jan 1998 east coast ice storm)
> > resulted in 2 hours of network isolation and forty-million dollars in
> > damages (at least that is what the resulting lawsuit claimed).
> >
> > --brian
> >
> >   
> 
> _______________________________________________
> Sigtran mailing list
> [email protected]
> https://www1.ietf.org/mailman/listinfo/sigtran

-- 
Brian F. G. Bidulock
[email protected]
http://www.openss7.org/