Re: Re: M2PA (SIGTRAN) and multi-homing

Hi Brian,

The thrust of your reply I think is that, since multiple M2PA links use
the same underlying IP network, you don't get as much redundancy as you
might expect from multiple M2PA links.  In some situations it can cause
problems.

I think in some network topologies there may be advantages to multiple
M2PA links, though a single multi-homed link may provide similar
characteristics.  For instance, suppose that you architect you IP
network into two distinct subnets that use distinct hardware.  Each end
of the M2PA link has 2 NICs, one on each subnet.  Having at least one
M2PA link per subnet seems not unreasonable in this case and shouldn't
be vulnerable to the failure you mentioned.

Having more than one M2PA link per subnet probably doesn't help much in
most cases, since your robustness and throughput are limited by the
underlying IP layer.  Having some additional links may help reduce the
effect of head-of-line blocking.  Having some additional links may help
throughput during SCTP slow-start (if for some reason a burst of traffic
suddenly appears).  Neither of these considerations is likely to be
relevant over a good (managed) IP network.  Additional links increase
the chance of cascading failure and increase overall bandwidth
(slightly).  The above considerations are largely contradictory.  There
are tradeoffs in each case.  Over a good IP network none of them are
likely to be serious.

Robby, regarding your original question, here are some considerations
that weren't mentioned yet.

If you're working over a good IP network, either single or multi-homed
M2PA links should work well.  The differences that I see are mainly
operational.

Consider a network topology with distinct subnets.

- in the multi-homed case, you get to choose your SCTP primary path. 
This is potentially useful if one ISP is cheaper.  Depending on your
SCTP parameters you may also see better throughput and less added
latency if a path fails, as compared to relying on MTP changeover.

- in the single-homed case you get MTP alarms for link failures, link
congestion, etc if one of your IP routes goes down or becomes less
performant.  This gives you some extra diagnostics for free, especially
if you don't get alarms for ISP router failures.

- in the single-homed case you have the option to deactivate a link to
redirect traffic away from the impaired IP network.

Comments welcome,
Andrew

Brian F. G. Bidulock wrote:
> Robby,
>
> Robby Pocornie wrote:                           (Tue, 12 Dec 2006 15:23:14)
>   
>>    Brian,
>>
>>
>>
>>
>>
>>    Refer  to  the  Sigtran scenario's specification (e.g. ETSI EG 202 360
>>    v1.1.1(2004-12)), chapter 4.3...
>>
>>    In case there is only a single M2PA link in the link set, the messages
>>    formerly  sent out in parallel over different links are now serialized
>>    over  one  channel,  as  M2PA  makes  no use of the " streams" concept
>>    provided by SCTP...
>>     
>
> Yes, I see that (thanks for the reference).  However, I do not think
> that ETSI gave due consideration to the impacts of multiple M2PA links
> in a link set.  M2PA runs a slightly modified T7 timer to ensure that no
> message is delayed beyond limits.  If, as is stated, an IAM becomes
> delayed, if the delay is within T7 no message misses a deadline.  If it
> is delayed beyond T7, a changeover will occur to another link in the
> link-set.  However, in this scenario, MTP3 will changeover to a link that
> follows the same network path as the original link and can be expected
> to experience the same problems.  This is radically different from
> traditional SS7, where the links in a link set are independent and are
> normally engineered for diversity to ensure independence.
>
> Let me give you an example of the bad thing we were trying to avoid with
> the recommendation to use only one link in a link-set.  Take 16 M2PA
> links in one link set (each using the same network path).
>
> On link 1, an IAM gets delayed T7+.  The link fails.  MTP3 starts
> changeover (time controlled for argument's sake).  The traffic through
> the network path drops by 1/16th allowing intermediate bottleneck
> routers to pass messages now without delay.  MTP3 tries to restore link
> 1, but proving adds additional burst traffic and link 1 fails alignment
> and goes into lockout.  After T2, MTP3 adds the buffered traffic to
> link 8.  Because buffered traffic is bursted by MTP3 on link 8, link 8
> subsequently fails.  The situation is bad already because the original
> IAM (and all the message behind it) has been delayed T7 + T2.  Going
> around again, after T2, link 8 adds traffic to link 2 and 9, both of
> which will fail because the burst traffic again exceeds the handling
> capacity of the intermediate bottleneck.  The original IAM and all
> messages behind it are now delay T7 + T2 + T2.  Two more iterations and
> all 16 links will fail.  The IAM and all messages behind it are now
> delayed T7 + 4 * T2.  MTP3 will then changeover to the other link-set
> (after another T2).  The IAM and all messages behind it have been
> delayed T7 + 5 * T2 and there could now be about 5 times the normal
> changeover burst traffic pent up.
>
> The other link-set, even when it uses a different network path, might not
> successfully handle the burst traffic which can lead to another another
> 16 link failures, and then C-links, and then another cascade failure in
> the associated STP in the STP pair, and then network isolation.  Large
> transmit buffers and M2PA's extremely large window (maximum RTB size)
> contribute to such a cascade failure.  SS7 is vulnerable to this
> whenever link failure is load related.
>
> At no point does MTP3 understand that these messages have been delayed
> long past their useful lifetime.  With a T7 of 1 second and a T2 of 2
> seconds, the initial IAM has been delayed 11 seconds and messages behind
> it in the buffers have been delayed up to 12 seconds.  Now, although
> ISUP T7 (awaiting ACM timeout) is 20-30 seconds, every other timer in
> ISUP is less than 12 seconds.  After the second link-set failure, there
> will be 24 seconds delay and after C-link failure, 36 seconds delay
> (given 16 C-links).
>
> Consider that if there was only one M2PA link in the link-set, the
> initial T7 timer expiry would have resulted in a changeover to the
> alternate link set restoring traffic within T7 + T2 and only buffering 3
> seconds worth of traffic.
>
> These types of failure scenarios are only possible because MTP3 assumes
> that the failure of one link in a link-set is completely independent of
> the failure of another link in the link-set and that the failure of a
> link is not load related.  That is, that the probability that link 8
> will fail given than link 1 has failed is the same as the probability
> that link 8 will fail given that link 1 has not failed.  Also, that the
> probability that link 8 will fail is independent of the load offered to
> link 8.  With multiple M2PA links in a link-set, the first condition no
> longer holds true.  Unfortunately, for M2PA the second condition (load
> related failure) never holds true.  Therefore, the recommendation as
> laid out in the RFC is that there only be one M2PA link in a link-set.
>
> There are two problems with the ETSI passage that you reference.  First,
> placing multiple M2PA links in a link-set does not decrease inter-link
> changeovers as is stated.  As show above, it increases inter-link
> changeovers which is bad, and, worse, can lead to catastrophic cascade
> failures.  Second, the messages behind the initial IAM do not get
> delayed as implied by the passage.  The messages behind the initial IAM
> will be immediately available in the buffers at the receiving SCTP once
> the IAM is filled in by Fast Retransmission and will not suffer another
> network queueing delay (that is, it is not the same as a Q.703 NACK).
> Providing redundant network paths and using SCTP's multi-homing feature,
> or CMT (concurrent multi-path transfer) can further reduce delay.
>
>   
>>    Advise  is: To avoid that messages are not delivered (message loss) in
>>    case  of  retransmission, more than one link using M2PA should be used
>>    to form a linkset.
>>     
>
> Bad advise.  You are better to use multi-homing over separate network
> paths and CMT to reduce delay in a single M2PA link in a link-set.
>
>   
>>    That  is  why  I  design  more than 1 M2PA link between two STP's (for
>>    telecommunication SG, not internet host)
>>     
>
> It is your network, but I would not in mine.
>
> BTW, the cascade failure above is not conjecture on my part.  In a
> previous life I diagnosed just such a cascade failure that resulted
> simply from introducing delay into SS7 B/D links between STPs and also
> introducing the characteristic of a load-related link failure.  The
> cascade failure (that occurred during the Jan 1998 east coast ice storm)
> resulted in 2 hours of network isolation and forty-million dollars in
> damages (at least that is what the resulting lawsuit claimed).
>
> --brian
>
>