Re: F5 LTM hitting SSL TPS license limit

Eric,

 

First-I LOVE your title; that's awesome.

 

Ok-now to business.  After seeing your message, I wanted to find out what
Product Development had to say about it and here is their direct, un-edited
response:

 

[quote]

The code provides for 10ms accuracy (5ms of backlogged time, and 5ms of
limiting) for a TPS limit, where a TPS is defined, essentially, as a SYN (or
other connection startup) message on a clientssl virtual.  The rate is
calculated on the expectation that over any given 10ms window, the rate of
new connections should not exceed the rate that would, if it were continued
for the rest of the second, exceed the TPS limit.  By only using a 10ms
window, we allow for continued connections at the maximum allowable TPS
limit for the entire second, thus saturating the entire limit while
servicing connections at a constant rate.  So naturally, if you try to drop
1000 connections on a box with a 1000 TPS /simultaneously/, the TPS limit
will fire because it expects that rate to continue /for the rest of the
second/.

 

An alternative method would be to accept all 1000 connections, and deny all
others for the rest of the second.

        * One reason not to do this approach is that it will block ALL new
connections for the rest of the time period.  This increases the
effectiveness of DOS attacks and decrease the general reliability of the box
itself under load.

        * Additionally, SYN's are expected to be sent three times by
most/all well behaving TCP stacks.  This provides us with an automatic
smoothing behavior - we ignore the SYN's during spikes, and the client will
automatically try again over macro-time, hopefully smoothing out the spike.
This is why, in a real world test, the current behavior would not actually
result in dropped or timed-out connections for users, despite momentary
spikes.

 

So, in summary, the higher resolution limit enforcement provides several
measurable benefits over allowing a full TPS limit's worth of connections at
any given point.  The first major advantage is to allow new connections to
continue to be serviced at a constant predictable rate over time.  The
second is increased DOS and spike resistence.  Plus, given normal connection
handshake behaviors by clients, there is a decreased likelyhood that
connections will be completely dropped.

[/quote]

 

I hope that makes more sense and helps you understand why our developers
made the decision they did.  Consequently, I also don't see this behavior
changing in any way as it is entirely by design-good, bad or otherwise.  The
positive thing is that with your posting and this reply-at least we've made
it much easier for other users to understand why they might see that
behavior.  For that, BTW, thank you.

 

Now-Hamish also made a comment concerning our SSL license limit and how we
handle it when that limit is exceeded.  It is true, that once we exceed the
SSL license on a BIG-IP LTM device, we no longer accept SSL connections.  It
is ALSO true that in the case of compression, we still process the
connections, but we no longer apply compression.  There is a significant
difference between these activities that make this distinction necessary.
When dealing with compression-it's not that we simply 'fall-back' on
software-we just quit doing compression beyond the license limit; in this
case, it has no affect on whether traffic continues to flow or not, just
whether it is compressed.  The act of compressing/not-compressing is not
relevant to the actual connection itself.

 

With SSL, however, the limit *is* connections.  We cannot simply choose
*not* to provide SSL services and somehow have the connection still exist
and be processed-they are mutually exclusive.  Hamish suggests that we could
'fall back' to software-but that is making the assumption that the license
limit is for 'hardware SSL', which it isn't; it is for SSL TPS for the whole
box. 

 

So, in this regard-comparing Compression licensing with SSL licensing simply
doesn't work.  However, that being said, both your (Eric) perspective as
well as Hamish's perspective are being heard and used as important criteria
when making future decisions about how we handle these processes.  Having
been around F5 for almost a decade and having been a customer before that-I
know firsthand that this is the way F5 products have continued to evolve
year-over-year; by taking the input and ideas of the people who use them
every day.

 

KJ (Ken) Salchow, Jr.  |  Manager, Technical Marketing

 

From: [email protected] [mailto:[email protected]] On Behalf Of
Rosenberry, Eric
Sent: Thursday, February 19, 2009 4:06 PM
To: Load Balancing Mailing List
Subject: [load balancing] F5 LTM hitting SSL TPS license limit

 

FYI-

 

I ran into this issue with my F5 LTM's recently and I wanted to share it
with the rest of the world in such a form that future Google searches would
pick it up since I could not find any information on it myself.

 

If you are getting a message similar to this in your LTM logs and you don't
think you are anywhere near your licensed SSL TPS limit, you might be
running into this issue:

tmm tmm[1253]: 01260008:3: SSL transaction (TPS) rate limit reached

 

After many hours of working with F5 support and analyzing packet dumps to
prove the issue to F5 it would appear that the F5 tracks your SSL TPS
license limit on some sub-second interval.  i.e. if you have a 2000TPS
license it seems to chop this up into smaller windows (say 1/10th of a
second for this example) and enforce the limit of 200 TPS on each 1/10th of
a second bucket.

 

I ran into this issue while running some load tests against our system at a
SSL TPS rate that should have been less than the license limit, however, due
to the burstiness of the load test it hit F5's artificial license rate
limit.

 

I know this issue exists in code version "BIG-IP 9.4.1 Build 29.0".  I can
not speak for any other versions.

 

F5 does not have a code fix for this issue and at the moment.  If you need
to prove to F5 support that this issue exists you can have them refer to
case C492020 where it is documented.

 

-Eric

_______________________________________________________________
Eric Rosenberry
Sr. Network Engineer | Chief Bit Plumber
Direct +1. 503.943.6763 | Mobile: +1.503.348.3625 | Fax: +1.503.224.1581

 

 
iovation
111 SW Fifth Avenue
Suite 3200
Portland, OR 97204
www.iovation.com <http://www.iovation.com/> 
 
The information contained in this email message may be privileged,
confidential and protected from disclosure. If you are not the intended
recipient, any dissemination, distribution or copying is strictly
prohibited. If you think that you have received this email message in error,
please notify the sender by reply email and delete the message and any
attachments.

_______________________________________________
lb-l mailing list
[email protected]
http://vegan.net/mailman/listinfo/lb-l
Searchable Archive: http://vegan.net/lb/archive
http://lbdigest.com Load Balancing Digest
http://lbwiki.com Load Balancing Wiki