Re: Re : Layer4 and Layer7 TPS

As always Tony, I bow to your supreme greatness!! :)

One thing to note also about measuring connections per second of any kind.
Many of the tests used in many reports have a zero payload.  They simply
measure setup and tear-down, not the impact of real traffic.  Remember that
if you are actually transmitting data it takes additional time and resources
to process that data--especially if any reassembly is required as well as
maintain state tables.  Therefore, most of the metrics Tony mentions are
truly 'best-case'.  

This gets me back to the "taste the pudding" comment.  The size of the
payload can drastically impact the performance as well. Real-world testing
of the application is really the only reliable metric.  Unfortunately, even
if you do a PoC, you may not be able to generate sufficient real-world
traffic to get an accurate picture of production performance.

KJ (Ken) Salchow, Jr. | Manager, Technical Marketing
D 651.423.1133
M 612.868.1258
P 206.272.5555
F 206.272.5555
www.f5.com



-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of
Tony Bourke
Sent: Friday, November 20, 2009 1:17 PM
To: Load Balancing Mailing List
Subject: Re: [load balancing] Re : Layer4 and Layer7 TPS

It depends on how you define "TPS", also.  While there's no ultimate 
authority on this (really? None?  Well then, I declare my self supreme 
authority! You are all witnesses to my greatness!) TPS is generally 
accepted to be a measure of SSL/TLS performance.  Specifically TLS is 
the measure of *new* SSL sessions initiated (as opposed to re-using 
existing SSL sessions, which can save quite a bit of resources).   
Sometimes TLS is used to reference the number of HTTP requests processed 
as well, but usually you see it with SSL/TLS only. 

SSL/TLS is a wrapper for a generic TCP connection, and with the HTTP 
protocol you can run multiple (tens, hundreds, perhaps thousands) of 
HTTP requests through a single TCP connection.  So TPS and "HTTP 
requests" per second are very different. 

Generally, you don't see a 1:1 ratio in terms HTTP requests per SSL 
connection.   The whole point of HTTP 1.1 was to allow multiple HTTP 
requests per TCP connection (and thus SSL/TLS connection) so as to be 
more efficient.  But the ratio you do see in the real world could vary 
between 1:1 to 1:1000 or more, depending on your site.  It's usually the 
SSL ASIC (or in some cases, just your general processor) that determines 
how many new SSL connection a device can handle.  Traffic with a 10:1 
ratio would mean that a device with a 10,000 TPS SSL chip could handle 
100,000 HTTPS requests per second. 

The performance numbers can be confusing, and it's gotten several people 
in trouble.  Not long ago a guy claimed to have bested F5 performance 
with off-the-shelf hardware and no SSL ASICs (He got his terms mixed up, 
he was comparing apples to oranges). 

So here's a few of the primary ADC/load balancer metrics defined:

Connection rate:  The rate of TCP connections per second a device can 
handle.  This could be either while the load balancer operates in Layer 
4 mode or Layer 7 mode. 
Request rate:  The rate of Layer 7 requests, such as HTTP requests, that 
a device can handle.  Multiples of these requests will probably be in a 
single TCP connection, so this number would likely be higher than the 
connection rate. 
L4 requests per second:  Could arguably be defined as Layer 7 protocol 
requests transiting through an ADC operating in L4 mode (the load 
balancer is only aware of the TCP connection, not what goes on inside), 
or just a simple TCP connection rate. 
L7 requests per second: Could arguably be defined as Layer 7 protocol 
request rate transiting through the ADC while operating in L7 mode, or 
just a simple TCP connection rate as well.
TPS: The rate of *new* SSL sessions.  A new SSL session requires an 
asymmetric encryption operation, which is what really gets the CPU and 
why most devices use an ASIC instead. 

Hope that helps,

Tony





ADC expert wrote:
> @Kenneth, I am not F5 employee.
>
> I am validating the existing ADC products for L4-L7 TPS. I really 
> appreciate F5 publishing their results. 
>
> Coming to the discussion, I believe and agree that hardware plays its 
> role to show up better performance. The recent press release from 
> Radware claims 340K TPS for its Alteon on their On demand hardware. 
> Compared to what Alteon 3408 used to have, its a tremendous 
> improvement. That proves software plays its role in performance.  At 
> the same time, it confuses when I see performance results of one ADC 
> vendor to be L7 -300K TPS that claims to have hardware excellence.
>
> Thanks,
>
>
> On Fri, Nov 20, 2009 at 12:41 AM, Kenneth Salchow <[email protected] 
> <mailto:[email protected]>> wrote:
>
>     While I might not be quite so exuberant in my response, I will
>     suggest the Surya has a couple valid points.
>
>      
>
>     First, however, let me say that, to the best of my knowledge, that
>     post was NOT made by an F5 employee.  If it is discovered that it
>     was, it was inappropriate and it will be rectified.  IF, however,
>     it truly is simply one of our customers who found some value in
>     our testing, I’m happy to see their excitement and enthusiasm.  I,
>     personally, believe we do a pretty fair job of presenting the
>     facts; we publish the results regardless of whether we come out on
>     top or not and we provide complete transparency as to the
>     configurations of all devices, testing equipment and methodology. 
>     Given the report, the time and the equipment, anyone should be
>     able to produce the same results.  I truly believe it is as
>     unbiased as it can be.
>
>      
>
>     That being said, I agree with Surya the no matter how unbiased a
>     performance report may or may not be, the true test is how the
>     device performs in YOUR environment with YOUR applications and
>     YOUR business processes.  As they say, the proof of the pudding is
>     in the tasting; make sure you get a true taste.  If possible, a
>     PoC is something we suggest all our prospective customers do. 
>     Anyone who has been a part of this list for any time knows that I
>     am a strong proponent of that philosophy.  In the end, that’s all
>     that matters.
>
>      
>
>     I wish the ADC Expert good luck (Bon Chance, I believe?) in their
>     search for the best product for their needs.  If you have any
>     specific questions, I’m sure everyone in the list will be happy to
>     give their perspective and input.
>
>      
>
>      
>
>      
>
>     *KJ (Ken) Salchow, Jr.* | Manager, Technical Marketing
>
>     *D 651.423.1133*
>
>     	
>
>     *M 612.868.1258*
>
>     	
>
>     *P 206.272.5555*
>
>     	
>
>     *F 206.272.5555*
>
>     	
>
>     *www.f5.com <http://www.f5.com/>*
>
>      
>
>      
>
>     *From:* [email protected] <mailto:[email protected]>
>     [mailto:[email protected] <mailto:[email protected]>]
>     *On Behalf Of *Surya ARBY
>     *Sent:* Thursday, November 19, 2009 12:43 PM
>     *To:* Load Balancing Mailing List
>     *Subject:* [load balancing] Re : Layer4 and Layer7 TPS
>
>      
>
>     Hello.
>
>     To me this kind of "performance reports" or "competitive analysis"
>     are just a bunch of bullshit.
>
>     You'll find a lot of other papers on Citrix / Cisco / any vendor
>     web site with a lot of case studies stating they are the best
>     vendor with the best appliance and that the end customer is the
>     happiest customer in the world with his solution :)
>
>     Make a POC and setup your own load testing :)
>
>     The real L4/L7 performance in a specific configuration varies
>     according to the internal architecture of the product you test.
>     Cisco ACE module distributes the load between two NPs according to
>     the result of a hash function to process load balanced flows, Ace
>     appliance relies on a single Cavium Octeon chipset for the data
>     plane, Citrix uses a purely software based appliance, F5 and
>     Foundry use another architectures (barrel processors in foundry
>     boxes are distributed like NPs in cisco ace module)...
>
>     When you perform load testing it's very important to understand
>     how the box you test really works internally, because results (and
>     comparison) will change with the choosen scenario that's why the
>     "performance report" you mention is necessarily biaised :)
>
>     regards,
>
>     Surya
>
>     --- En date de : *Jeu 19.11.09, ADC expert /<[email protected]
>     <mailto:[email protected]>>/* a écrit :
>
>
>     De: ADC expert <[email protected] <mailto:[email protected]>>
>     Objet: [load balancing] Layer4 and Layer7 TPS
>     À: [email protected] <mailto:[email protected]>
>     Date: Jeudi 19 Novembre 2009, 19h09
>
>     Hi Experts,
>        I was looking for a  ADC device that can give best L4 and L7
>     TPS. After considerable search, I got few things to get clarified
>     before selecting the best.
>        F5 has extensive performance report in its website.
>        http://www.f5.com/pdf/reports/f5-performance-report.pdf
>        I wonder why other vendors did not report in such detail. Even
>     if, there is  no free access.
>        I appreciate if some one sends me across cisco, citrix and
>     other vendors performance details.
>
>       -Thanks.
>        
>
>
>     -----La pièce jointe associée suit-----
>
>     _______________________________________________
>     lb-l mailing list
>     [email protected] <http://mc/[email protected]>
>     http://vegan.net/mailman/listinfo/lb-l
>     Searchable Archive: http://vegan.net/lb/archive
>     http://lbdigest.com Load Balancing Digest
>     http://lbwiki.com Load Balancing Wiki
>
>      
>
>
>     _______________________________________________
>     lb-l mailing list
>     [email protected] <mailto:[email protected]>
>     http://vegan.net/mailman/listinfo/lb-l
>     Searchable Archive: http://vegan.net/lb/archive
>     http://lbdigest.com Load Balancing Digest
>     http://lbwiki.com Load Balancing Wiki
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> lb-l mailing list
> [email protected]
> http://vegan.net/mailman/listinfo/lb-l
> Searchable Archive: http://vegan.net/lb/archive
> http://lbdigest.com Load Balancing Digest
> http://lbwiki.com Load Balancing Wiki
>   

_______________________________________________
lb-l mailing list
[email protected]
http://vegan.net/mailman/listinfo/lb-l
Searchable Archive: http://vegan.net/lb/archive
http://lbdigest.com Load Balancing Digest
http://lbwiki.com Load Balancing Wiki

_______________________________________________
lb-l mailing list
[email protected]
http://vegan.net/mailman/listinfo/lb-l
Searchable Archive: http://vegan.net/lb/archive
http://lbdigest.com Load Balancing Digest
http://lbwiki.com Load Balancing Wiki