Re: A n00b's Load Balancer Questions (Updated)

Ken,

I've seen that you are an active contributor on this list, and I greatly 
appreciate your answers. I thought I'd reply back to add a couple things 
that came to mind while reading your responses: *[GF]*

** Happy Thanksgiving to All - including those far away in other lands! **

Gary,

Let me take a quick crack at a first response.   And, if you don't see
it--full disclosure--I work for F5 Networks, but I'll try and be as unbiased
as I can.  See below:

KJ (Ken) Salchow, Jr. | Manager, Technical Marketing


Here goes:

First, I assume these basic services are handled easily by most LB
solutions:
- DNS

[[KJSJ]] DNS shouldn't be that difficult--just remember both TCP and UDP.

- NTP (not super critical, since nothing immediately stops if this goes
off-line)

[[KJSJ]] Never done NTP, but I don't see why not.

- SMTP

[[KJSJ]] In a pure 'fail-over' this is pretty straight forward too.
Load-balancing is a little more tricky as most devices can only load balance
per TCP connection.  Plus, you still have backend message store
synchronization issues either way.

*[GF]* I'm only concerned about SMTP sends from various client 
applications, so it sounds like this is straightforward, as expected.

Then, the expected HTTP/HTTPS.
- Users of internal web apps are sensitive to latency, just because the
desktops are running real-time. They need to stay snappy.

[[KJSJ]] a lot of the products provide HTTP caching and compression options
which should improve performance.  One of the nice things about terminating
SSL on the ADC is that you can still do caching and compression prior to
encrypting the outbound traffic.  It's a nice solution.

*[GF]* SSL isn't a requirement on the internal web servers. Generally, a 
connection from an internal client will be busier and more 
latency-sensitive than (the more plentiful) public connections. This was 
more of a note that, even if the functionality is met by an entry-level 
device, I want to make sure performance isn't an issue. Frankly, though, 
we're almost always driven by functionality and not performance with 
modern hardware. Perhaps you can tell me if there's some sort of class 
of service that could be applied to applications so that my internal 
clients would be given preference to other jobs if the thing gets congested.

- Of course, these are web *applications*, so persistent connections are
important, though it seems most everything supports methods for that now.

[[KJSJ]] Remember to be careful using SSL session ID persistence as it
renegotiates frequently.  In your case though, with all internal users,
simple source IP persistence will probably do just fine.  If not, look for a
device that can maintain persistence on payload and/or header information.
That's another good reason to terminate SSL at the ADC.

- I'd like to do some reasonably sophisticated health checks. Basically
query the web server a few different ways and see that I'm getting
answers that verify the web service is up, that it has database
connectivity, etc.

[[KJSJ]] The level of sophistication of the health checks is one thing that
I think is still pretty much across the board for commercial devices.  Some
provide very basic functionality and some provide complete
self-determination.  Even among devices that have sophisticated
capabilities, see which ones offer a decent starting point with templates or
built-in health checks that can get you up and running quickly.  Also--what
happens when a node fails a health check is important.  With basic HTTP,
however, I'm sure most devices can give you pretty decent capability.

One comment--as soon as you said 'sophisticated'.  Remember that health
checks add overhead to the ADC, the back-end network AND the server.  BE
CAREFUL.  I've seen a customer completely seize their network because they
had a health check that took 45 seconds to complete and they scheduled it to
run every 15 seconds.  ;-)

*[GF]* Good point! I was reluctant to use "sophisticated" because I 
think it may imply more complexity than I intend. I'm thinking a diverse 
set of small checks that gives me a complete picture of availability vs. 
some basic canned check that doesn't tell me much (e.g. ping tells me 
there's power and an active stack, but not much else). But point taken - 
it's something to be careful with.

- SIP telephony. As mentioned above, we use Asterisk servers running
SIP, IP soft phones on the (Mac) desktops, Cisco AS5400 gateways, and
Polycom IP phones. What I'm trying to provide for here is a failover
from the primary Asterisk server to a standby system. There's a few
obvious ways to do this, but simply I envision a case where the LB would
be in the loop during call setup, re-directing a (new) call to the
active system. The Asterisk server and endpoint are then in the call,
and the LB has no more part in it. If the Asterisk server dies, any
calls connected through it are dropped (that's acceptable), but new
calls are then routed by the LB to the standby box. In a "graceful
shutdown" mode, the LB would be notified to send all new calls to the
standby system. We can then down the primary system after clearing its
calls.

[[KJSJ]] SIP is similar to SMTP in the fact that it streams multiple
'messages' over a single connection.  I'm certainly not an expert with the
AS5400 or Asterisk, but, in a pure "fail-over" mode this should run much
like SMTP and provide the functionality you desire.  There are devices that
will also "de-mux" SIP, i.e. instead of load balancing a single connection
between the gateway and the server, they can pull out and separate the
multiple messages and load balance them across multiple servers.  Some of
these devices are additional boxes to the ADC, some are additional modules
and some are simply part of the standard fair.  While you might only want
fail-over today, having this functionality might be good down the road.
(BTW, if you can handle the message store synchronization, some of these
products can do the same message-based load balancing for SMTP as well as
other protocols like Diameter).


- Directory and Authentication. We're running a Win 2008 AD mainly for
user management across the back-end servers. This is kind of like DNS.
There are methods for availability and redundancy, but the Mac clients
(particularly) don't always behave nicely when the primary fails.
"Timeout and try the secondary" just causes uncomfortable hiccups in
operations.

[[KJSJ]] This one worries me a tad.  I don't remember the traffic flow, but
if I remember correctly, AD has a tendency to insert real IP addresses of
the server into the payload and cause some problems.  BUT--it's been too
long for me to provide any real guidance.  Maybe someone else with more
recent experience can chime in.  I don't believe it's insurmountable, but
might be a little trickier than the others. Sorry I can't remember.

- SMB/CIFS. We do have a file store on a plain old Win 2003/2008 file
server. It's accessed by the Macs, of course, and Linux and Windows
clients too; and it always needs to be there. Keep in mind that the real
"load balancer" purpose is to merely serve clients from a standby system
in case of a primary failure here. (I'm not expecting that the LB would
have any role in synchronizing or monitoring sync of the real servers.)
Basically, can any LBs proxy a file server?

[[KJSJ]] I know that we have a product that is totally built for doing
pretty much what you want (and then some), but again, I seem to recall
several gotchas with maintaining connections and embedded IP addresses using
a straight load balancer.  Again, I don't believe it is completely
insurmountable and you'll still have to deal with file synchronization, It's
just been too long for me to say anything more with any certainty.


- SQL Server (2005). I've seen a number of references to LB/ADC fronting
databases, though usually with Oracle for some reason. Our current plan
is to run SQL's Mirroring to provide a H/A db server pair. The only
catch is that this requires specific MS-native drivers that support the
mirroring. There are no native drivers for some client systems (mainly,
but not exclusively, the Macs), so we end up writing a (Java/JDBC)
middleware layer to bridge the gap. So, I'm wondering if any LB can
proxy SQL by talking ODBC or some such to our various client systems,
while talking "native" back to SQL Server. There would only be one
"real" SQL server in this case, even though it's really a virtual system.

[[KJSJ]] There are many documented solutions out there (as you mentioned) as
to how to make this work.  Usually the problem is the synchronization of the
db on the back side, not load balancing a simple connection to a server.
And, most devices aren't going to "proxy" SQL in the manner I think you're
talking about. The typical deployment I'm familiar with is to use SQL
mirroring to synchronize the back end system, but use the load balancer to
send traffic to the physical servers instead of using the "virtual" provided
by the mirror. I'm probably stretching my memory and current skills here
again--so, someone else is welcome to chime in. :)

*[GF]* This is a potential point of real value to us, so any other 
chiming-in is good. Specifically it's this "proxy" function that's of 
value - at least as we envision implementing the db - and I was hoping 
it may be there as a byproduct in some product's LB repertoire. I 
misspoke on the SQL Mirroring. It's not a virtual server (as with 
clustering), but it specifically only allows connections to one 
"Primary" (physical) server. SQL rejects any attempt to connect to the 
secondary (Mirror) server.

- All of these functions are met performance-wise by two or three real
servers behind the LB. But, I count up to 8 different functions served
by the LB, with possibly 3 - 4 web server groups, for perhaps a dozen
different "virtual" LB applications. Some of these real server groups
operate on different subnets.

[[KJSJ]] shouldn't be an issue and there are multiple solutions from
multiple physical interfaces to VLAN segmentation.

- One possibility, if all were simple functions, would be to simply get
several old Alteon 180 boxes. People seem to have a lot of regard for
that line. If our more complex needs (wishes) can't be accomplished,
then this may be a practical route. This brings up the idea of several
cheap boxes vs. one higher-end box (two really, since we would obviously
need an H/A pair of load balancers). We often deploy servers with very
limited functionality - not so "green", I know, but it reduces the
impact of errors/failures (yes, while increasing the chances of having
one). I would be concerned about a single high-end box (pair) becoming a
single point of failure for multiple services. If we have one box, then
it needs to partition well so we don't screw up one thing and kill
several at the same time.

[[KJSJ]] Whether "high-end" or not, an H/A pair is, by definition, not a
single point of failure if you run them in active/stand-by; that's classic
N+1 redundancy. You mentioned management (in the next point) as an issue, so
why would you want to manage multiple configurations and multiple boxes as
well as multiple points of trouble-shooting when you don't have to?  In
addition, most boxes provide various forms of partitioning to help
'separate' various components.  Of course, if that's what you prefer to
do--that's what you prefer.

*[GF]* Nah, I don't want LBs proliferating through the racks like 
rabbits. I do want to know that I could, say, pilot a new application on 
one LB, and easily move back or forward to the alternate box/prior state 
without a lot of tedious: tear down the H/A config --- try your new 
stuff --- move the applications --- reconfigure the H/A pair. I can't 
take it for granted that's not the case. A key thing for any H/A setup 
to protect against, imho, is personnel screw-ups (or is that personal?). 
I also didn't know if partitioning was a fairly standard, or fairly 
exotic capability.

- Ease of administration is important (of course!). I saw a note here
some weeks ago that an F5 LLB requires a knowledgeable consultant and a
week to setup. That doesn't sound very appealing. And yes, I expect some
of the more complex functions, if they're doable, to require a more
complex machine; and I also don't want to learn more than one. So, we'll
just have to evaluate the trade-off between complexity and the value it
delivers.

[[KJSJ]] No offense intended to anyone, but one person's experience is NOT
necessarily indicative of an entire product line. I would submit that EVERY
device has had its share of nightmare installs where things just don't go
the way they normally do.  I've personally done installs that took 15
minutes and installs that have taken a week.  Much like the performance
conversation in a different thread on the list, I'd suggest you get a chance
to 'play' with the management interface of the devices and get a feel for
the overall management.  Talk to the SE's about what the install for your
environment would look like.  MOST OF ALL, the more prepared you are and the
more upfront planning, the better ANY install will go.  Most products
provide some form of installation as part of the purchase--if you're
prepared and plan, it should go pretty well in most cases.

*[GF]* Agreed, and I certainly didn't intend to characterize an entire 
product line by that one comment (especially yours :). That was a 
specific install reference on this list awhile back, and the only one I 
recall. And granted this is a *very* subjective point. That said, there 
should be some difference in the learning curve for different products. 
While that may correlate to their capabilities, it's not a given. Think 
of a programming a modern IP-PBX with a nice web setup vs. an AT&T 
Definity Console. They both have similar capabilities, but I can get one 
working much faster than the other when starting from scratch. In 
fairness, I think it's still a case where we need to define the 
functions we want to use, then satisfy ourselves that the install and 
management will work for us.

- Application development. We do have the advantage of developing most
applications that we use in-house, so we can be flexible when it comes
to interfaces with the LB.

[[KJSJ]] That's very cool--and many devices provide programmatic interfaces
into the device.  This is something that can alleviate some of the overhead
of things like health monitoring and maintenance.

- Cost. Obviously it depends to an extent on the value delivered. We're
a small company, so we're cost-conscious (still have 10 year-old servers
running some jobs). It also means that I don't have fixed budgets or
bureaucratic processes to deal with either. My understanding is that
some of the entry-level "high-line" boxes are now in the $15-20K range.
We could do that if delivers all we're looking for. If, however, a
couple $2K boxes do the trick with some sacrifices (or not?!?), we can
work with that too.

[[KJSJ]] I can't tell you what to spend (can I? :) ) so you've hit the nail
on the head--it's a tradeoff between actual cost and value derived.
Especially when times are tight, many organizations settle for 'good
enough'.  I, personally, try to imagine my needs 3-5 years out and make sure
that my solutions can adapt and maintain value.  Again--that's your call.

*[GF]* Well, I typically would buy a $20 set of pliers, when a $2.99 set 
will do - I like good tools. We also do things for the long-term here. 
Again, I'm hoping for some guidance from the answers that would point me 
to either the $3 tool or the $20 tool.

- Finally, I have seen references to newer system incorporating
security/firewall/ips functions. How robust are these? This box will
necessarily be in the loop for most externally accessible systems, so I
wonder to what extent it can function as a secure router, in effect?
That would theoretically let me move some firewall dollars towards the
LB project? Sound plausible, or am I asking too much?

[[KJSJ]] Some would argue that most ADC or LB solutions have always been
secure routers/firewalls.  If you look at how VIPs are created, they are
essentially IP:port combinations defining allowed traffic, not much
different than a firewall rules list.  BUT, I don't want to start that
debate again and almost NO ONE that I know of sells an ADC as a complete
network firewall replacement.  However, yes, many devices provide additional
levels of security from IP Chains through Web Application Firewalls and just
about anything in between.  You'll have to evaluate what you can/can't do or
feel comfortable with based on your circumstances.


I hope this helps some.

*[GF]* It does, and thank you!!



_______________________________________________
lb-l mailing list
[email protected]
http://vegan.net/mailman/listinfo/lb-l
Searchable Archive: http://vegan.net/lb/archive
http://lbdigest.com Load Balancing Digest
http://lbwiki.com Load Balancing Wiki