Re: how to achieve URL hash LB algorithm on ZXTM

Interesting! The problem here is that you have two conflicting goals: load 
balancing the workload evenly across all squid servers, and ensuring that 
the same URLs get sent to the same servers. These goals are naturally at 
odds with each other.

If a session persistence mapping points to a dead backend server (e.g. the 
squid server while it was stopped) then the session gets remapped to 
another machine. These new mappings are then preserved. I suspect that 
this caused the imbalance. This is a downside to the initial approach of 
having a 'permanent' mapping of URL -> backend server.

You could also run into the same problem if you are caching a limited 
range of URLs, and more happened to map to one server than another. 
However, if you've got a cluster of squid servers, I assume that you are 
caching quite a bit of data and many URLs!

If you want to ensure that all squid servers get similar workloads, then 
you will need to let the URL mappings 'migrate' to other machines over 
time, this will let ZXTM apply its own load balancing to the choice of 
server as well as the session mappings.

Here is one way to arrange this compromise: First of all, we'll change the 
TrafficScript that hashes a URL to a session. Before, we used 1 byte of 
the MD5 of a URL to pick the session. That gives 256 combinations. To give 
us better granularity, we'll first increase this to 2 bytes, which is 
65536 sessions:

$session = string.left( string.hashMD5( http.getHostHeader() .
                                         http.getRawURL() ), 2 );
connection.setPersistenceKey( $session );

Next, you need to change the size of the 'universal session persistence' 
cache size (found on the UI page System->Global settings). It defaults to 
2048 entries. Increase this, to around 50000.

As before, as requests arrive at ZXTM, their URL gets hashed to a magic 
number 0-65535, which determines the session persistence mapping to use 
(which then determines which squid server to use). The difference is that 
before, with only 256 mappings, ZXTM would permanently remember the 
URL->squid mapping, whereas now it can only remember 50000 of the 65536. 
This means that ZXTM will 'forget' the squid server to use for the least 
frequently seen URLs. As a result, it will pick a new server to use, and 
this occasional server re-selection should ensure that the load across the 
squid servers will even out over time.

The magic '50000' number is key. Reduce this number, and URL->squid 
mappings will be forgotten sooner, so load distribution will happen more 
often. Increase the number, and URL->squid mappings will be preserved for 
longer. It also depends on the usage of your squid caches, i.e. if there 
are only a few popular URLs v.s. hundreds of thousands of different URLs. 
Some experimentation will be needed to find the value that suits your 
systems best.


Ben


On 06/08/2009 10:28, Eric Y. Zhang wrote:
> Hi Ben
> one more question, based on my test, I found that after I restart squid on one of my backend server,
>
> zxtm would not hash the URLs back. that caused my backends are not equal to serve requests.
>
> now, the traffic between them are 4:2:1.
>
> any thoughts?
>
> thanks
>
> Eric
_______________________________________________
lb-l mailing list
[email protected]
http://vegan.net/mailman/listinfo/lb-l
Searchable Archive: http://vegan.net/lb/archive
http://lbdigest.com Load Balancing Digest
http://lbwiki.com Load Balancing Wiki