Re: Tux as a remote server caching device?

Brian Akins wrote:

>Will tux cache these dynamic requests?
>
>On Thu, 21 Nov 2002 01:58:17 -0500
>Mike Savage <[email protected]> wrote:
>  
>
>>I have looked into doing the same thing.  I setup Apache running with tux on
>>one server, this being my static content server, having apache proxy all
>>dynamic content requests to a separate box.  It worked pretty well, and
>>allowed the dynamic box, a Tomcat-JBoss app server, to have a little more
>>"breathing room".
>>

No, TUX won't cache the dynamic requests;  They always get passed 
through without much intervention from TUX.

If you want dynamic requests to be cached for speed/load issues, you can 
use a script and some cron jobs to handle it.  I use Apache Tomcat as 
the dynamic backend with TUX in front and it works well for me on 
resources that can be up to a minute stale.

The Perl script I use is called "graburls"
------BEGIN SCRIPT--------

#!/usr/bin/perl

use strict;

my ( $interval ) = $0 =~ /\.([^.\/]+)$/;
die "No interval" if !$interval;
open(LOG, ">>/home/gremlin/web.cache.log");
print LOG localtime() . ": " .  $interval . "\n";
close(LOG);
my $basedir = "/var/local/www/docroot";
&grab_urls($interval);

sub grab_urls ($) {
  my $interval = shift or die "grab_urls() called without an interval!";
  my @urls = `find $basedir -type f | grep \\.cache/url-list.$interval`;

  my $url;
  foreach $url ( @urls ) {
    chomp( $url );
    my ( $path, $file ) = $url =~ /^(.+\/\.cache\/)(url-list\..+)$/;
    chdir( $path );
    open(LOG, ">>" . $path . "cache.log");
    print LOG localtime() . ": " . $path . $file . "\n";
    `wget -b -t 1 -q -a error.log -N -i $file`;
    close(LOG);
  }
}

------END SCRIPT--------

Then I made symbolic links to this script with names like
  graburls.1minute
  graburls.5minutes
  ...
  graburls.1week

Then I made some cron entries for my maintenance user (gremlin).  You 
can edit a user's cron entries with 'crontab -e'.  Do NOT do this with 
the system crontab (/etc/crontab) and choosing a user other than root is 
advisable.

------BEGIN CRONTAB--------

  # crontab: minute - hour - date - month - day - script/program
  0-59 * * * * ~/bin/graburls.1minute
  0,5,10,15,20,25,30,35,40,45,50,55 * * * * ~/bin/graburls.5minutes
  0,10,20,30,40,50 * * * * ~/bin/graburls.10minutes
  0,15,30,45 * * * * ~/bin/graburls.15minutes
  0,30 * * * * ~/bin/graburls.30minutes
  0 * * * * ~/bin/graburls.1hour
  0 0,4,8,12,16,20 * * * ~/bin/graburls.4hours
  0 0,8,16 * * * ~/bin/graburls.8hours
  0 0,12 * * * ~/bin/graburls.12hours
  0 0 * * * ~/bin/graburls.1day
  0 0 * * 1 ~/bin/graburls.1week
  0 0 1 * * ~/bin/graburls.1month

------END CRONTAB--------

Then I made directories called ".cache" in any directory I wanted to 
cache.  Inside this directory, I put a file or files called 
url-list.[interval] where [interval] is 1minute, 5minutes, etc.  (eg. 
url-list.1hour).

url-list.[interval] contains a simple list of URLs separated by newlines.

And finally I made symbolic links from the .cache directory to the 
actual working directory.  This makes it easy to see which items are 
getting served as a TUX dynamic cache and which are actual static files: 
an 'ls -l' looking for links pointing to the .cache directory.

As long as items can be at least a minute stale, it works well.  For 
items that change less than once an hour, it can be huge efficiency win 
for the server.  Obviously, anything that requires a smaller granularity 
than 1 minute cannot be handled, but then you have constantly dynamic/up 
to the minute data that can't be effectively cached anyway.

Hope someone out there finds this useful.  Also, if anyone sees any 
bugs/security holes, let me know.

- Miles