splitting configurations for faster uploads

--=-LynaN+2bDCvgyx/2SwB6
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit

Hi all

I've used sitecopy to upload the cpt-city website 

  http://soliton.vm.bytemark.co.uk/pub/cpt-city/29s

for some years, but as the site has grown the upload 
times have too: 250 MB in 48,117 files was taking 
8:30 minutes to update, even if there were no changes 
to upload.

Reading on the mailing list that one solution is to
split the upload configuration, I wrote a script which 
takes an existing configuration and creates a new 
file with a configuration for each top level subdirectory.  

For cpt-city (which has 56 top-level subdirectories) the 
effects were dramatic, the same no-change upload now takes 
20s, a speedup factor of 25. 

Possibly the script could be useful to other sitecopy
users. It is written in Perl and uses only standard library
modules, tested on Linux, but should be easy to port to 
other systems, and is contributed to the public domain.

Enjoy
-- 
J.J. Green
http://soliton.vm.bytemark.co.uk/pub/jjg/


--=-LynaN+2bDCvgyx/2SwB6
Content-Type: application/x-perl; name="sitecopy-generate-rc.pl"
Content-Disposition: attachment; filename="sitecopy-generate-rc.pl"
Content-Transfer-Encoding: 7bit

#! /usr/bin/perl -w
# 
# split a sitecopyrc file up into a series of smaller
# targets
#
# J.J. Green, 2010
# Contributed to the public domain

use strict;

my $myname = "sitecopy-generate-rc";
my $dirsep = "/";
my $namesep = "-";

my $usage = "
$myname : generate per-directory sitecopy configuration

Usage :

  $myname [options] <rcfile>

where <rcfile> is a sitecopy configuration and [options] may include

  -h          this help
  -o <file>   write output to file, not standard output
  -v          verbose operation
  -x <dirs>   exclude the specifed (comma separated list of) 
              directories from the bottom level install target
";

# parse command line

use Getopt::Std;
use POSIX;

my %opt = ();
getopts('ho:vx:',\%opt);

if ($opt{h})
{
    print $usage;
    exit EXIT_SUCCESS;
}

my $verbose = $opt{v};

sub info
{
    $verbose &&	print @_;
}

my ($infile) = @ARGV;

if (! $infile)
{
    warn "no input specified\n";
    warn $usage;
    exit EXIT_FAILURE;
}

if (! -e $infile)
{
    warn "input file $infile not found\n";
    exit EXIT_FAILURE;
}

my $outfile = $opt{o};

my %excl = ();

if ($opt{x})
{
    %excl = map { $_ => 1 } (split ',', $opt{x});
}

# say hello

info "This is $myname\n";

# read input

unless (open CONF,"< $infile")
{
    warn "failed to open $infile for reading: $!\n";
    exit EXIT_FAILURE;
}

my @lines = <CONF>;

close CONF;

info "read $infile\n";

# generate a hash of sites

my $sites;
my $name;

foreach my $line (@lines)
{
    chomp $line;

    # if a site line then record the name

    if (my ($match) = ($line =~ /^site\s+(\S+)/))
    {
	$name = $match;
	$sites->{$name} = undef;
	next;
    }

    # don't parse other lines until we have a site name

    $name or next;

    # save data for this site

    if (my ($key,$val) = ($line =~ /^(local|remote)\s+(.+)/))
    {
	# local/remote as separate hash entries

	# strip trailing $dirsep if present

	unless ((my $lastchar = chop $val) eq $dirsep)
	{
	    $val .= $lastchar;
	}

	$sites->{$name}->{$key} = $val;
    }
    else
    {
	# everthing non-empty is dumped into an array

	unless ($line =~ /^\s*$/o)
	{
	    push @{$sites->{$name}->{other}}, $line;
	}
    }
} 

my $ost;

if ($outfile)
{
    open $ost, "> $outfile";
}
else
{
    $ost = *STDOUT;
}

# process the sites

foreach my $name (keys %$sites)
{
    my $site = $sites->{$name};

    info "  $name\n";

    # get remote directory

    my $rem = $site->{'remote'};

    unless ($rem)
    {
	warn "site $name has no remote entry\n";
	exit EXIT_FAILURE;
    }

    # get local directory

    my $loc = $site->{'local'};

    unless ($loc)
    {
	warn "site $name has no local entry\n";
	exit EXIT_FAILURE;
    }

    if (not -d $loc)
    {
	warn "site $name local field $loc not found\n";
	exit EXIT_FAILURE;
    }

    # other lines

    my $other = $site->{'other'};

    # get the list of subdirectories of $loc

    my @subdirs = ();

    unless (opendir LOC, $loc)
    {
	warn "failed to open directory $loc\n";
	exit EXIT_FAILURE;
    }

    while (my $entry = readdir(LOC))
    {
	next if $entry eq ".";
	next if $entry eq "..";

	next if $excl{$entry};

	my $path = join($dirsep,($loc,$entry));

	next unless -d $path;

	push @subdirs, $entry;
    }

    closedir LOC;

    # generate the subdirectory configurations and at the
    # same time generate the bottom level exclude lines

    my @ignlines = ();

    foreach my $subdir (@subdirs)
    {
	my $subloc  = join($dirsep,($loc,$subdir));
	my $subrem  = join($dirsep,($rem,$subdir));
	my $subname = join($namesep,($name,$subdir));

	push @ignlines,"exclude $dirsep$subdir/*";

	foreach my $line ("site     $subname",
			  "local    $subloc",
			  "remote   $subrem",
			  @$other)
	{
	    print $ost "$line\n";
	}

	print $ost "\n";
    }

    # finally, print out the bottom level

    foreach my $line ("site    $name",
		      "local   $loc",
		      "remote  $rem",
		      @$other,
		      @ignlines)
    {
	print $ost "$line\n";
    }

    print $ost "\n";

}

close $ost;

info "done.\n";

--=-LynaN+2bDCvgyx/2SwB6
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
sitecopy mailing list
[email protected]
http://lists.manyfish.co.uk/mailman/listinfo/sitecopy

--=-LynaN+2bDCvgyx/2SwB6--