Re: Need help with (really) huge site

On Tue, Jan 03, 2006 at 07:14:25PM +0100, Kai Rittmeyer wrote:
> I have a directory with about 5000 sub-directories, each of these
> sub-directories has 3 sub-directories and these contain about 100
> .jpg-files each. I gave sitecopy about 1 day of CPU time - then my SSH
> client shutdown due to a reconnect - I could run it via cron - but 1 day
> is just too long for a single job. 

So about 1.5 million files in total?  You are using "state timesize" (or 
no state directive at all), I presume?

You could try compiling with --disable-debug, that might speed things up 
a bit.  But other than that, I'd strongly concur with Adam's 
recommendation to use rsync if you have control over the remote server.

But the problem is probably just that sitecopy currently runs with 
O(N^2) where N is the number of files, i.e. it scales badly to large 
number of files.

> I tried to do this by excluding all other directories like this 
> 
> exclude /home/mydir/b*
> exclude /home/mydir/c*
> exclude /home/mydir/d*

If you have "local /home/mydir" just use "exclude /b*" instead, and that 
should work as you expect; use a path relative to the local root 
directory.

joe
_______________________________________________
sitecopy maillist  -  [email protected]
http://dav.lyra.org/mailman/listinfo/sitecopy