AW: Need help with (really) huge site

Hello Joe!

Thank you for the fast help! 

I tried your hint "exclude /b*" ...and it works pretty fine ...and really fast for my about 150 directories with the letter 'a'. 

...sometimes I should read the manuals more precisely - but time is an issue as you might know yourself - so you were right, that I didn't dig through the "state timesize" directive yet ...but I'll later this evening.

By the way - I'd love to use rsync - but unfortunately my hosting company just gives me a 'naked' FTP access to my backup server.

Many thanks,
Kai ;o)

-----Ursprüngliche Nachricht-----
Von: Joe Orton [mailto:[email protected]] 
Gesendet: Dienstag, 3. Januar 2006 21:11
An: Kai Rittmeyer
Cc: [email protected]
Betreff: Re: [sitecopy] Need help with (really) huge site

On Tue, Jan 03, 2006 at 07:14:25PM +0100, Kai Rittmeyer wrote:
> I have a directory with about 5000 sub-directories, each of these
> sub-directories has 3 sub-directories and these contain about 100
> .jpg-files each. I gave sitecopy about 1 day of CPU time - then my SSH
> client shutdown due to a reconnect - I could run it via cron - but 1 day
> is just too long for a single job. 

So about 1.5 million files in total?  You are using "state timesize" (or 
no state directive at all), I presume?

You could try compiling with --disable-debug, that might speed things up 
a bit.  But other than that, I'd strongly concur with Adam's 
recommendation to use rsync if you have control over the remote server.

But the problem is probably just that sitecopy currently runs with 
O(N^2) where N is the number of files, i.e. it scales badly to large 
number of files.

> I tried to do this by excluding all other directories like this 
> 
> exclude /home/mydir/b*
> exclude /home/mydir/c*
> exclude /home/mydir/d*

If you have "local /home/mydir" just use "exclude /b*" instead, and that 
should work as you expect; use a path relative to the local root 
directory.

joe



_______________________________________________
sitecopy maillist  -  [email protected]
http://dav.lyra.org/mailman/listinfo/sitecopy