Re: Efficiency with large directories

On Thu, Mar 22, 2007 at 10:00:46AM -0700, David King wrote:
> I started running Sitecopy last night, and noticed that nearly 8  
> hours later it hadn't started uploading files. This is a large  
> directory (67 GB in 452,000 files over 95,000 directories), so it  
> doesn't surprise me that it takes a while, but it does surprise me  
> that it doesn't do some of the directory-walking and uploading in  
> parallel (since these block on separate I/O devices), and it also  
> surprises me that it takes that long to walk the tree at all (since  
> 'find $DIR -type f' takes about 4 minutes to complete, as does mtree,  
> which reads out more metadata than Sitecopy does).

Yeah, sorry, the sitecopy files list code is O(N^2) for N=total number 
of files in site, it just won't work for a site that big.

(With some hash tables added it could scale reasonably well; it wouldn't 
be a great deal of work)

Regards,

joe
_______________________________________________
sitecopy maillist  -  [email protected]
http://dav.lyra.org/mailman/listinfo/sitecopy