Re: multiprocess

On Wed, Sep 22, 2010 at 07:56:04PM +0200, Anders Bo Rasmussen wrote:
> I use sitecopy to update my webpages and find it real useful.
> 
> As my main site is around 19GB it takes a long time to make the md5's.
> Sitecopy uses 100% cpu on one core in around 2.5 minutes. 
> 
> So I have a feature request, if it is possible to make without too much
> effort. It would be good if it was possible to use multiple cores to
> calculate the md5's. As my current cpu has 4 cores it would speed up the
> process a lot. And as new cpus are getting more and more cores it would
> in the long run be an even better improvement.

sitecopy is not designed for large sites.  The scanning process is very 
linear; it's not trivial to break it up such that multiple checksum 
operations could be done simultaneously.

You could use multiple rcfile entries for your site and use multiple 
invocations of sitecopy.

If you are really CPU bound rather than I/O bound I'd expect the easiest 
performance win would be to switch lib/neon/ne_md5.c to use the OpenSSL 
MD5 API; that uses hand-tuned native assembler on most platforms and is 
much faster than C code.

This would be a nice project to try if you were willing to have a go.

Regards, Joe