Re: multiprocess

On Thu, Sep 23, 2010 at 02:15:09AM +0100, Joe Orton wrote:
> > So I have a feature request, if it is possible to make without too much
> > effort. It would be good if it was possible to use multiple cores to
> > calculate the md5's. As my current cpu has 4 cores it would speed up the
> > process a lot. And as new cpus are getting more and more cores it would
> > in the long run be an even better improvement.
> 
> sitecopy is not designed for large sites.  The scanning process is very 
> linear; it's not trivial to break it up such that multiple checksum 
> operations could be done simultaneously.

If it really needed to be done, it could be done by first making a
prescan finding all the files. Having that in a list it would be easy
for an external program to make the md5's in parallel. Then the second
and actual scan would have to use the output of the external program
instead of calculating the md5.

> You could use multiple rcfile entries for your site and use multiple 
> invocations of sitecopy.

I have thought about that, but it is not that easy as one of the places
that grows is a place where I just put data in a flat structure.

> If you are really CPU bound rather than I/O bound I'd expect the easiest 
> performance win would be to switch lib/neon/ne_md5.c to use the OpenSSL 
> MD5 API; that uses hand-tuned native assembler on most platforms and is 
> much faster than C code.
> 
> This would be a nice project to try if you were willing to have a go.

That sounds good, I might try that.

/Anders