Re: Optimizing MD4

Andrew Stubbs wrote:
>> I would be interested in your thoughts on how to speed that part up.
>
> My implementation, which does a bunch of other things besides, hence why
> it's not fit to post[*], launches a background task which creates a unix
> domain socket in the cache directory (the windows version uses plain old
> TCP).
>
> Each invocation of ccache then connects to that socket and asks the
> daemon to do the MD4 scan on its behalf. The daemon checks the mtime on
> the file and serves the MD4 from its memory cache if nothing has
> changed. The stat call could probably be optimized away if the cache is
> very fresh (<1s?)

So basically something similar to the "sloppiness" file_stat_matches ?
Compare size/mtime/ctime, rather than rehashing the content of a file.

> In theory, what you get is ccache spending less time in MD4, but more
> time in I/O wait. It does seem to be faster, over all, but that might
> depend on your hardware.
>
> However, even if the latency of each ccache invocation is the same, the
> fact that they're basically idle means you can usefully crank up the
> parallelism for all but the initial build.
>
> You could, in principle, use this communication to limit how many
> cache-miss compilations are permitted to run in parallel, and therefore
> run "make -j" for maximum parallelism without fear of melting your memory.

Something like that is what I meant with a new "prefix" wrapper for cpp.
Similar to the current wrapper for cc, which does that with e.g. distcc.

> Unfortunately, I've moved on to other projects and don't have much time
> to work on this stuff any more.

Thank you for your ideas. Will check the code out too if I get the time.
It seems that there are some opportunities left for faster manifest/cpp.

/Anders