Bug#900749: Bug #900749: [www.debian.org] webwml build takes too long

On Tue, Jun 05, 2018 at 11:42:53AM +0200, Laura Arjona Reina wrote:
>Hello all
>
>I'm providing here some info about the current situation.
>
>The last build job was done with no updates in the git repo.
>This time the webwml job took 1 hour approx:
>
>3stattrans finished (at Tue Jun  5 07:27:30 UTC 2018)
>run-parts: executing /srv/www.debian.org/cron/parts/4webwml
>4webwml finished (at Tue Jun  5 08:29:51 UTC 2018)
>
>The logs of this job can be found in:
>
>https://www-master.debian.org/build-logs/webwml/often.no-changes-git.log
>https://www-master.debian.org/build-logs/webwml/git_update.no-changes-git.log
>https://www-master.debian.org/build-logs/webwml/stattrans.no-changes-git.log
>https://www-master.debian.org/build-logs/webwml/wml_run.no-changes-git.log
>
>So I think this is the "bare minimum" that we can aim without changes to the
>makefiles or scripts.

(Sharing here for posterity...)

And I pushed more changes to the scripts and Makefiles last night to
add a shared on-disk cache of git revisions that should make a big
improvement for performance - see commits

533db96be6c Major Makefile overhauls
f035d5bf508 Move locking into the cache build utility
c9d8b4e7e2e Merge branch 'cache-files'
06a8db57de5 Tell git to ignore our on-disk cache
45fc485b91c Add a simple wrapper script to (re)generate the on-disk cache
da40997da40 Update test wrapper to test saving the cache to disk
3912e3865cb Add support for a simple on-disk cache of git revs

There's now a new order-only prerequisite on "$(VCSREVCACHE)" that
I've added in a lot of places in Makefiles to check/regenerate the
cache when running make rules to generate html files from wml
source. That seems to work well locally for me; please report any
problems you see.

I also updated ../cron/4webwml to explicitly check/build the cache
before we start running make, as that will be slightly faster.

At this point, I *think* I'm done with sciprt updates for the git
conversion itself, although obviously I'll investigate bugs as people
find/report them. :-)

There are a few other things I'm going to look at in the build scripts
- more on that separately...