Bug#851241: www.debian.org: packages.debian.org update hangs every now and then

Cyril Brulebois <[email protected]> Tue, 24 Jan 2017 04:45:07 +0100
Newsgroups	gmane.linux.debian.devel.www
Organization	Debian
Message-ID	<20170124034507.GA32231__13400.9324283854$1485229723$gmane$org@mraw.org>
Control: tag -1 patch pending

Hi,

Rhonda D'Vine <[email protected]> (2017-01-13):
>  I got notified today that the packages website didn't update, and that
> this seems to be an issue appearing more regularly.  I didn't had much
> time unfortunately to investigate it further, but this is what appeared
> to me:
> 
> #v+
> pkg_user 21873  0.0  0.0 103296  5192 ?        S    Jän07   0:07 sshd: pkg_user@notty
> pkg_user 21887  0.0  0.0   4336   764 ?        Ss   Jän07   0:00  \_ dash -c /srv/packages.debian.org/bin/push_update
> pkg_user 21898  0.0  0.0  13348  1656 ?        S    Jän07   0:00      \_ /bin/bash /srv/packages.debian.org/bin/push_update
> pkg_user 21958  0.0  0.0   4224  1476 ?        S    Jän07   0:00          \_ run-parts --verbose /srv/packages.debian.org/cron.d
> pkg_user 22309  0.0  0.0  13344  1640 ?        S    Jän07   0:00              \_ /bin/bash /srv/packages.debian.org/cron.d/200proce
> pkg_user 25601 99.0  0.7 154500 122188 ?       R    Jän07 7804:48                  \_ /usr/bin/perl -w ./bin/parse-contents
> pkg_user 20378  0.0  0.0   4336   712 ?        S    Jän07   0:00                      \_ sh -c sort  -T /srv/packages.debian.org/tm
> pkg_user 20379  0.0  0.0   9080  3864 ?        S    Jän07   3:24                          \_ sort -T /srv/packages.debian.org/tmp -
> #v-
> 
>  The running time of parse-contents was increasing, but the sort didn't
> do anything. An strace showed that it was hanging:
> 
> #v+
> $ strace -p 20379
> Process 20379 attached
> write(1, "-data:kfreebsd-i386\nyp.margorp_e"..., 4096^CProcess 20379 detached
>  <detached ...>
> #v-
> 
>  Unfortunately I didn't had the time to investigate further, but if it
> hangs again and maybe at the same place I guess the tmp file(s) would be
> useful to investigate further.

Following a ping on #debian-devel then #debian-www I've checked what was
happening (running still Jan 15): sort was stuck on a write, sh -c stuck
on wait, and perl wasn't really doing much (no syscalls, but eating CPU,
apparently somewhere in libdb according to gdb, but I'm not sure this is
accurate / relevant).

Looking at the sizes of sid files, the aggregation is between 15-16 GB,
which might explain why piping sort's output to a perl file handle might
be suboptimal / sometimes getting stuck.

I've experimented a bit with the idea of creating a temporary file
instead, and it seems to do the job / work around this issue properly:

  https://anonscm.debian.org/cgit/webwml/packages.git/commit/?id=1b065791a50c7a0e606f2577bcae02b7c5aa190b

I've taken the opportunity to fix progress reporting, since it can be
off by a lot with the big fat files we have in today's distributions,
through a sub called from that specific part of parse-contents, but also
from earlier in the code, cheating a bit with gzip -l's help:

  https://anonscm.debian.org/cgit/webwml/packages.git/commit/?id=a45ab468c3d18ee671f1f65ef0cc64ebb8e9d408
  https://anonscm.debian.org/cgit/webwml/packages.git/commit/?id=657f137df07726c41a399652618928cdc2fe8b97

Following Paul's advice, I've pushed that to master, merged into
debian-master, and deployed it on picconi since I'm still in the right
pkg group (which dates me quite a lot but can be useful at times it
seems). I'll keep an eye on cron.log, since I've killed a number of
occurrences while I was running final tests there.

Tagging with patch & pending until we get a confirmation stuff looks
better…
 

KiBi.
signature.asc (application/pgp-signature, 819 B)