Bug#977006: packages.debian.org: filelist is currently broken for some packages

Ok, so more debug info, as apparently as a DD I do have access to
picconi.debian.org

1. The file list data is stored in Berkeley DBs in
srv/packages.debian.org/files/db/contents/filelists_${suite}_${arch}.db

2. The packages I referenced before are missing, but only in certain
archs. For example, both "supysonic" and "isbg" are arch "all" packages,
but only end up in:

* filelists_sid_alpha.db
* filelists_sid_riscv64.db
* filelists_sid_ppc64.db
* filelists_sid_sh4.db
* filelists_sid_x32.db
* filelists_sid_hppa.db
* filelists_sid_sparc64.db
* filelists_sid_m68k.db

libprismatic-schema-clojure (also arch "all) ends up in:

* filelists_sid_alpha.db
* filelists_sid_riscv64.db
* filelists_sid_ppc64.db
* filelists_sid_sh4.db
* filelists_sid_x32.db
* filelists_sid_hppa.db
* filelists_sid_sparc64.db
* filelists_sid_m68k.db
* filelists_sid_mips.db
* filelists_sid_hurd-i386.db
* filelists_sid_powerpcspe.db
* filelists_sid_kfreebsd-amd64.db
* filelists_sid_kfreebsd-i386.db

So it's not a fixed list of archs being processed correctly.

3. _Something_ clearly is wrong with the filelists.db files generated,
as some are _much_ smaller that other. For example, one would expect
amd64 to be the largest file...

31M	../filelists_sid_all.db
31M	../filelists_sid_armel.db
31M	../filelists_sid_s390x.db
32M	../filelists_sid_mips64el.db
32M	../filelists_sid_mipsel.db
33M	../filelists_sid_armhf.db
34M	../filelists_sid_ppc64el.db
35M	../filelists_sid_i386.db
36M	../filelists_sid_amd64.db
81M	../filelists_sid_s390.db
84M	../filelists_sid_ia64.db
89M	../filelists_sid_sparc.db
107M	../filelists_sid_powerpc.db
110M	../filelists_sid_hurd-i386.db
111M	../filelists_sid_kfreebsd-amd64.db
111M	../filelists_sid_kfreebsd-i386.db
117M	../filelists_sid_arm64.db
121M	../filelists_sid_mips.db
122M	../filelists_sid_sh4.db
123M	../filelists_sid_powerpcspe.db
127M	../filelists_sid_hppa.db
127M	../filelists_sid_m68k.db
128M	../filelists_sid_alpha.db
128M	../filelists_sid_riscv64.db
128M	../filelists_sid_sparc64.db
129M	../filelists_sid_x32.db
131M	../filelists_sid_ppc64.db

4. /srv/packages.debian.org/bin/parse-contents L101-181 is the code
responsible for generating the filelists.db files.

5. /srv/packages.debian.org/bin/daily is the script calling
parse-contents. My guess is it's called by a cron job or triggers on an
archive refresh, but I don't have the permissions to find out.

6. looking at the cron logs in /srv/packages.debian.org/files/logs,
parse-contents seems to run correctly and update the filelists on a
frequent basis without outputting errors in the log.

7. the local debian mirror seems to be up to date, so the error doesn't
come from there.

As to why those packages aren't listed in the filelists, no clue!

Hopefully that'll help someone if I don't get around fixing this bug
myself (chances I won't...)

-- 
  ⢀⣴⠾⠻⢶⣦⠀
  ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
  ⢿⡄⠘⠷⠚⠋   [email protected] / veronneau.org
  ⠈⠳⣄