Bug#1037490: www.debian.org: clean up old files

Cyril Brulebois <[email protected]> Wed, 14 Jun 2023 20:59:40 +0200
Newsgroups	gmane.linux.debian.devel.www
Organization	Debian
Message-ID	<20230614185940.ohyloxdekthzjefz__21285.4519065999$1686769426$gmane$org@mraw.org>
(Cc-ing Adam for DSA following the ping about disk space earlier.)

Cyril Brulebois <[email protected]> (2023-06-13):
> Spotted while working on #1037479: we have a bunch of files around that
> are no longer useful, because relevant suites are EOL and archived, and
> have been dropped from the relevant config files, templates, etc.
> 
> They could probably be cleaned up, but I don't want to rush this kind of
> things, hence this bug report.

Apparently we're going to have to do something soon, since the recent
updates in picconi have led to a noticeable and worrisome bump in
storage used:

    /dev/mapper/vg0-srv  246G  234G  822M  100%  /srv

Some cleanup happened in user directories, but I suppose it would make
sense to keep track of our disk space usage:

    /dev/mapper/vg0-srv  246G  136G  99G    58%  /srv

And beware, there are peaks during runs, with e.g. tmp/ filing up.

Also, the mirror (pkgmirror-csail) should like this as well given:

    /dev/vdb             148G  129G  12G    92%  /srv

See also graphs (with the usual guest authentication):
  https://munin.debian.org/debian.org/picconi.debian.org/df.html
  https://munin.debian.org/debian.org/pkgmirror-csail.debian.org/df.html


Our directory looked like this when I started:

    426M    ./archive
    200K    ./bin
    268K    ./cache
    8.0K    ./cgi-bin
    29M     ./conf
    64K     ./cron.d
    8.0K    ./debian
    116G    ./files
    16M     ./.git
    133M    ./home
    300K    ./lib
    14M     ./mail
    8.4G    ./mirror
    4.0K    ./po
    196K    ./static
    116K    ./templates
    7.5G    ./tmp
    2.6G    ./www
    -------------------
    135G    total

I was thinking of getting rid of these:

    $ du -sch $(find -name 'jessie*')
    65M     ./mirror/202306141247/www/jessie-backports
    17M     ./mirror/202306141247/www/source/jessie-backports
    87M     ./mirror/202306141247/www/source/jessie
    3.5M    ./mirror/202306141247/www/source/jessie-updates
    5.3M    ./mirror/202306141247/www/source/jessie-backports-sloppy
    505M    ./mirror/202306141247/www/jessie
    12M     ./mirror/202306141247/www/jessie-updates
    6.4M    ./mirror/202306141247/www/jessie-backports-sloppy
    196K    ./archive/backports/jessie-backports
    26M     ./www/jessie-backports
    5.8M    ./www/source/jessie-backports
    11M     ./www/source/jessie
    2.3M    ./www/source/jessie-updates
    3.4M    ./www/source/jessie-backports-sloppy
    176M    ./www/jessie
    3.1M    ./www/jessie-updates
    4.0M    ./www/jessie-backports-sloppy
    ----------------------------------------------------------------
    930M    total

and these:

    $ du -sch $(find -name 'stretch*')
    85M     ./mirror/202306141247/www/source/stretch
    5.1M    ./mirror/202306141247/www/source/stretch-backports-sloppy
    15M     ./mirror/202306141247/www/source/stretch-backports
    3.4M    ./mirror/202306141247/www/source/stretch-updates
    416M    ./mirror/202306141247/www/stretch
    6.2M    ./mirror/202306141247/www/stretch-backports-sloppy
    52M     ./mirror/202306141247/www/stretch-backports
    4.9M    ./mirror/202306141247/www/stretch-updates
    11M     ./www/source/stretch
    3.4M    ./www/source/stretch-backports-sloppy
    5.8M    ./www/source/stretch-backports
    2.2M    ./www/source/stretch-updates
    179M    ./www/stretch
    3.9M    ./www/stretch-backports-sloppy
    22M     ./www/stretch-backports
    2.6M    ./www/stretch-updates
    -----------------------------------------------------------------
    813M    total

But those easy ones would only account for 1.7G combined, as mentioned
yesterday.

Searching for files containing but not starting with either jessie or
stretch, there are a lot of other things that looked like they could go
away:
 - filelists_*
 - filenames_*
 - new_package_info_*
 - package_names_*
 - packages_all_*
 - reverse_*
 - source_names_*
 - sources_all_*

and I expected no collateral damages.

To be on the safe side, I created two tarballs, one for jessie, one for
stretch, containing all those files, before removing them. I've compressed
them so that they take less space than actual files, moved them in a
directory that shouldn't be mirrored to pkgmirror-csail, and I've
scheduled their deletions in 6 months (via at), which should leave plenty
of time to restore them if needed, while making sure full cleanup happens
eventually.

    pkg_user@picconi:/srv/packages.debian.org$ echo 'rm -rv /srv/packages.debian.org/obsolete/' | at 'now + 6 months'
    warning: commands will be executed using /bin/sh
    job 5 at Thu Dec 14 18:56:00 2023
    pkg_user@picconi:/srv/packages.debian.org$ atq
    5	Thu Dec 14 18:56:00 2023 a pkg_user

Also removed, a long obsolete file:

    -rw------- 1 pkg_user pkg_maint 7.5G May  2  2021 tmp/sort01fAge

Not sure why this file stays behind, but at least the name is stable, so
the non-freed space is getting reclaimed when the next run starts:

    -rw-r--r-- 1 pkg_user pkg_maint 14794575872 Jun 14 17:37 tmp/sid.sorted

This is rather strange since the commit adding this temporary file aimed
at removing it properly, and should error out if it fails to do so (which
cron logs claim is not happening):
  https://salsa.debian.org/webmaster-team/packages/-/commit/1b065791a50c7a0e606f2577bcae02b7c5aa190b


At the moment, I'm not sure the file deletion (jessie, stretch) is going
to be mirrored to pkgmirror-csail automatically. A sync just finished, but
it might have started before the actual deletion happened. Let's keep an
eye on this after the next one…


Cheers,
-- 
Cyril Brulebois ([email protected])            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
signature.asc (application/pgp-signature, 833 B)