Authentication soon required to mirror the entire LVFS

Hi all,

As you might know you can create a local copy of the entire LVFS archive (o=
ver 10,000 files, and over 100GB in size) to use locally. This allows big c=
ompanies, security vendors, government agencies and educational networks to=
 deploy firmware updates privately, often to machines without public intern=
et access. There will soon be a new LVFS requirement that might affect how =
you mirror the LVFS.

Over the last few years the number of mirror clones has grown, with some ve=
ndors downloading the *entire archive* again and again, sometimes even in t=
he same 24h duration. Tools such as Pulp and the standalone sync-pulp.py[1]=
 will only download changed or new files, and so downloading the entire arc=
hive each time is totally unnecessary, hugely wasteful, and somewhat expens=
ive to provide. It's also something that badly-configured bots scraping the=
 LVFS have started doing -- which is hugely frustrating even if I can block=
 the user agent string after the event.

Whilst I encourage large companies to mirror the LVFS whenever possible, we=
 shouldn't provide TBs of bandwidth and add thousands of log entries for no=
 good reason. I=E2=80=99m going to merge a feature next week that limits th=
e number of daily downloads per-IP to 1000. To put this in perspective, usu=
ally less than 10 files are added to the stable metadata each day so any ex=
isting mirror sync scripts should not need modification. To mirror the enti=
re archive from scratch (without any kind of download limit) you will need =
to provide authentication details with your LVFS username and a user profil=
e token as the password. If anybody needs a new user account set up please =
either file an issue[2], or email me directly =E2=80=93 it=E2=80=99s comple=
tely free of charge of course.

I=E2=80=99d also ask that anyone who is currently mirroring the LVFS doesn=
=E2=80=99t start doing the process at exactly midnight UTC =E2=80=93 there=
=E2=80=99s currently quite a spike in those first few minutes of the day an=
d although the LVFS spins up additional resources as required, it would be =
much better to leave them idle and spread the load over the day.

Thanks,

Richard.

[1] https://gitlab.com/fwupd/lvfs-website/-/raw/master/contrib/sync-pulp.py
[2] https://gitlab.com/fwupd/lvfs-website/-/issues