Re[2]: Moving to Github ... ?

>     It seems that many sites are under pressure by IA crawlers. However it isn't proven these are attacks.

  These are not attacks, but they are indistinguishable. LLM crawlers
do not care about etiquette, robots.txt, or proper resource usage; they
hammer the server as hard as they need, and do everything they can to
avoid getting blocked, typically by running from large swaths of
residential IPs. They're not trying to pwn the server, but they know
they're doing something wrong. It is valid to consider them attacks.


>    In some cases, the bots require non-existing resources, and the servers respond by "404 Not found", but, according to RFC2616, this response is ambiguous because it does not tell wether the resource is permanently or temporarily absent, which sort of means "retry later". This might be the reason why the bots retry.

  404s are not the problem, they barely consume any resources. The
problem is cgit and bugzilla: invoking a CGI is what is costly, and
something like cgit gives a incredibly large amount of links to harvest.
That is why the busybox.net site is usable again after Denys disabled
cgit and bugzilla: bots may still hit static pages and 404s, but that
doesn't overload the server.

  Short of using a provider that has enough resources to deploy anti-
crawler measures or absorb the load, such as github (for how long? their
service level is at zero nines this year, lol), I only know of two ways
to defend against the LLM crawler spam. The first is Anubis. The second
is something I did last year and talked about on the fediverse:
https://social.treehouse.systems/@ska/115384879517972291

  Github will do for now, but I'm afraid that soon a more durable
solution will be needed. Moving to another provider could be one.

--
  Laurent