Re: Concerns about the evidence for the AI scraper problem and the resulting blocking

Hi Kalamatee,

On Mon, 2026-08-24 at 14:16 +0100, Kalamatee via Gcc wrote:
>    1. the traffic is actually abusive;
>    2. it is responsible for a significant portion of the resource
>    consumption;
>    3. the attribution to AI/LLM scraping is justified;

It is really abusive. See for example this analysis from our logs:
https://xeiaso.net/notes/2026/check-your-smart-tv/
Note that the article says it is for the last few months, the logs were
actually for the last few weeks..., so it is way worse. But it is so
bad that it almost doesn't matter whether it is millions, tens of
millions or hundred of millions of distinct IP addresses hammering our
servers.

>    4. the proposed mitigation materially reduces the problem; and
>    5. the collateral damage to legitimate users is acceptable.

The mitigations do really reduce the problem, resource use is ten to
hundred times less. And "registered" developers can use the services
again. But of course the collateral damage to users is never
acceptable.

With the latest round of nginx limiters we were hoping that we would
accumulate enough "good" ips that we could easily let through while
limiting others/unknown ones. This only works partially, we are
currently marking ~1350 ips as "good". This is a significant number of
real developers, but clearly is way too low a number to cover other
users.

We have some ideas to a) make it easier to get users register their ips
as good, and b) split the limiter between services so that an attack on
say the gcc cgit service (which is currently ongoing) doesn't impact
e.g. sourceware bugzilla.

Sorry this is impacting you.

Cheers,

Mark

P.S. Note that it looks like the AI scraperbots haven't found the forge
yet (and I am really hoping I am not jinxing it now...) so you will
have a much nicer experience using https://forge.sourceware.org/gcc/gcc