Concerns about the evidence for the AI scraper problem and the resulting blocking

I have some concerns about the way the current “AI scraper” problem is
being characterised, and particularly about the measures being taken in
response to it.

I don't doubt that Sourceware has seen unusual or excessive traffic, and I
don't think it is unreasonable to investigate whether automated scraping is
contributing to load. What I am less convinced about is the extent to which
the public evidence actually supports the conclusions being drawn from that
traffic.

In particular, there seems to be very little evidence beyond assertions
that the traffic being classified as AI scraping is actually illegitimate
traffic, rather than simply automated traffic. I haven't seen request
statistics, representative logs, request patterns, resource-consumption
figures, or anything else that would allow people outside the
infrastructure team to understand what was actually happening and how
significant the problem was.

The same applies to the suggestion that AI scraping companies are obtaining
access to large numbers of residential IP addresses through libraries or
services which effectively sell users' bandwidth. That may well be what is
happening, but the discussion on the mailing lists appears to present this
as a hypothesis based on an external lead rather than something
demonstrated from Sourceware's own traffic data. I don't think we should
treat that explanation as established fact without evidence.

There is also an important distinction between seeing a large number of
requests and establishing that those requests were the cause of a
particular infrastructure problem. If the concern is resource exhaustion, I
would expect to see some indication of what resources were actually being
exhausted: CPU, database queries, bandwidth, connection counts, response
times, or whatever the relevant bottleneck was. Without that information it
is difficult to judge either the scale of the original problem or whether
the measures being taken are proportionate.

What concerns me particularly is that there is much stronger evidence for
problems caused by the mitigation itself.

Broad IP blocking has demonstrably caught legitimate users, and the
mailing-list discussions acknowledge that this is an unavoidable
consequence if residential/proxy addresses are involved. More importantly,
after Anubis was introduced there were reports of legitimate GitHub Actions
CI jobs intermittently failing to download files from Sourceware. That is
not a hypothetical concern about what might happen to legitimate users; it
is an actual example of legitimate infrastructure being affected.

This makes the chronology rather important. We have a relatively poorly
quantified and attributed problem on one side, followed by increasingly
intrusive countermeasures, and then concrete reports of legitimate traffic
being disrupted by those countermeasures.

I don't think that necessarily means the original scraping problem didn't
exist. It means that the two issues need to be kept separate.

There can be a genuine scraping problem while the response to it is still
technically misguided or disproportionately harmful. “There are bots
hitting us” doesn't by itself establish that broad IP blocking, browser
challenges, or putting Anubis in front of additional services is the
appropriate solution.

I also think there is a danger in framing this specifically as an “AI”
problem. At the HTTP level, the relevant question is whether a request is
consuming an unreasonable amount of resources, not whether it was generated
by an AI company, a conventional search engine, a CI system, a script, or a
human using curl. If the infrastructure is vulnerable to large numbers of
requests, it seems preferable to make the expensive operations resistant to
abuse through caching, rate limiting, resource quotas, endpoint-specific
controls, query limits, and similar measures, rather than trying to
determine whether the remote client is sufficiently “human”.

There is a useful counterexample here: LWN appears to have dealt with a
similar scraping problem by addressing the resource bottlenecks rather than
relying solely on identifying and blocking the scrapers. The fact that it
was possible to make the service usable again without necessarily stopping
the scraping itself seems quite relevant.

Ultimately, I think the burden of evidence should increase as the
intervention becomes more intrusive. If we're talking about quietly
rate-limiting an endpoint, that's one thing. If we're blocking large IP
ranges and putting JavaScript challenges in front of services used by
developers, CI systems and other automated tooling, it seems reasonable to
ask for evidence that:

   1. the traffic is actually abusive;
   2. it is responsible for a significant portion of the resource
   consumption;
   3. the attribution to AI/LLM scraping is justified;
   4. the proposed mitigation materially reduces the problem; and
   5. the collateral damage to legitimate users is acceptable.

At the moment, the public mailing-list discussions seem to provide
considerably more evidence for the last of those points than for the first
four.

I would therefore be cautious about presenting the current measures as an
established response to a demonstrated “AI scraper” problem. There may well
be a real underlying problem, but I don't think the evidence currently
available justifies treating the explanation for that problem, or the
necessity of the current blocking mechanisms, as settled.

In particular, I think it is worth distinguishing between “we observed
traffic we believe to be abusive” and “we have demonstrated that this
traffic is responsible for our problems and that blocking it in this
particular way is necessary”. Those are very different claims, and the
latter needs considerably more evidence.

Yours,


Nick Andrews.