Concerns about the evidence for the AI scraper problem and the resulting blocking

Hello,

I'm Xe Iaso, the creator of the Anubis project. As such, note my bias 
when I am talking about issues related to this flavour of problems.

I have been working with the Sourceware team for over a year to try and 
minimize impact on users and help notify them of emergent threats as 
they pop up. This includes a lot of behind the scenes advocacy that is 
not generally visible to the public, such as the (futile) work I've been 
doing to try to get the Lightpanda team to respect RFC 9309 among other 
internet standards.

As an aside, while I was reading your email, I noticed signs that it was 
in part generated using a generative AI tool. Charitably, I assume you 
did this to translate your original message from your native language so 
that your email looks "professional". In the future, if you are going to 
use AI to translate emails, please mark it as so and attach the original 
version in your native language.

On 8/24/26 9:16 AM, Kalamatee wrote:
> I have some concerns about the way the current “AI scraper” problem is
> being characterised, and particularly about the measures being taken in
> response to it.

I don't like that this has to be the option either. I have tried many 
things to reduce the level of paranoia involved, but at some level this 
is becoming a big data problem and I really don't want to receive 
additional threats in order to collect that data to make a better 
option. I'm working on a reputation database that I will talk about more 
in the future.

> I don't doubt that Sourceware has seen unusual or excessive traffic, and I
> don't think it is unreasonable to investigate whether automated scraping is
> contributing to load. What I am less convinced about is the extent to which
> the public evidence actually supports the conclusions being drawn from that
> traffic.

As a rule of thumb: if multiple organizations are all coming to the same 
conclusions without colluding, it's probably because a thing is actually 
happening. I have no way to prove this in a way you'll likely accept, 
but I honestly don't have the time or energy to run a grand conspiracy 
like you're imagining.

I would like to know what your hypothesis is as to the source of this 
traffic.

> In particular, there seems to be very little evidence beyond 
> assertions that the traffic being classified as AI scraping is
> actually illegitimate traffic, rather than simply automated traffic. I
> haven't seen request statistics, representative logs, request
> patterns, resource-consumption figures, or anything else that would
> allow people outside the infrastructure team to understand what was
> actually happening and how significant the problem was.

Pedantically, yes, there is no way to prove that these scrapers are for 
getting _generative AI training data_ in particular. The nature of a 
bunch of random Googles Chrome coming from random residential IP ranges 
globally means it's hard to track the threat actors down.

However based on the few scrapers that are honestly identifying 
themselves (eg: the x.ai scraper that seemed precision-designed to make 
GitLab/Gitea/Forgejo servers fall over, or the AmazonBot run by the 
Alexa team) this seems to be a non-unreasonable conclusion.

In this era of internet, everything has to be C10K ready[3] and a lot of 
critical infrastructure just isn't. This isn't something that can be 
easily solved overnight. This is a big enough problem that GitHub and 
other key services are falling over from the load too. Why do you think 
everything is becoming login only? Hint: it's not for fun.

The only real assumption that a lot of us are left with is that it's 
being done by AI companies in an effort to strip the Internet for parts 
to gather as much training data as possible in order to brute force 
improvements.

Speaking as someone that genuinely does have domain expertise in AI 
usage and training in general, this is one of the few methods of getting 
foundation model training data left. The others include scan -> shred 
pipelines to ingest books picked for pruning from libraries and unsold 
inventory from bookstores.

Stripping the Internet for parts is the only real thing left because it 
requires the least investment and capitalism optimizes for low cost.

> The same applies to the suggestion that AI scraping companies are obtaining
> access to large numbers of residential IP addresses through libraries or
> services which effectively sell users' bandwidth. That may well be what is
> happening, but the discussion on the mailing lists appears to present this
> as a hypothesis based on an external lead rather than something
> demonstrated from Sourceware's own traffic data. I don't think we should
> treat that explanation as established fact without evidence.

Speaking as someone with half their foot in the AI industry, people 
actually do this. There are tutorials spread with paid notion notebook 
shares that go into explicit detail on how to do this. When the data[1] 
shows 2.6 million IP addresses covering 229 countries (for context: ISO 
3166-1 comprises 249 geographic entities, of which 193 are members of 
the United Nations) and 21,116 ASNs. If you look with your eyes, you can 
plainly see that a good fraction of those ASNs account for residential 
Internet service.

Given the data, it is not unreasonable to conclude that some kind of 
proxy service that terminates traffic through residential addresses is 
being used. Security researcher Brian Krebs has been tracking this and 
at the very least LG is trying to tackle the problem of people 
installing apps turning their TVs into zombies[2].

If this is all lies, then there sure has been a lot of effort by someone 
to make everyone believe this instead of the more likely explanation: 
it's actually happening.

> There is also an important distinction between seeing a large number of
> requests and establishing that those requests were the cause of a
> particular infrastructure problem. If the concern is resource exhaustion, I
> would expect to see some indication of what resources were actually being
> exhausted: CPU, database queries, bandwidth, connection counts, response
> times, or whatever the relevant bottleneck was. Without that information it
> is difficult to judge either the scale of the original problem or whether
> the measures being taken are proportionate.

One of the main ways resources are being exhausted is that git blame 
queries can take up gigabytes of ram and consume entire CPU cores to 
answer a single query that links to other pages that will also consume 
gigabytes of ram and consume entire CPU cores.

Based on my experiments with the Anubis honeypot subsystem[4] (created 
to figure out how widespread the problem actually is), every single 
request comes from a separate residential IP address. This makes any 
kind of per-IP rate limiting pointless. All rate limiting approaches 
assume there is some kind of pattern between traffic and its source. If 
there is no pattern beyond "the traffic comes from the Internet", all 
attempts to rate limit or throttle based on IP address pointless.

Hell, storing them in ram or trying to iptables them is also pointless 
as the spread of IP addresses is sparse enough that it will create a 
kernel ram exhaustion scenario. When building my own experimental 
databases to try and at least track previously used residential proxies, 
collapsing entries down to the minimal BGP routable units managed to 
save me over 400Mi in maxmind format on disk and 6Gi of ram when 
building those databases. I am still trying to implement this so that 
there can be less user disruption in general, but this is hard.

> There is a useful counterexample here: LWN appears to have dealt with a
> similar scraping problem by addressing the resource bottlenecks rather than
> relying solely on identifying and blocking the scrapers. The fact that it
> was possible to make the service usable again without necessarily stopping
> the scraping itself seems quite relevant.

I'm sure the Sourceware team would appreciate your contributions in 
removing the bottleneck. Why are you not leading with an invitation to 
contribute to the solution instead of complaining about the solution 
that was already put in place?

A really easy win would be to provide an API, UI, and link-compatible 
drop-in replacement for Bugzilla that isn't a CGI script requiring 
mod_perl to run and actually implements caching in a sensible way. That 
alone would account for a vast majority of the MySQL and Apache load of 
the Sourceware servers. Even better if you can edit the cgit / gitweb 
interface to not link to git blame without authentication.

I will provide the Sourceware team a script to fetch the IP addresses 
for GitHub Actions and a few other CI providers so that they can get 
through with minimal interference. If you know of any CI providers that 
are having issues, please link them so they can be allowlisted.

> I would therefore be cautious about presenting the current measures as an
> established response to a demonstrated “AI scraper” problem. There may well
> be a real underlying problem, but I don't think the evidence currently
> available justifies treating the explanation for that problem, or the
> necessity of the current blocking mechanisms, as settled.
> 
> In particular, I think it is worth distinguishing between “we observed
> traffic we believe to be abusive” and “we have demonstrated that this
> traffic is responsible for our problems and that blocking it in this
> particular way is necessary”. Those are very different claims, and the
> latter needs considerably more evidence.

These two arguments read like they were hallucinated by an AI model.

Please provide the original text used when prompting your model and 
details about which AI setup was used to fabricate the utterances in 
your email.

Xe Iaso
(zi yah-so)
[email protected]

.i la budza pu cusku lu
  <<.i ko snura .i ko kanro
    .i ko panpi .i ko gleki
  >>li'u vau iau

[1]: https://xeiaso.net/notes/2026/check-your-smart-tv/full-table/
[2]: 
https://krebsonsecurity.com/2026/07/lg-to-ban-residential-proxies-from-smart-tv-apps/
[3]: https://en.wikipedia.org/wiki/C10k_problem
[4]: https://anubis.techaro.lol/docs/admin/honeypot/overview

PS: I'm sorry if I come across as angry at all in my email. I genuinely 
want to help make the Internet a better place but I work two full time 
jobs and have been recovering from medical leave earlier this year. I'm 
at least no longer in constant pain, so that's a plus.

PS the second: Sorry Kalamatee, I am new to sending emails to mailing 
lists and assumed that pressing "Reply" in my mail client like I have my 
entire career would result in the mailing list being CC'd too.