Re: Concerns about the evidence for the AI scraper problem and the resulting blocking

Hi, Nick -

Thank you for your inquiry.

> I have some concerns about the way the current “AI scraper” problem is
> being characterised, [...]  In particular, there seems to be very
> little evidence beyond assertions that the traffic being classified as
> AI scraping is actually illegitimate traffic, rather than simply
> automated traffic.

I too have doubts about the exact nomenclature.  I don't have a problem
in principle with AI tools or automation accessing our services, in
legitimate ways for users/developers.  However, the recent access
pattern is evidently not that.  Receiving a firehose of O(10^6)
different one-shot-IP requests per day overwhelms in terms of esp. CPU
consumption of servicing bugzilla / git.  So we are stuck with
throttling accesses by our best guess at reputation heuristics.


> [...] If we're blocking large IP ranges and putting JavaScript
> challenges in front of services used by developers, CI systems and
> other automated tooling [...]

We're building out mechanisms for making it easier identification of IP
addresses and access patterns as high-reputation.  Please offer
suggestions here (or better, [email protected]).

- FChE