Re: Linux 7.2

Computer Nerd Kev <[email protected]> wrote:
> Rich <[email protected]> wrote:
>> The pixie girl is there to *try* to halt the onslaught of the AI 
>> companies that do abjectly stupid things like download every version 
>> of linux on kernel.org from 0.0.1 through to 7.2 278 times a day, 
>> every day, times 75 AI companies to feed to their AI models.
>> 
>> I setup a really simple markov chain text generator as an AI honey 
>> pot on a server I admin and linked it to one, hidden (display: none; 
>> in the css) link on the page footer, with a robots.txt saying "don't 
>> access this link".
> 
> You don't need to try hard to trap one of these crawlers like that.  
> Just enable PHPSESSID on a PHP website and every time it gets a new 
> session variable it will think it's found a whole new website, while 
> remembering to still go back and re-crawl the site with all the old 
> PHPSESSIDs it's ever received.  "AmazonBot" is still trying old 
> PHPSESSID URLs on one site many months after I disabled generating 
> them.

Don't have php running on the site, and wanted a "poison honeypot" to 
fuck with their models.

> It's more than a little ironic that the "AI revolution" seems to have 
> born a whole new level of idiocy in web crawling.  We'll just have to 
> keep waiting until their money runs out I suppose...
> 
>> That's why the pixie girl is there on kernel.org.  You can like or 
>> dislike it, but the "burn" from the AI companies irresponsible accesses 
>> is the underlying cause.
> 
> But how much "burn" for real users of such websites is acceptable?

That's a question only the actual sites can answer.  But from what has 
been reported, the AI company "burn" is orders of magnitude greater 
than the "real user burn".  As in, maybe a site has 1000 real users, 
browing 2000 pages a day.  Meanwhile, they get hit by 200,000 AI 
crawlers retreiving all 4000 pages on the site every single crawl, for 
800 000 000 pages served a day to the irresponsible AI companies.

> They could password-protect the whole site and send out passwords
> to just the people who they think matter, telling everyone else to
> bugger off.

They could...  And sites that are "members only" like this are, mostly, 
immune from the AI crawl mess.  Other than returning 200,000 403 not 
authorized pages to the 200,000 fetches the AI bot's *still* perform, 
because they don't respect robots.txt or 403's or anything for that 
matter.  So there's still a low level of server load and of bandwidth 
traffic just to tell the AI crawler "you can't see what is here".

> Somewhere between that and trying to optimise/upgrade their website 
> to handle the new AI load without adding any new access restrictions 
> is the "pixie girl" solution which just tells people like me using 
> obscure web browsers to bugger off.  I still won't be OK with 
> effectively being told to bugger off, especially by an organisation 
> with as much money as The Linux Foundation (though browsing their Git 
> repos is still working for me in Dillo).

The problem is the AI bots are destroying the "freely open web for 
humans".  These websites putting up the pixel girl (or the other 'are 
you a robot' checks) are trying to remain "open to humans" while 
becoming "closed to AI crawlers".  The problem there lies in the fact 
that there really are no "humans" on the internet, it is 100% machines, 
just some of the machines are direct controlled by humans (web 
browsers) and some are controled by other machines.  So there's no "you 
are a human" signal that can be given, with the result that we get 
these "are you a robot" things that try to detect "human controled 
machines" from "machine controlled machines".

While I don't like the damn things any more than anyone else here, at 
least the pixie girl, while looking like an adolescent prank, does not 
often force me to "click on all the busses" three times in a row (after 
clicking on "all the busses", because it does not realize the panel 
truck in frame 5 of 9 is **not a bus**) followed by select the bicycles 
(and now it thinks motorcycles are bicycles) just to get past its 
screening.