Re: Re[2]: Moving to Github ... ?

On 14 May 2026 at 17:19, Laurent Bercot via busybox wrote:

To:             	[email protected]
Subject:        	Re[2]: Moving to Github ... ?
Date sent:      	Thu, 14 May 2026 17:19:39 +0000
From:           	Laurent Bercot via busybox 
<[email protected]>
Send reply to:  	Laurent Bercot <[email protected]>

> 
> >     It seems that many sites are under pressure by IA crawlers. However it isn't proven these are attacks.
> 
>   These are not attacks, but they are indistinguishable. LLM crawlers
> do not care about etiquette, robots.txt, or proper resource usage; they
> hammer the server as hard as they need, and do everything they can to
> avoid getting blocked, typically by running from large swaths of
> residential IPs. They're not trying to pwn the server, but they know
> they're doing something wrong. It is valid to consider them attacks.
> 
> 
> >    In some cases, the bots require non-existing resources, and the servers respond by "404 Not found", but, according to RFC2616, this response is ambiguous because it does not tell wether the resource is permanently or temporarily absent, which sort of means "retry later". This might be the reason why the bots retry.
> 
>   404s are not the problem, they barely consume any resources. The
> problem is cgit and bugzilla: invoking a CGI is what is costly, and
> something like cgit gives a incredibly large amount of links to harvest.
> That is why the busybox.net site is usable again after Denys disabled
> cgit and bugzilla: bots may still hit static pages and 404s, but that
> doesn't overload the server.
> 

Believe the 404 errors were caused the gitc shutdown that causes 
the git links and patches lines to not be created, and thus they are 
405 error

This creates a version of the html file without the 404 links.

curl -s https://busybox.net/ | sed -E 
'/href=.*(fixes-|__stable|_stable)/d' > busybox_clean.html

This command clears the how to patch message as well.

curl -s https://busybox.net/ | sed -E 
'/href=.*(fixes-|__stable|_stable)|fix.html/d' > busybox_clean.html
diff busybox.html busybox_clean.html

Opening the create file removes thos link. More detail in ealier 
message sent to list.

firefox busybox_clean.html






>   Short of using a provider that has enough resources to deploy anti-
> crawler measures or absorb the load, such as github (for how long? their
> service level is at zero nines this year, lol), I only know of two ways
> to defend against the LLM crawler spam. The first is Anubis. The second
> is something I did last year and talked about on the fediverse:
> https://social.treehouse.systems/@ska/115384879517972291
> 
>   Github will do for now, but I'm afraid that soon a more durable
> solution will be needed. Moving to another provider could be one.
> 
> --
>   Laurent
> 
> _______________________________________________
> busybox mailing list
> [email protected]
> https://lists.busybox.net/mailman/listinfo/busybox


+------------------------------------------------------------+
 Michael D. Setzer II - Computer Science Instructor (Retired)     
 mailto:[email protected]                            
 mailto:[email protected]
 mailto:[email protected]
 Guam - Where America's Day Begins                        
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
+------------------------------------------------------------+