Re: Moving to Github ... ?

On Fri, 15 May 2026 11:16:43 +0000
Laurent Bercot via busybox <[email protected]> wrote:

> >The problem with blocking IP addresses is that large chunks of the
> >world are behind 'carrier grade' NAT.
> >
> >So you block all the users of the ISP.  
> 
>   You mean, if the IP gets dynamically reallocated to another user after
> having been used by a LLM crawler bot?

That as well.

But I believe that in a lot of world (especially outside the US and Europe)
ISPs don't give out 'real' IPv4 addresses.
Instead they give out (eg) 10.x.x.x addresses and use NAT so that a lot of
users are concurrently using the same IPv4 address.

So even if you use a sensible expiry time for the blocks you could be
cutting of a large chunk of users in one go.

> 
>   That is unfortunate.
> 
>   Maybe the burden of fixing this should be on the ISPs that reallocate
> IP addresses in the year of our Lord 2026, rather than on the person
> trying to protect their server.

A lot of people see the dynamic addresses as a benefit - they make it
marginally more difficult to track users between sessions.

>   Maybe there should even be a class action against crawler bots that
> squat residential IP addresses, misusing the commons. You know?

They may not even be 'crawler bots' - just people requesting the LLM
do something for them that means it hits your server hard (I've no idea).
But I'd have thought bots looking for training data would come from a
much smaller number of IP addresses.

Did you look at expiring the blocks after a timeout (put the time in the
symlink data?) rather than using a permanent block?

BTW the usual way to avoid directories with millions of files is to use the
first 3 or 4 characters to select a subdirectory.
That can bring it all under control.

But be careful of the size of the inode reference count.
A certain internet payment company had serious issues with a directory
that held a subdirectory for each company when it hit 32k companies.
Solaris just crapped all over the filesystem :-)

-- David


> 
> --
>   Laurent
> 
> _______________________________________________
> busybox mailing list
> [email protected]
> https://lists.busybox.net/mailman/listinfo/busybox