Re: Correctly dealing with bots and scrapers.
Andrew Kirch via NANOG <[email protected]>
| Newsgroups | gmane.org.operators.nanog |
|---|---|
| Message-ID | <CALA5tJK5bu3cpV0tqgD60qd5g7RRaeWR6J86_JhwVYFS1TWzMw@mail.gmail.com> |
Get a small version of a very old very fast very inaccurate LLM. Have it generate a couple terabytes of endless nonsense. Redirect scrapers to it, and poison whatever LLM they are trying to train. Andrew On Wed, Jul 16, 2025 at 12:49 PM Andrew Latham via NANOG < [email protected]> wrote: > I just had an issue with a web-server where I had to block a /18 of a > large scraper. I have some topics I could use some input on. > > 1. What tools or setups have people found most successful for dealing > with bots/scrapers that do not respect robots.txt for example? > > 2. What tools for response rate limiting deal with bots/scrapers that > cycle over a large variety of IPs with the exact same user agent? > > 3. Has anyone written or found a tool to concentrate IP addresses > into networks for IPTABLES or NFT? (60% of IPs for network X in list > so add network X and remove individual IP entries.) > > -- > - Andrew "lathama" Latham - > _______________________________________________ > NANOG mailing list > > https://lists.nanog.org/archives/list/[email protected]/message/Z2J6CFBKU3ARYSQYKBRDCV6ZFUOS3WMU/ > _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/7N5RWJOU745D4GV4C3FQYQQQ2WTKOJOR/