Re: cvsweb anti-bot protection (was: Retrieving MAC address from struct ifnet)

[Christof]
> Personally, I am seeing gptbot crawling at a rate of up to about 1
> request per second.

[Joerg]
> My own web sites see moderate scraper traffic, but they don't have a
> large site graph either.  On various other sites like anonhg.n.o, the
> main Mercurial bug tracker etc we have been observing persistent high
> load.

Yeah, and I'm seeing some crawlers, but nothing even I would call high
load, much less what something like cvsweb or anonhg would...even
though I have, from a Web client's perspective, *lots* of source trees,
including many (slightly different) NetBSD source trees.

Makes me wonder what the relevant difference is.  Border router blocks?
Lack of HTTPS support?  They don't recognize the interface I present?
They just haven't noticed me yet?

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email protected]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B