Re: abcl.org/trac/* being spidered by multiple bots
evenson <[email protected]> Mon, 8 Jan 2024 13:11:42 +0100
| Newsgroups | gmane.editors.j.devel |
|---|---|
| Message-ID | <73EC4E6C-FA64-451A-BC49-5AB710BE95A1__42631.4729776763$1704715989$gmane$org@panix.com> |
--Apple-Mail-AD9FAB4B-9B1A-4E9E-965F-FDCA70DFB4C6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D= utf-8"></head><body dir=3D"auto"><div>On the road back to Austria today; wil= l commit when I have a chance.</div><br id=3D"lineBreakAtBeginningOfSignatur= e"><div dir=3D"ltr">Sent from my iPad</div><div dir=3D"ltr"><br><blockquote t= ype=3D"cite">On Jan 7, 2024, at 11:52, Erik Huelsmann <ehuelsmann@common-= lisp.net> wrote:<br><br></blockquote></div><blockquote type=3D"cite"><div= dir=3D"ltr">=EF=BB=BF<div dir=3D"ltr"><div>Hi Mark,</div><div><br></div><di= v><br></div><div>Last night a number of disruptions in <a href=3D"http://cli= ki.net">cliki.net</a> availability were reported, a happens regularly lately= -- and definitely during the weekly backup.</div><div><br></div><div>So I c= hecked the server this morning. The server had a load of 50 (>15 per cpu)= ; 80% of that was going to Trac, however the access log trac.common-lisp.net= _access.log was silent. Turns out all the traffic was on <a href=3D"http://a= bcl.org">abcl.org</a>. It was being spidered like crazy. Trac - or our serve= r setup (or both) - really isn't up to that task, which is why there is only= limited spidering allowed on <a href=3D"http://trac.common-lisp.net">trac.c= ommon-lisp.net</a>. (Enough to spider "current" trunk of projects and their i= ssues.)</div><div><br></div><div>To restrict spidering, I've changed the rob= ots.txt file at /project/armedbear/public_html/robots.txt.</div><div><br></d= iv><div>Could you please commit that robots.txt (or an even stricter o= ne) to the repository that the site is being synced from?</div><div><br></di= v><div><br></div><div>Thanks!</div><div><br></div><div>Regards,</div><div><b= r></div><div>Erik.<br></div><div><br></div><div><br></div></div> </div></blockquote></body></html>= --Apple-Mail-AD9FAB4B-9B1A-4E9E-965F-FDCA70DFB4C6--