Re: Trac issues once again

On Sun, 06 Sep 2009 07:32:23 +0100, Lester Caine <[email protected]>
wrote:
> [email protected] wrote:
>> The msnbot has decided to index phpeclipse.com website. This is killing 
>> the server and causing trac to consume all memory in the box. I have 
>> given the box 3Gig of RAM, it worked for a while but now is being a 
>> problem again. I have a few thoughts on solutions for this. (i did 
>> enable a memory check we created some time ago to check the system 
>> memory and if it fell below a level to kill all apache processes).
>> 
>> 1. Put another server up and load balance the two. I would move trac 
>> from using SQLite to PGSql so i could get both servers to read the 
>> database, however this may be possible using sqlite and an nfs or iscsi 
>> share. Input welcomed.
>> 
>> 2. Move Trac website back to SF. They support trac now. Don't know how 
>> much work this would be, and don't know if this is the correct solution.

>> We may just be moving the problem around.
>> 
>> 3. Move away from trac to some other tool that knows how to manage it's 
>> memory. I have no idea on this one.
>> 
>> 4. Just hope our little script works and leave it as is.
>> 
>> That's all i have. Just let me know what you think would be a good 
>> solution for this problem. Also the httpd processes consume huge amounts

>> of CPU. I could give the box another CPU. I don't know how much that 
>> would help. It would help, just how much of a help would it be? Input is

>> wanted.
> 
> Or just disable msnbot via robots.txt?
> 
> Half the trouble seems to be that these search engines trawl EVERYTHING, 
> so every historic link with the result that every single update has a 
> search link created.
> At one point I had 60 search engines all trawling my server, once one of 
> the big boys lists you, all of the others join the band wagon :( So 
> although it would possibly be nice that a search on one of them produces 
> a result - is the cost of supporting them in computer power and 
> bandwidth worth it?
> 
> My local machines simply has a deny all robots.txt, while the ones at 
> 1&1 have just a few select ones allowed nowadays - and nofollow helps on 
> and historic stuff
> http://blog.searchenginewatch.com/050118-204728

I have added a robots.txt file. It has the following content
User-agent: *
Disallow: /browser
Disallow: /log
Disallow: /changeset
Disallow: /report
Disallow: /newticket
Disallow: /search

I also had to add the following to the conf file for phpeclipse domain
<Location "/robots.txt">
SetHandler None
</Location>

This is after the mod_python stuff.

I checked this internal before i allowed the firewall to pass traffic back
to the site, and it worked.

However i see the bot getting the robots.txt file
65.55.209.70 - - [06/Sep/2009:09:36:52 -0500] "GET /robots.txt HTTP/1.0"
200 126 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
Now i won't see that host come around again, but i get the others
65.55.106.162 - - [06/Sep/2009:09:51:47 -0500] "GET
/browser/trunk/net.sourceforge.phpeclipse/src/net/sourceforge/phpeclipse/phpeditor/php/HTMLCompletionProcessor.java?rev=117
HTTP/1.0" 200 37604 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.159 - - [06/Sep/2009:09:51:48 -0500] "GET
/browser/trunk/net.sourceforge.phpeclipse.phpmanual.htmlparser/build.properties?rev=1582
HTTP/1.0" 200 13347 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.182 - - [06/Sep/2009:09:51:49 -0500] "GET
/browser/trunk/net.sourceforge.phpeclipse.debug.core?rev=1441 HTTP/1.0" 200
21769 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

i guess it will take some time for them to get the robots.txt file. Until
then phpeclipse.com will be slow or down completely.

However if i have configured the robots.txt or apache setup wrong let me
know.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july