Re: Trac issues once again

[email protected] wrote:
> The msnbot has decided to index phpeclipse.com website. This is killing 
> the server and causing trac to consume all memory in the box. I have 
> given the box 3Gig of RAM, it worked for a while but now is being a 
> problem again. I have a few thoughts on solutions for this. (i did 
> enable a memory check we created some time ago to check the system 
> memory and if it fell below a level to kill all apache processes).
> 
> 1. Put another server up and load balance the two. I would move trac 
> from using SQLite to PGSql so i could get both servers to read the 
> database, however this may be possible using sqlite and an nfs or iscsi 
> share. Input welcomed.
> 
> 2. Move Trac website back to SF. They support trac now. Don't know how 
> much work this would be, and don't know if this is the correct solution. 
> We may just be moving the problem around.
> 
> 3. Move away from trac to some other tool that knows how to manage it's 
> memory. I have no idea on this one.
> 
> 4. Just hope our little script works and leave it as is.
> 
> That's all i have. Just let me know what you think would be a good 
> solution for this problem. Also the httpd processes consume huge amounts 
> of CPU. I could give the box another CPU. I don't know how much that 
> would help. It would help, just how much of a help would it be? Input is 
> wanted.

Or just disable msnbot via robots.txt?

Half the trouble seems to be that these search engines trawl EVERYTHING, 
so every historic link with the result that every single update has a 
search link created.
At one point I had 60 search engines all trawling my server, once one of 
the big boys lists you, all of the others join the band wagon :( So 
although it would possibly be nice that a search on one of them produces 
a result - is the cost of supporting them in computer power and 
bandwidth worth it?

My local machines simply has a deny all robots.txt, while the ones at 
1&1 have just a few select ones allowed nowadays - and nofollow helps on 
and historic stuff
http://blog.searchenginewatch.com/050118-204728

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk//
Firebird - http://www.firebirdsql.org/index.php

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july