Re: Trac issues once again

On Sat, 5 Sep 2009, [email protected] wrote:

> To: Phpeclipse devel <[email protected]>
> From: [email protected]
> Subject: [PHPEclipse-devel] Trac issues once again
> 
>
>
> The msnbot has decided to index phpeclipse.com website. 
> This is killing the server and causing trac to consume all 
> memory in the box. I have given the box 3Gig of RAM, it 
> worked for a while but now is being a problem again. I 
> have a few thoughts on solutions for this. (i did enable a 
> memory check we created some time ago to check the system 
> memory and if it fell below a level to kill all apache 
> processes).

What about blocking msnbot from everything bar the homepage?

Analyzing file http://www.phpeclipse.com/robots.txt

No errors found in this robots.txt file

You can find some warnings at the bottom of the analysis

Hide empty and comments lines: on

The following block of code DISALLOWS the crawling of the 
following files and directories:  /browser /log /changeset 
/report /newticket /search  to all spiders/robots.
   Line 1  User-agent: *
   Line 2  Disallow: /browser
   Line 3  Disallow: /log
   Line 4  Disallow: /changeset
   Line 5  Disallow: /report
   Line 6  Disallow: /newticket
   Line 7  Disallow: /search
   Line 8

WARNING: The tool has found some directory paths that 
don't include a trailing slash character.

Since a missing trailing slash can be both a deliberate 
decision or an error, and since this tool can't ipotize the 
real intentions of the webmaster, here follow some 
clarifications that could prevent a potential problem:

The following command will disable just the directory 
"private" and all its contents:
Disallow: /private/

...while the following command will disable both the 
"private" directory and any file or directory path starting 
with the text "/private" (so "/private-eye.html", 
"/privateroom/page.html", etc.):
Disallow: /private

Please be sure to use the correct syntax, according to your 
needs.

Robots.txt Checker at:

http://tool.motoricerca.info/robots-checker.phtml

Kind Regards,

Keith Roberts

BTW. I have installed Fedora Eclipse and the PHPEclipse 
extension. I really like the added documentation for setting 
up debugging. I'm sure that will be a great help to new user 
of PHPEclipse :)

> 1. Put another server up and load balance the two. I would 
> move trac from using SQLite to PGSql so i could get both 
> servers to read the database, however this may be possible 
> using sqlite and an nfs or iscsi share. Input welcomed.

PostgreSQL requires text file dumps to backup the databases. 
Unlike MySQL where you can just stop the server and copy the 
databases directly to back them up.

> 2. Move Trac website back to SF. They support trac now. 
> Don't know how much work this would be, and don't know if 
> this is the correct solution. We may just be moving the 
> problem around.
>
> 3. Move away from trac to some other tool that
> knows how to manage it's memory. I have no idea on this one.
>
> 4. Just hope
> our little script works and leave it as is.
>
> That's all i have. Just let me know what you think would 
> be a good solution for this problem. Also the httpd 
> processes consume huge amounts of CPU. I could give the 
> box another CPU. I don't know how much that would help. It 
> would help, just how much of a help would it be? Input is 
> wanted.
>
> Thanks.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july