Re: Huge Cricket Installations

We poll over 60,000 targets in 15 minutes.  My hardware is a Sun V440
w8GB memory and 4 1593 MHz CPUs. 

Perhaps someday if you want to consolidate and if it's within your
budget you would want to consolidate your installation on a unix box.  

I can't say I'm much of an expert on tuning cricket (thanks so much for
those suggestions from Dave--my config files are generated daily by a
perl script and I could eliminate most variables probably!), but I can
say that when we first set up this monitoring system (using MRTG the
first time around) the show-stopper was disk I/O.  We were using an
internal disk arrangement that simply could not handle all the I/O.
Once we set up a SAN partition (EMC disk--very fast with fiber
connections), disk I/O was LESS of a problem.  When we switched from
MRTG to cricket enabling us to consolidate targets into fewer RRD files
we cut back on CPU usage by 30%!

I have 27 configuration directories (cricket-config1, cricket-config2).
There are >13,000 files.  My config.db files are 3MB in size.  To spread
the load I have 3 subtree sets per config directory!   (ie cricket0a,
cricket0b, cricket0c).  Most sets contain <1000 targets and complete in
<1 minute.  My sets that contain around 1000 targets complete in around
3 minutes.

Because the config file generation is automated, there isn't much of a
maintenance headache even with all those config files.

Tanya Ruttenberg - RSIS Contractor
OTSO/DNE/NMPEB
[email protected]
410-965-9605


-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Dermot
Williams
Sent: Tuesday, November 21, 2006 7:34 AM
To: [email protected]
Subject: [cricket-users] Huge Cricket Installations

Hi,

 

I've a Cricket install (1.0.5) that polls over XXXX nodes. Obviously
this is too many for a single box to poll, so I've spread the load over
three machines. The actual Cricket installation is in a directory on a
NetApp NAS that is mounted on each machine over NFS. This in itself is
problematic because of locking problems and so on but for the most part
it works pretty well and facilitates a quick turn-around in the event
that a machine needs to be replaced.

 

I've created multiple subtree sets in order to reduce the time taken to
complete a polling cycle. We use Cricket to report on our CPE and new
CPE are being added to the network all of the time so manual management
of the subtree sets would be very time consuming. Some of the subtree
sets are actually automatically generated out of a database backend
(I've a script that automagically 'discovers' every node on the network,
identifies what it is and adds it to the appropriate table in the DB) -
the sets are treated like buckets: as soon as a set has 700 nodes in it,
we start to fill another set. These sets are spread across the three
polling machines.

 

Generally this setup works well for us but there are number of issues
that we are experiencing. Firstly, the load on the boxes is a constant
problem. I've tried to optimise things so that each subtree set takes
less than ten minutes to complete. This is fine except smaller subtree
sets means more collection process which means higher memory usage and
higher load. This is particularly problematic on the machine that runs
the UI since it is very noticeable to users that the machine is under
such a heavy load.

 

Compiling the Cricket config often takes an absolute age and consumes
huge amounts of memory (at the moment I've a compile that has been
running for over 15 minutes and still isn't complete; it's consuming
3798M of memory, 700 of which is resident). More often than not the
compile bombs out with a segmentation fault (11) and compiling is a hit
and miss affair regardless of which of the three machines I run the
compile on. The resulting config.db is over 100M in size.

 

The machines are all 64-bit Dells (two are PowerEdge 1850s, two are
1950s) with 1 GB RAM, 5GB swap, running FreeBSD 6.1-STABLE.

 

Does anyone on the list have experience with large Cricket
installations? If so, have you any pointers or tips that you'd like to
share?

 

Dermot Williams

Senior Network Engineer

Irish Broadband Internet Services Ltd.

 

+353 1 4818481

+353 86 3887961

 


--
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.430 / Virus Database: 268.14.11/543 - Release Date:
20/11/2006 21:20
 
------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT Join
SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
cricket-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/cricket-users

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV