Re: Huge Cricket Installations
"Dermot Williams" <[email protected]>
| Newsgroups | gmane.network.cricket.user |
|---|---|
| Message-ID | <[email protected]> |
David, Thanks for the tips, definitely something I'll have to look into. I left off the number of nodes in my email - where it says XXXX it should say 29850. In 1921 files. Definitely some pruning to be done! Dermot -----Original Message----- From: David Koski [mailto:[email protected]] Sent: 21 November 2006 12:54 To: Dermot Williams Cc: [email protected] Subject: RE: [cricket-users] Huge Cricket Installations Strip down all your %variables% where it makes sense to prior to run time. This will dramatically increase run time since it doesn't have to slosh through the database tree for each poll (Or disk, depending if you load the database in memory or not before the poll. Cricket is EXTREMELY database heavy, so if you don't have to do variable expansions at run time, it runs a lot quicker. Also standard configs, leave as far up in the tree as possible and don't redefine them if possible (Since most of my configs are the same among most leafs, I have moved most of my configs into the root Defaults file and stripped them out of the leafs all together where possible), cricket reads the tree and limp EVERY variable it polls for, so the less you have in the config, and the less variables you have to expand, the less disk and memory thrashing). Some of the contrib programs that cricket comes with don't even take any this into consideration, and some of the example configs are horrible at this (They work well for small installations, but run time add up quickly in having to do any of this). Simply doing something like this: In Defaults file: snmp-host = %router% In targets router = 111.222.333.444 Causes cricket to have to search the tree 1 time just to expand the variable... Changing this to Remove snmp-host from the Defaults file... And adding this to the targets files.. snmp-host = 111.222.333.444 Saves a lot of thrashing around. Examples: snmp-host = %auto-target-name% snmp-community = public snmp-port = 161 snmp-timeout = 2.0 snmp-retries = 5 snmp-backoff = 1.0 snmp-version = 1 snmp = %snmp-community%@%snmp-host%:%snmp-port%:%snmp-timeout%:%snmp-retries%:%snmp -backoff%:%snmp-version% Could become: snmp-host = %auto-target-name% snmp-community = public snmp-port = 161 snmp-timeout = 2.0 snmp-retries = 5 snmp-backoff = 1.0 snmp-version = 1 snmp = public@%snmp-host%:161:2.0:5:1.0:2 Providing your not redefining communit or snmp version some where else and then you'd have to put %snmp-community% and %snmp-version% back in. dataSource ifInOctets ds-source = snmp://%snmp%/ifInOctets.%inst% could become (Not this is the only place you can replace a non %% variable, but it works): dataSource ifInOctets ds-source = snmp://%snmp%/1.3.6.1.2.1.2.2.1.10.%inst% Etc... Pre-expand any variable you have prior to run time that you can (Obviously things like %inst% have to be expanded at run time since it changes). Use as few variables as possible, and only put those in that your going to change later on. This should increase run time and reduce disk or memory thrashing. David -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Dermot Williams Sent: Tuesday, November 21, 2006 7:34 AM To: [email protected] Subject: [cricket-users] Huge Cricket Installations Hi, I've a Cricket install (1.0.5) that polls over XXXX nodes. Obviously this is too many for a single box to poll, so I've spread the load over three machines. The actual Cricket installation is in a directory on a NetApp NAS that is mounted on each machine over NFS. This in itself is problematic because of locking problems and so on but for the most part it works pretty well and facilitates a quick turn-around in the event that a machine needs to be replaced. I've created multiple subtree sets in order to reduce the time taken to complete a polling cycle. We use Cricket to report on our CPE and new CPE are being added to the network all of the time so manual management of the subtree sets would be very time consuming. Some of the subtree sets are actually automatically generated out of a database backend (I've a script that automagically 'discovers' every node on the network, identifies what it is and adds it to the appropriate table in the DB) - the sets are treated like buckets: as soon as a set has 700 nodes in it, we start to fill another set. These sets are spread across the three polling machines. Generally this setup works well for us but there are number of issues that we are experiencing. Firstly, the load on the boxes is a constant problem. I've tried to optimise things so that each subtree set takes less than ten minutes to complete. This is fine except smaller subtree sets means more collection process which means higher memory usage and higher load. This is particularly problematic on the machine that runs the UI since it is very noticeable to users that the machine is under such a heavy load. Compiling the Cricket config often takes an absolute age and consumes huge amounts of memory (at the moment I've a compile that has been running for over 15 minutes and still isn't complete; it's consuming 3798M of memory, 700 of which is resident). More often than not the compile bombs out with a segmentation fault (11) and compiling is a hit and miss affair regardless of which of the three machines I run the compile on. The resulting config.db is over 100M in size. The machines are all 64-bit Dells (two are PowerEdge 1850s, two are 1950s) with 1 GB RAM, 5GB swap, running FreeBSD 6.1-STABLE. Does anyone on the list have experience with large Cricket installations? If so, have you any pointers or tips that you'd like to share? Dermot Williams Senior Network Engineer Irish Broadband Internet Services Ltd. +353 1 4818481 +353 86 3887961 -- No virus found in this outgoing message. Checked by AVG. Version: 7.5.430 / Virus Database: 268.14.11/543 - Release Date: 20/11/2006 21:20 ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ cricket-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/cricket-users !DSPAM:4562f27129802044995788! -- No virus found in this incoming message. Checked by AVG. Version: 7.5.430 / Virus Database: 268.14.11/543 - Release Date: 20/11/2006 21:20 -- No virus found in this outgoing message. Checked by AVG. Version: 7.5.430 / Virus Database: 268.14.11/543 - Release Date: 20/11/2006 21:20 ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV