performance issues, 50% checks (if lucky) in 5 Mins !!

Hey All !!

For the longest time now, Nagios 1.1 is unable to check more than 50% of
the services in 5 minutes, it takes more than 15 minutes for 90%+
services to get checked. 
granted, I have about 1600 Services distributed on 110 hosts but it used
to run at around 80% in 5 minutes. 

<= 1 minute:92 (5.9%)
<= 5 minutes:641 (41.4%)
<= 15 minutes:1354 (87.4%)
<= 1 hour:1513 (97.6%)
Since program start:  1549 (99.9%)

Check Latency avg seems to be very high though ...

					avg
Check Execution Time:  < 1 sec75 sec 	5.151 sec
Check Latency:< 1 sec466 sec		304.960 sec
Percent State Change:0.00%0.00%0.00%



I Used to run lots of MRTG processes in the same box and used to have
the machine very busy .. (Anywhere from 15 - 20 constantly) but for
months now, we have been running at weights no bigger than 4 and 
the system is still unable to process more than 50 % of the checks in
five minutes.

I have checked all the nagios settings, including the max number of
concurrent checks (which is set to 0), and I cannot find anything that
is limiting the checks, Is there a way to set a flag that will show me 
in a log somewhere, where the system is waiting ? 

Here is my cpu load graph
http://nagios.dallasairmotive.com/cgi-bin/14all.cgi?log=svr-fp-netmon03_cpu_load-1
Here is my cpu use graph
http://nagios.dallasairmotive.com/cgi-bin/14all.cgi?log=svr-fp-netmon03_cpu_use-1
memory use graph 
http://nagios.dallasairmotive.com/cgi-bin/14all.cgi?log=svr-fp-netmon03_stor-101
and finally nagios performance 5 and 15 minutes
http://nagios.dallasairmotive.com/cgi-bin/14all.cgi?log=svr-fp-netmon03_nagiosperf-1

I noticed some web sites were not allowing pings so I wrote a quick 
bogus check-host-alive-bogus and used it instead of the
check-host-alive, I noticed that that was slowing things down but by no
means I think it was holding the train ... 
Any Ideas ? Anything is welcome, Need to get it back to a decent level 

Thanks a bunch in advance .. 

Adrian Salaices.
Applications Manager/DBA 
DallasAirmotive. 


-- 
Windows [n.]
A thirty-two bit extension and GUI shell to a sixteen bit patch to an
eight bit operating system originally coded for a four bit
microprocessor and sold by a two-bit company that can't stand one bit of
competition."
(Anonymous USEnet post)



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf