Netsaint server memory trouble investigation

Greetz list,

We've been getting low memory errors from our Netsaint 0.0.7
monitoring host for the last few weeks and just now had time to put
some investigation into it.It is a dual-850 HP server with
approximately 650 MB RAM running as follows:

Red Hat Linux release 7.0 (Guinness)
Kernel 2.2.16-22enterprise on a 2-processor i686

'top' shows as a pretty usual average:
--
Mem:   646760K av,  631796K used,   14964K free
--

Netsaint is monitoring 70 Hosts, 429 Services. Checking into a run
from 'gtop' shows some alarming information:

Some resident memory counts of the running Netsaint processes:
--
28@ 53600KB
78@ 161000KB
71@ 147000KB
105@ 217000KB

The netsaint processes are all of the form:
/usr/local/netsaint/bin/netsaint -d /usr/local/netsaint/etc/netsaint.cfg
--

Some resident memory counts of the running httpd (Apache/1.3.14)
--
10@ 51000KB
14@ 70000KB
--

Some resident memory counts of the running check_ping processes
--
30@ 15000KB
--

And finally, viewing gtop sorted by Total Size of Processes (Memory)
is a process called 'minilogd':
--
1@ 238708KB
--

There are other services such as mysql and a couple of seperate web
apps running on this server as well, but end up taking a fraction of
the memory.

So in begging for help from the community, what's the analysis? Is the
box simply overloaded? Is the minilogd process to blame for the
near-depleted memory? (And if so, what to do with it?) Or are the many
and memory-hungry httpd and netsaint processes the direct result of
Netsaint? (i.e can the heavy httpd usage be tied directly into
Netsaint in any way..)

Assuming that Netsaint is taxing the system right now, would a
clustered or distruted model be in order at this point? Such as in the
number of checks running on the server are driving it to its limit, or
is there any way to optimize what is currently running on (it's not
exactly a puny server). Would more memory be the correct answer, or
only provide a greater amount to be chewed up all the same?

TIA for any insight.

-- 
Darren Spruell
www.sancho2k.net




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf