Re: PoCo::Client::Ping intermittently fails to emit pings

On Apr 15, 2009, at 07:59, Stuart Kendrick wrote:

> my favorite pinging application (in-house app, uses  
> POE::Component::Client::Ping) intermittently reports a rash of  
> missed replies. happens roughly the same time each night; the  
> condition lasts for several minutes

Time-based failures implicate periodic jobs.  What's cron doing around  
that time?

> Details:
> -  perl-5.10.0, POE 1.0.4, POE::Component::Client::Ping 1.14
>
> -  CentOS 5.3 (2.6.18-128.el5 x86_64 x86_64 x86_64 GNU/Linux)
>
> -  the application is pinging ~120 hosts every 30 seconds

Does the application ping them all in a burst?  This could cause lost  
packets.  As I mentioned, ICMP isn't reliable, and congestion on the  
network or interface could cause silent packet drops.

> -  the box gets busy during this time, in terms of disk I/O (average  
> latency of
>   ~100ms ... fifteen minute rolling load average spikes to ~18:   
> likely stems
>   from several disk-intensive jobs which run during this period).   
> this
>   busyness lasts for ~5 hours

Heavy load could be causing your monitor program to lag, triggering  
timeouts---possibly before the ICMP requests are sent.  Reducing the  
pinger's niceness slightly may help, but you may just be overloading  
the system.  See what the machine needs to handle the work you've  
given it.  Consider distributing the work across multiple machines.

> -  the box hosts a number of pinging applications (Nagios, a bunch  
> of in-house
>   apps), typically employing fping

Probably not an issue, unless they stream an awful lot of requests.   
Load spikes may push the pinging activity together, which could result  
in lost packets.  For example, 120 pings over 30 seconds is fine, but  
a lag spike might delay them.  The lagged ones will try to catch up in  
a burst when the lag spike ends---or when the OS says your pingers  
have been CPU starved long enough.

> -  i've been running the application for a handful of years now, a  
> handful of
>   instances, monitoring ~100 - 500 hosts per instance.  this is  
> novel behavior

Novel behavior suggests the question "What has changed?"  The variable  
could be something new and significant or something that has gradually  
approached a capacity threshold.  Some things to think about:  Has the  
number of hosts to ping increased by a lot recently or steadily over  
time?  Have the disk-intensive jobs' workload increased by a lot  
recently or steadily over time?

-- 
Rocco Caputo - [email protected]