Alarm event when monitoring log files
"Emmanuel Gadaix" <[email protected]>
| Newsgroups | gmane.comp.lang.perl.poe |
|---|---|
| Message-ID | <[email protected]> |
Hello POEers.
I am using Wheel::FollowTail to monitor a set of 24 log files that indicates
the status of some links connected to a system (for the curious: SS7 links
connected to an SMSC, the system that sends SMS messages to and from your
mobile phones). The idea was to get an alert when one of the links goes
down, as well as an alert when it comes up again.
Please allow me to post the relevant sections of the code:
sub InitLogs
{
my ($xdx, $sys, $name, $log, $rec);
$poe_records{_start} = \&begin_watchers;
for ($xdx = 1; $xdx <= 24; $xdx++)
{
$name = "link$xdx";
$log = "$PATH1/xdxmanmx$xdx.log";
$smsc_logs{$name} = $log;
$rec = $name . "_record";
$poe_records{$rec} = \&xdx_got_record;
}
$poe_records{log_reset} = \&generic_log_reset;
$poe_records{log_error} = \&generic_log_error;
}
sub begin_watchers
{
my $heap = $_[HEAP];
while ( my ($service, $log_file) = each %smsc_logs )
{
my $log_watcher = POE::Wheel::FollowTail->new
( Filename => $log_file,
InputEvent => $service . "_record",
ResetEvent => "log_reset",
ErrorEvent => "log_error",
);
$heap->{services}->{ $log_watcher->ID } = $service;
$heap->{watchers}->{ $log_watcher->ID } = $log_watcher;
}
}
sub xdx_got_record
# Handle log events
{
my $record = $_[10];
my $wheel_id = $_[11];
my $heap = $_[3];
my $service = $heap->{services}->{$wheel_id};
if ($record =~ /eactivated/) { &SendAlert("$service deactivated"); }
if ($record =~ /onnection is ready/) { &SendAlert("$service
activated"); }
}
&InitLogs;
POE::Session->create(inline_states => {%poe_records});
$poe_kernel->run();
Pretty simple. All log files get the same treatement. A simple regex match
then sends an alert whenever we have a link status change that appears in
one of the 24 log files.
It works great. POE is a fantastic tool, I can't even contemplate the
headache of doing that the traditional way.
Now, here is my problem. Sometimes a link goes down for a few seconds and
comes up again. And sometimes this up/down for a few seconds situation
happens continuously for a few hours, e.g. in case of transmission network
instability. So my little POE code generates A LOT of alerts during those
times, which are received by 5 support staff through their phones, often
waking them up in the middle of the night for a problem that isn't really
one.
So I have been asked the following: " would it be possible to send such
alerts only when the time between 'deactivated' and 'connection is ready' is
more than 60 seconds? "
At first I answered cockily "of course, anything can be done". Then I
started looking into how to do this, and my headache started. The subrouting
that handles log events (xdx_got_record) has no idea on when the next event
will come up so it is impossible to measure the time there.
I have been reading a bit the POE docs to find how this could be done but I
must admit I am still confused.
Before I dig further I would love to hear about your comments on this,
whether or not this is possible at all and what would be possible ways to
implement this. Any hint would be really appreciated.
Best regards
Emmanuel