Re: Snmptrapd fails to read config after being sent 4096 SIGHUPs?

Hi!

Sometimes people on this list proudly state that "you always get an answer to your questions" or similar. As you can see, this case is a very good example of this. Problem solved in three emails, including the initial question - great job! [Big smiley here!]

/Fredrik

> 4 feb 2014 kl. 17:10 skrev Joel Hansell <[email protected]>:
> 
> Hello,
> I was unable to reproduce the problem on 5.7.1, so I assume that this was already fixed. I'll upgrade our production installation to 5.7.2 as soon as possible.
> BR,
> Joel Hansell
> 
> 
>> On Tue, Feb 4, 2014 at 3:09 PM, Joel Hansell <[email protected]> wrote:
>> Hi again,
>> 
>> As an update, I was able to reproduce the error by writing a script that issues 4096 HUP signals to snmptrapd. The process starts issuing the "maximum conf file count (4096) exceeded" error, and forgets the configured logging format.
>> I don't suppose this is a known bug which was fixed some time after 5.6.1.1? It looks like an issue with the signal handling - there is only one config file, which is read multiple times. Seems like the config file counter should be reset when SIGHUP is received.
>> I can't find the source for 5.6.1.1, but in the 5.6.2 code, I don't see what the problem could be since "files" is in function-local scope. It should be zeroed every time the read_config() function is called. I guess there could be some scope confusion (or confusion on my part, I'm not a habitual C programmer), or the problem was already fixed in 5.6.2...
>> 
>> I'm trying to reproduce the bug on 5.7.1 now, but unfortunately, since snmptrapd takes about a second to restart on SIGHUP, the test takes just over an hour to execure.
>> 
>> BR,
>> Joel Hansell
>> 
>> 
>> 
>> 
>>> On Tue, Feb 4, 2014 at 10:58 AM, Joel Hansell <[email protected]> wrote:
>>> Hi list,
>>> 
>>> Here's one I've been scratching my head over lately.
>>> 
>>> We have a bit of an oddball solution, where we've got snmptrapd logging a lot of traps into a file, which is parsed by an external tool. This is all running on a HP-UX 11.2 system, with Net-SNMP version 5.6.1.1 delivered with the "HP-UX Internet Express" package.
>>> 
>>> A cron job runs "logrotate" every 15 minutes, and if the file is too big, it's rotated, and the postrotate script issues a SIGHUP to snmptrapd. That normally triggers the daemon to re-read its config and to restart the logging into a new file.
>>> 
>>> The snmptrapf.config is set up to use a particular one-line trap logging format.
>>> 
>>> It seems that every so often, the snmptrapd fails subtly on SIGHUP. It only seems to happen after more than a couple of months have passed. 60 days, 91 days, 101 days, 113 days are some of the fault intervals.
>>> 
>>> I've observed the following about the failure state after it happens:
>>> - Snmptrapd is executing and logging traps
>>> - The trap logging format has changed to the default trap logging format (three lines per trap). This causes our parser to fail
>>> - snmptrapd logs the error "[...]/snmptrapd.conf: line 0: Error: maximum conf file count (4096) exceeded" at the start of the log file.
>>> 
>>> The flow of traps is such that the log file is usually rotated every 30 minutes, but it goes up and down a bit. Could this failure be happening after 4096 SIGHUPs? That would explain the varying time between failures.
>>> 
>>> I'm grateful for any input.
>>> 
>>> Regards,
>>> Joel Hansell
> 
> ------------------------------------------------------------------------------
> Managing the Performance of Cloud-Based Applications
> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
> Read the Whitepaper.
> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
> _______________________________________________
> Net-snmp-users mailing list
> [email protected]
> Please see the following page to unsubscribe or change other options:
> https://lists.sourceforge.net/lists/listinfo/net-snmp-users

------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk

_______________________________________________
Net-snmp-users mailing list
[email protected]
Please see the following page to unsubscribe or change other options:
https://lists.sourceforge.net/lists/listinfo/net-snmp-users