Re: random collector hangs

Mike Fisher wrote:
> Mike Fisher wrote:
>> Ed Ravin wrote:
>>> Dumb ass idea but it might work - switch from the default SNMP library
>>> to Net-SNMP's library.  The Net-SNMP code has been tested against
>>> modern platforms, and is less likely to have 64-bit porting bugs.
>>>   
>>>     
>> That's a possibility.  I'll give it a shot when I get back to it.
>>   
> 
> That didn't help fix it, but it did help push the issue to a lower 
> level.  The hangs happen when using  the Net-SNMP module as well.

Turns out the problem is actually described perfectly in the "Bugs" 
section of the select(2) man page on Linux:

'Under Linux, select() may report a socket file descriptor as "ready for 
reading",  while  nevertheless a subsequent read blocks. This could for 
example happen when data has arrived but  upon  examination  has  wrong 
checksum  and is discarded. There may be other circumstances in which a 
file descriptor is spuriously reported as ready.  Thus it may be  safer 
to use O_NONBLOCK on sockets that should not block.'

After a watching the machine's interface most of the day with ethereal, 
I found that the the last packet received by the hung collectors did 
indeed have a corrupt UDP checksum.  The actual data corruption turned 
out to be easy to verify since it occurs in the middle of the SNMP 
community string.  I haven't found the source of the corruption yet, but 
that's a different issue.

The important point here is that both SNMP_Session and Net-SNMP use a 
blocking recv after a select, which after a corrupt packet will in fact 
block forever.  I've already sent up a bug report for SNMP_Session.  The 
fix there is simply adding the MSG_DONTWAIT flag to the recv call. The 
existing error checking handles the failure case.  I'll dig into the 
Net-SNMP code later this week and generate a bug report there as well.

Depending on how fast the fixes get pushed out, and how much time I have 
in the near future, I may also add this to the Cricket FAQ.

Mike

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/