Re: net-snmp-5.4.2.1 snmpd cpu usage 100%

Hello Wes Hardaker,

I have answers to some your questions again:

WH:>> 2) make sure the code is supporting caching.  With 4000 entries in the
WH:>> /proc/net/dev file you *don't* want to re-read it frequently.

As earlier said with strace out put as well while attaching snmpd to gdb to
do frequent back tracing i found that cache_load was being called
frequently.
One sample backtrace goes like this:

--------------------------------------------------------------
#0  0x0000005555b74d60 in write () from /lib64/libc.so.6
#1  0x0000005555b1d9dc in _IO_new_file_write () from /lib64/libc.so.6
#2  0x0000005555b1d570 in new_do_write () from /lib64/libc.so.6
#3  0x0000005555b1d970 in _IO_new_do_write () from /lib64/libc.so.6
#4  0x0000005555b1d724 in _IO_new_file_xsputn () from /lib64/libc.so.6
#5  0x0000005555af83f0 in vfprintf () from /lib64/libc.so.6
#6  0x0000005555afe1f4 in fprintf () from /lib64/libc.so.6
#7  0x0000005555835948 in log_handler_file () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#8  0x0000005555835bb4 in snmp_log_string () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#9  0x0000005555835ccc in snmp_vlog () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#10 0x0000005555830b84 in debugmsg () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#11 0x0000005555867bc0 in netsnmp_compare_netsnmp_index () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#12 0x00000055558682fc in array_qsort () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#13 0x00000055558684c0 in array_qsort () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#14 0x00000055558684c0 in array_qsort () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#15 0x000000555586861c in Sort_Array () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#16 0x0000005555869094 in netsnmp_binary_array_get () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#17 0x000000555586925c in netsnmp_binary_array_insert () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#18 0x00000055558691d0 in _ba_insert () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#19 0x00000055557374bc in CONTAINER_INSERT_HELPER () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#20 0x00000055557373fc in CONTAINER_INSERT () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#21 0x0000005555737238 in _netsnmp_ioctl_ipaddress_container_load_v4 () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#22 0x0000005555732e60 in netsnmp_arch_ipaddress_container_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#23 0x0000005555731bc8 in netsnmp_access_ipaddress_container_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#24 0x000000555570bb88 in ipAddressTable_container_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#25 0x000000555570a49c in _cache_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmpmibs.so.15
#26 0x00000055555eb948 in _cache_load () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmphelpers.so.15
#27 0x00000055555ea574 in _timer_reload () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmphelpers.so.15
#28 0x0000005555843ae8 in run_alarms () from
/opt/nokiasiemens/SS_NetSnmp//lib/libnetsnmp.so.15
#29 0x0000000120005498 in receive ()
#30 0x00000001200049f0 in main ()

--------------------------------------------------------------


WH:>>  3) Is it possible that parsing is failing?  IE, is it possible there
is
WH:>>   a bug in the code that is preventing the function from ever
quitting?
WH:>>   Does the CPU stay at 100% forever or does it eventually drop back
down?

YES, CPU stays 100% for ever, i think your guess might be right. But as an
observation when i had less than 2000 VLAN's this was not the case it is
used to fluctuate between 10-40-100 but predominently 100%.

WH:>> If it takes too long to fill the cache because of that many entries,
you
WH:>> may want to increase the caching time for that particular code.

But from the grpofile output i attached it does not look like that, ofcourse
i woudl like to introduce cache->timeout in ipAddressTable_initialise().

Let me know.

Thanks very much.

Best Regards,
Bheemesh




On Fri, Nov 20, 2009 at 12:52 PM, bheemesh v <[email protected]> wrote:

> Hello Wes Hardaker,
>
> Thanks very much for your valuable inputs here.
> We are starting the snmpd with -f option, so i guess i am going right
> there.
>
> I am attaching a sample gpofile output file to your refrence generated from
> my HW environnment.
> May be this can give better data for investigations.
>
> I will get back on the remaining questions.
> But as an observation for cache->timeout, it is being used for
> ifTable_initialization, but not for ipAddressTable_initialization.
> So cache-> flags are only being used commonly.
>
> Let me know.
>
> Thanks very much.
>
> BR,
> Bheemesh
>
>
>
>
>
> On Thu, Nov 19, 2009 at 7:10 AM, Wes Hardaker <
> [email protected]> wrote:
>
>>
>> bv> Has anybody done so far profiling with gprof or any equivalent tool
>> bv> for src code profiling for net-snmp-5.4.2.1 or any prev versions?
>> bv> So far we are finding it tough to generate the gmon.out file for
>> bv> profiling data generation.
>>
>> I have, but it's been a while.  In the past I successfully used gprof to
>> profile various aspects of the agent.
>>
>> bv> We have seen reasons being the start directory, multi threaded
>> programs
>> bv> unsuitable for profiling etc.
>>
>> We don't use multithreading.  Do make sure you run the agent with -f
>> though.
>>
>> bv> Also so far for this mail chain, no one has given any leads, is it
>> because
>> bv> my question is ambiguos or do i have to give bit more background of
>> info
>> bv> regarding our system setup?
>>
>> Actually, I think you've done a really nice job explaining your
>> problem (much better than our average user).  Unfortunately, your
>> problem is also fairly complex which makes it hard to answer quickly
>> (and a large number of developers got very busy with other aspects of
>> their life and jobs all at the same time).
>>
>> > netsnmp_arch_interface_container_load() function from
>> > "if-mib/data_access/interface_linux.c" has a file pointer to
>> "proc/net/dev"
>> > from it tries to get updates of each interface in a while loop, which
>> takes
>> > too much of cpu time when we have many VLAN interfaces defind in our
>> server
>> > this loads CPU 100%.
>>
>> There are a few things I can think of:
>>
>> 1) the code in question isn't well optimized for 4000 vlans and takes a
>>   long time to load.  I'm actually not sure, but I'd think it would
>>   actually be ok.
>> 2) make sure the code is supporting caching.  With 4000 entries in the
>>   /proc/net/dev file you *don't* want to re-read it frequently.
>> 3) Is it possible that parsing is failing?  IE, is it possible there is
>>   a bug in the code that is preventing the function from ever quitting?
>>   Does the CPU stay at 100% forever or does it eventually drop back down?
>> 4) You should try running a *single* snmpgetnext on the iftable with the
>>   following options: "-r 0 -t 600", which will make snmpgetnext wait
>>   for a long time and only try once.  See if you ever get a response.
>>   If you do, rerun the command *immediately* (and hopefully the next
>>   request will return more quickly if it's cached).
>>
>> If it takes too long to fill the cache because of that many entries, you
>> may want to increase the caching time for that particular code.
>> --
>> Wes Hardaker
>> Cobham Analytic Solutions
>>
>
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july

_______________________________________________
Net-snmp-users mailing list
[email protected]
Please see the following page to unsubscribe or change other options:
https://lists.sourceforge.net/lists/listinfo/net-snmp-users