Problem with SNMP.pm

Hello,

We use extensively the SNMP.pm module from the net-snmp software to
manage our numerous network devices (switches, wireless base stations).

There are now a lot more than 1000 devices, and recently we noticed that
something strange was happening in snmp.pm.

If a session is opened to each device within the same script, then after
about 1000 connections this happens:

	9:callback:lock: locked (LIB,SESSION_INIT)
	trace: snmp_call_callbacks(): callback.c, 304:
	callback: START calling callbacks for maj=0 min=5
	trace: snmp_call_callbacks(): callback.c, 318:
	callback: calling a callback for maj=0 min=5
	trace: snmp_call_callbacks(): callback.c, 330:
	callback: END calling callbacks for maj=0 min=5 (1 called)
	trace: _callback_unlock(): callback.c, 150:
	9:callback:lock: unlocked (LIB,SESSION_INIT)
	trace: netsnmp_tdomain_transport(): snmp_transport.c, 332:
	tdomain: no specifier in "u1000-s-ip2-shpym-1"
	trace: netsnmp_tdomain_transport(): snmp_transport.c, 350:
	tdomain: specifier "udp" address "u1000-s-ip2-shpym-1"
	trace: netsnmp_tdomain_transport(): snmp_transport.c, 356:
	tdomain: specifier "udp" matched
	trace: netsnmp_sockaddr_in(): snmpUDPDomain.c, 653:
	netsnmp_sockaddr_in: addr 0xbfffce70, peername
"u1000-s-ip2-shpym-1"
	trace: netsnmp_sockaddr_in(): snmpUDPDomain.c, 714:
	netsnmp_sockaddr_in: hostname (couldn't resolve)
<-------------------------------------------------------- why?????
	trace: _sess_open(): snmp_api.c, 1428:
	_sess_open: couldn't interpret peername

Then for following alternate connections the same thing happens,
resulting in a 50% failure rate.

Looking a ulimit (RedHat linux ES3):

core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) 4
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 10240
cpu time             (seconds, -t) unlimited
max user processes            (-u) 7168
virtual memory        (kbytes, -v) unlimited

So, "open files" looks like a good candidate.  I increase it to 2048.
Now no new sessions fail.

However, we are using async mode with mainloop.  After opening the
sessions the mainloop now hangs:

verbose:sess_select: timer overdue
verbose:sess_select: setting timer to 0.0 sec, clear block (was 1)
trace: snmp_sess_select_info(): snmp_api.c, 5843:
sess_select: for all sessions: 1505 1503 (to in 1155139122.622117 sec)
1479 (to in 1155139122.518342 sec) 1471 (to in 1155139122.493013 sec)
1462 (to in 1155139122.434828 sec) 1460 (to in 1155139122.427641 sec)
1448 (to in 1155139122.390311 sec) 1422 (to in 1155139122.288978 sec)
1345 (to in 1155139120.690619 sec) 1342 (to in 1155139120.589010 sec)
1319 (to in 1155139120.55406 sec) 1313 (to in 1155139119.926811 sec)
1309 (to in 1155139119.841165 sec) 1307 (to in 1155139119.808367 sec)
1294 (to in 1155139119.505699 sec) 1285 (to in 1155139119.274562 sec)
1281 (to in 1155139119.162909 sec) 1273 (to in 1155139118.994661 sec)
1267 (to in 1155139118.842084 sec) 1262 (to in 1155139118.746482 sec)
1251 (to in 1155139118.502396 sec) 1230 (to in 1155139117.993455 sec)
1228 (to in 1155139117.960243 sec) 1216 (to in 1155139117.673066 sec)
1198 (to in 1155139117.252478 sec) 1189 (to in 1155139117.42507 sec)
1182 (to in 1155139116.881051 sec) 1181 (to in 1155139116.879215 sec)
1168 (to in 1155139116.562468 sec) 1147 (to in 1155139116.102187 sec)
1146 (to in 1155139116.38964 sec) 1141 (to in 1155139115.930982 sec)
1137 (to in 1155139115.828048 sec) 1110 (to in 1155139115.193471 sec)
1106 (to in 1155139115.138039 sec) 1081 (to in 1155139114.567048 sec)
1077 (to in 1155139114.470653 sec) 1072 (to in 1155139114.353467 sec)
1047 (to in 1155139113.792517 sec) 1045 (to in 1155139113.715843 sec)
1044 (to in 1155139113.689459 sec) 1025 (to in 1155139113.259374 sec)
991 (to in 1155139112.446986 sec) 981 (to in 1155139112.231009 sec) 971
(to in 1155139111.974637 sec) 968 (to in 1155139111.944264 sec) 965 (to
in 1155139111.861528 sec) 964 (to in 1155139111.829581 sec) 961 (to in
1155139111.756376 sec) 933 (to in 1155139111.123397 sec) 926 (to in
1155139110.979337 sec) 923 (to in 1155139110.862609 sec) 912 (to in
1155139110.628480 sec) 908 (to in 1155139110.540900 sec) 898 (to in
1155139110.297442 sec) 888 (to in 1155139110.85955 sec) 872 (to in
1155139109.695594 sec) 843 (to in 1155139109.16280 sec) 824 (to in
1155139108.587137 sec) 823 (to in 1155139108.558285 sec) 807 (to in
1155139108.167971 sec) 788 (to in 1155139107.743487 sec) 762 (to in
1155139107.160013 sec) 750 (to in 1155139106.845375 sec) 739 (to in
1155139106.630270 sec) 731 (to in 1155139106.422567 sec) 689 (to in
1155139105.454200 sec) 687 (to in 1155139105.411810 sec) 679 (to in
1155139105.195000 sec) 666 (to in 1155139104.883297 sec) 661 (to in
1155139104.743775 sec) 641 (to in 1155139104.295959 sec) 610 (to in
1155139103.558996 sec) 593 (to in 1155139103.139737 sec) 579 (to in
1155139102.821352 sec) 576 (to in 1155139102.766179 sec) 568 (to in
1155139102.558702 sec) 540 (to in 1155139101.925179 sec) 535 (to in
1155139101.815648 sec) 515 

Does anyone have any clue how to make SNMP.pm work with >~1000
concurrent connections?

Cheers,

Nick

------------------
Dr. Nick Garfield
IT/CS Group
CERN
CH-1211 Geneve 23
Switzerland

T: +41 22 767 4533
M: +41 76 487 3282
------------------ 

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Net-snmp-users mailing list
[email protected]
Please see the following page to unsubscribe or change other options:
https://lists.sourceforge.net/lists/listinfo/net-snmp-users