Re: Pyro 4.14 MemoryError using Shinken 1.2

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14-9-2012 20:06, Francois kilian wrote:
> Summary description:
> 
> Server is amd64 , satellites are Sheevaplugs (armv5l) Debian squeeze python is
> 2.6. Pyro 4.14 on server and satelittes. Only one satellite is exibiting the
> error.

Only one? Are the sattelites identical in every respect? All the others work just fine?
That surely means that it is not a problem in Pyro or Shinken itself, but rather one
of the following:

1) there is a hardware problem that causes the error, for instance a faulty memory
chip or bad harddrive sector.
2) a software package installed on the failing sattelite has become corrupted.
3) or perhaps, and probably very unlikely, that the network infrastructure
(cables/router/etc) experience a fault and deliver corrupt packets to the failing
sattelite, whereas the other sattelites get good data.

> See the shinken forum thread for the traceback error and all the tests that have
> been executed.
> 
> Link to discussion of the error from the Shinken forums: 
> www.shinken-monitoring.org/forum/index.php/topic,548.0/topicseen.html
> 
> This one has us stumped. The user can reproduce the issue at will.
> 
> Irmen, do you have any suggestions to help in tracking down the problem.

for 1): run a hardware diagnostics check and a memtest (http://www.memtest.org/) on
the failing device
for 2): check and/or reinstall all software on the sattelite from scratch, including
the operating system
for 3): replace cabling etc.

Other than that, here are some general thoughts. A memoryerror means:
"Raised when an operation runs out of memory but the situation may still be rescued
(by deleting some objects). The associated value is a string indicating what kind of
(internal) operation ran out of memory. Note that because of the underlying memory
management architecture (C’s malloc() function), the interpreter may not always be
able to completely recover from this situation; it nevertheless raises an exception so
that a stack traceback can be printed, in case a run-away program was the cause. "

On the shinken forum you write "I can't even see the memory raising. When the crash
occurs there is at least 5gb free. There is no log of oom in the kernel messages." So
it doesn't seem like a real out of memory situation is the cause here.
Another possible reason for malloc() to fail can also be memory fragmentation, which
means that there's enough free RAM but just not a free chunk somewhere in between that
is large enough to hold the required new allocation size. No idea if this could be the
case in your situation, and I have no idea on how to debug for this.

It is not entirely clear to me where exactly the memoryerror occurs: is it indeed
raised on the sattelite device, and received and logged on the server? Or is the
server throwing it by itself?

Try running the python interpreter with warnings on (-Wall).
Try using the HMAC key feature of Pyro to validate the network packets.
Try using Pyro's multiplex server instead of the threadpool server.


Hope this helps,

Irmen de Jong



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (Cygwin)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQVDCpAAoJEHC9o7RHFlf2vn4H+wbVZQBP0aSsPJPPkQ+ri1oB
DMaItgaMJiIJYvtGEgDYITWUesNvuYE3b1+umDr2MrzNVvrCvRX95mHXl34+wddy
NpVNDpOOeqORw8WpLs+CtD90bT+3AUkqq7oqmpqxOioN5aVWqr9u+YKk2Fia+30c
icM2K/3Tc2GoPK32Qwh9+rO7GwT14vHyhI/QaXtt2JAD8accEolG/SOb0/BElDm6
+Lu19YDTX9ObOEbMNTGdA2K2Dk96pf5t3jkwfTqfMseIolihG5NsOe33PfwdR+Lf
eKxtAeW/nLH72/ee6ovIEYKI30HjKt6npEcjRZsRmtNkpB3hMSn1AdKjIIoa4CU=
=IJ3v
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
How fast is your code?
3 out of 4 devs don\\\'t know how their code performs in production.
Find out how slow your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219672;13503038;z?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html