Re: Pyro 3 event server hang

On 22 Apr 2013, at 21:07, Irmen de Jong <[email protected]> wrote:

> On 22-4-2013 9:54, Christoph Gohle wrote:
>> Dear Irmen,
>> ...
> This doesn't mean we can't at least try to help with your problem :)
Ok, that's great. 
> 
> 
>> Event rate is definitely not too high. The queue only starts filling after the worker
>> has stopped delivering. I did play with the PYRO_ES_BLOCKQUEUE. That helps keeping
>> the producers happy (as they don't block once the worker is and it's queue fills).
>> But of course this does not fix that the listener does not recieve events anymore. I
>> checked that the event handler finishes normally after the last event it recieved. I
>> found that we are running an older version of Pyro on our event server machine (3.14)
>> and I have seen that some modifications regarding timeouts were done in 3.16. Could
>> it be that the socket.send is blocking in the worker? I will try to upgrade and see
>> if it helps.
> 
> Please let us know. Btw, what platform are you working on (OS/python version)?
Upgrading pyro did not change the situation. The event and name services are running on Ubuntu 12.04/python2.7. Our servers and clients are currently running on windows7/python2.7.

I was trying to get a hold of what's going on by inserting print statements into the event service. It looks as if our event consumers first stops recieving events (but the consumer does finish it's event callback before the hang). Then at some point the socket stops accepting data, which makes the worker hang and only once the queue of the worker filled up it blocks on publish calls. Unfortunately I know too little about the low level communication to figure out why the events stop being delivered to the consumer.
> 
>> Yes, this might be an option. Although we were hoping to not have to do the debugging
>> of an event service our selves… oh well.
> [...]
>> Yes, debugging race conditions is a pain. That's why we liked the idea of using
>> established code.
> 
> Fair enough. Too bad you're encoutering one of the problems. How easy/hard is it to
> reproduce it? Do you have a minmal set of test scripts that reliable make the problem
> occur? What feature set of the event server are you using?
We have one procedure that reproducibly creates the problem at roughly the same point during execution. The procedure is however quite involved and by no means minimal. So far we have not gotten to the point where we know which part of our script is actually triggering the problem. The producer(s) only use the publish function. The consumer(s) use subscribe and override the event callback. It is sufficient to have one producer and one consumer to reproduce the problem.

I will let you know once we get more hints.

Best,
Christoph Gohle

------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service 
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr

_______________________________________________
Pyro-core mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/pyro-core