Re: using mod_pubsub in the middle of a production app

Wow Kragen!  Great doc!  Can we put it in the wiki?  :-)  JP


--- Kragen Sitaker <[email protected]> wrote:
> So we've been slowly increasing our use of mod_pubsub in our product,
> and I thought I'd share my experience with all of you.
> 
> First, the good news: we're mostly pretty happy with the improvements
> in performance and maintainability that are coming from restructuring
> most of our IPC to use mod_pubsub.  It's easy to integrate.
> 
> A note about YAML: we tried using YAML to serialize Perl data structures,
> but it was amazingly slow.  A small hash of 1000 entries took 0.1 seconds
> to serialize and deserialize.  Now we're using Data::Dumper to serialize,
> and Perl eval to deserialize, which is about 100 times as fast.
> 
> We're using the Perl microserver.
> 
> One of the first problems we hit was that the microserver's 'subscribe'
> call wasn't persistent enough for us.  Normally we start pubsub.py
> (the python_pubsub server) at the same time as all the other daemons.
> Sometimes it takes longer to initialize than the other daemons, so
> they start trying to subscribe too early, and get 'connection refused'.
> So we added a 'subscribe_blocking' method that retries every few seconds
> for a couple of minutes.
> 
> Another one is that the microserver's PubSub::Tunnel::parse_events
> function is deeply recursive, if you're processing a large batch of
> events, and at least Perl 5.8 seems to use a lot of memory in this case.
> (Probably because each stack frame has its own local copy of the remaining
> data in the network buffer.)  We fixed this by breaking it apart into
> parse_event, which returns zero or one events, and parse_events, which
> is a loop around parse_event.
> 
> A minor problem is that PubSub::Client::handle_events doesn't have any way
> to block until there are events, so we settle for calling it frequently.
> It's pretty efficient when there aren't any events.
> 
> One of the reasons for wanting to run things in an event-driven fashion
> is to decouple the different parts of your software.  For example,
> our software monitors network nodes and periodically collects some data
> from them, which several programs analyze as they get it.  It's nice for
> the data collector to have a separate set of resources, and not share
> failure modes, with the data analysis programs.  So the data collector
> posts its data to a topic as it reads it, and the data analyzer subscribes
> to the topic and analyzes data as it can, which is often slower than
> the collector can collect it.
> 
> This causes some big problems with pubsub.py.  As soon as an event is
> posted, it appends an encoded copy of the event to the outbound network
> buffers of all the subscribing tunnels, then tries to flush them.
> When the client sucking events down the tunnel isn't keeping up with
> the event stream, then eventually the network buffers associated with
> the client's socket and the socket on pubsub.py's side fill up, and
> pubsub.py starts getting EAGAIN errors.
> 
> It handles these temporary failures "correctly" in that it keeps
> the data buffered and retries later when it can, but this creates a
> secondary problem.  The outgoing data buffer can get very large; I've
> observed over 100MB.  Events in the outgoing data buffer don't expire,
> and don't get updated by more recent events; this happens by design,
> although I can't remember why.  But eventually the unbounded growth of
> this buffer starts paging everything else out to disk.
> 
> It's even worse than that, actually, because the buffer is just a Python
> string.  So when we append to it, we create a new string, copy the old
> contents into it, and append the new data.  This happens frequently
> enough that it becomes a performance bottleneck.
> 
> A further problem is that falling behind like this means that you're
> always processing outdated data.  This is a problem for a real-time
> network monitoring system, where late answers are often wrong answers.
> 
> We've adopted the solution of closing connections that get too much
> outgoing data buffered for them, by the simple expedient of raising an
> error in Connection.send if there's more than 50M of data already in
> the outbuffer before we append to it (by calling our superclass's send).
> (This could, of course, cause problems with large events, but as mentioned
> before, we don't handle those very well anyway.)  Our subscribers see the
> EOF on the tunnel and die, and then get restarted externally.  The error
> gets logged in pubsub.err.log and also in the individual client's log.
> 
> This solution prevents the slow client from rendering the entire machine
> unusably slow, but it still produces late answers.  So we have adopted a
> client-side solution to improve matters further.  Instead of analyzing
> each event as it arrives, we simply deserialize the events and store
> them in a hash (indexed by network node ID), and periodically analyze the
> entire hash --- the most recent data for each network node.  Events still
> pile up while we're analyzing, but the analyze time is bounded by the
> number of network nodes, so we keep up if we analyze rarely enough.
> 
> So the question remains: how do we analyze rarely enough?  We initially
> tried calling PubSub::Client::handle_events, then analyzing, and
> that worked up to a point.  It would read all the available events,
> then analyze them.  The trouble was, we could read the events into the
> hash fast enough that we could empty out our 128K of socket buffers
> and return from get_events before pubsub.py had a chance to get out of
> its select() and send us the other 4MB of data it had queued for us.
> So now we publish an event to a 'wake-up' topic, and analyze our hash
> of events when we receive that event back.  We don't publish that event
> again until we've gotten it back, so we know that when we receive it,
> we've received all the events that got sent to us during our previous
> analysis phase.
> 
> This seems to work reasonably well, but needs some kind of throttle to
> keep us from spending all of our time sending and handling the 'wake-up'
> event.  Right now, we sleep 0.3 seconds, which puts us in the
> uncomfortable position of adjusting a polling interval to overcome
> problems in our event system.  Perhaps instead we should wait for some
> other event to show up.
> 
> I'm curious who else is doing stuff like this, what problems you've
> run into, and what solutions you've come up with.  And when is the next
> mod-pubsub user's group?



-------------------------------------------------------
This SF.net email is sponsored by OSDN developer relations
Here's your chance to show off your extensive product knowledge
We want to know what you know. Tell us and you have a chance to win $100
http://www.zoomerang.com/survey.zgi?HRPT1X3RYQNC5V4MLNSV3E54