using mod_pubsub in the middle of a production app

So we've been slowly increasing our use of mod_pubsub in our product,
and I thought I'd share my experience with all of you.

First, the good news: we're mostly pretty happy with the improvements
in performance and maintainability that are coming from restructuring
most of our IPC to use mod_pubsub.  It's easy to integrate.

A note about YAML: we tried using YAML to serialize Perl data structures,
but it was amazingly slow.  A small hash of 1000 entries took 0.1 seconds
to serialize and deserialize.  Now we're using Data::Dumper to serialize,
and Perl eval to deserialize, which is about 100 times as fast.

We're using the Perl microserver.

One of the first problems we hit was that the microserver's 'subscribe'
call wasn't persistent enough for us.  Normally we start pubsub.py
(the python_pubsub server) at the same time as all the other daemons.
Sometimes it takes longer to initialize than the other daemons, so
they start trying to subscribe too early, and get 'connection refused'.
So we added a 'subscribe_blocking' method that retries every few seconds
for a couple of minutes.

Another one is that the microserver's PubSub::Tunnel::parse_events
function is deeply recursive, if you're processing a large batch of
events, and at least Perl 5.8 seems to use a lot of memory in this case.
(Probably because each stack frame has its own local copy of the remaining
data in the network buffer.)  We fixed this by breaking it apart into
parse_event, which returns zero or one events, and parse_events, which
is a loop around parse_event.

A minor problem is that PubSub::Client::handle_events doesn't have any way
to block until there are events, so we settle for calling it frequently.
It's pretty efficient when there aren't any events.

One of the reasons for wanting to run things in an event-driven fashion
is to decouple the different parts of your software.  For example,
our software monitors network nodes and periodically collects some data
from them, which several programs analyze as they get it.  It's nice for
the data collector to have a separate set of resources, and not share
failure modes, with the data analysis programs.  So the data collector
posts its data to a topic as it reads it, and the data analyzer subscribes
to the topic and analyzes data as it can, which is often slower than
the collector can collect it.

This causes some big problems with pubsub.py.  As soon as an event is
posted, it appends an encoded copy of the event to the outbound network
buffers of all the subscribing tunnels, then tries to flush them.
When the client sucking events down the tunnel isn't keeping up with
the event stream, then eventually the network buffers associated with
the client's socket and the socket on pubsub.py's side fill up, and
pubsub.py starts getting EAGAIN errors.

It handles these temporary failures "correctly" in that it keeps
the data buffered and retries later when it can, but this creates a
secondary problem.  The outgoing data buffer can get very large; I've
observed over 100MB.  Events in the outgoing data buffer don't expire,
and don't get updated by more recent events; this happens by design,
although I can't remember why.  But eventually the unbounded growth of
this buffer starts paging everything else out to disk.

It's even worse than that, actually, because the buffer is just a Python
string.  So when we append to it, we create a new string, copy the old
contents into it, and append the new data.  This happens frequently
enough that it becomes a performance bottleneck.

A further problem is that falling behind like this means that you're
always processing outdated data.  This is a problem for a real-time
network monitoring system, where late answers are often wrong answers.

We've adopted the solution of closing connections that get too much
outgoing data buffered for them, by the simple expedient of raising an
error in Connection.send if there's more than 50M of data already in
the outbuffer before we append to it (by calling our superclass's send).
(This could, of course, cause problems with large events, but as mentioned
before, we don't handle those very well anyway.)  Our subscribers see the
EOF on the tunnel and die, and then get restarted externally.  The error
gets logged in pubsub.err.log and also in the individual client's log.

This solution prevents the slow client from rendering the entire machine
unusably slow, but it still produces late answers.  So we have adopted a
client-side solution to improve matters further.  Instead of analyzing
each event as it arrives, we simply deserialize the events and store
them in a hash (indexed by network node ID), and periodically analyze the
entire hash --- the most recent data for each network node.  Events still
pile up while we're analyzing, but the analyze time is bounded by the
number of network nodes, so we keep up if we analyze rarely enough.

So the question remains: how do we analyze rarely enough?  We initially
tried calling PubSub::Client::handle_events, then analyzing, and
that worked up to a point.  It would read all the available events,
then analyze them.  The trouble was, we could read the events into the
hash fast enough that we could empty out our 128K of socket buffers
and return from get_events before pubsub.py had a chance to get out of
its select() and send us the other 4MB of data it had queued for us.
So now we publish an event to a 'wake-up' topic, and analyze our hash
of events when we receive that event back.  We don't publish that event
again until we've gotten it back, so we know that when we receive it,
we've received all the events that got sent to us during our previous
analysis phase.

This seems to work reasonably well, but needs some kind of throttle to
keep us from spending all of our time sending and handling the 'wake-up'
event.  Right now, we sleep 0.3 seconds, which puts us in the
uncomfortable position of adjusting a polling interval to overcome
problems in our event system.  Perhaps instead we should wait for some
other event to show up.

I'm curious who else is doing stuff like this, what problems you've
run into, and what solutions you've come up with.  And when is the next
mod-pubsub user's group?


-------------------------------------------------------
This SF.net email is sponsored by OSDN developer relations
Here's your chance to show off your extensive product knowledge
We want to know what you know. Tell us and you have a chance to win $100
http://www.zoomerang.com/survey.zgi?HRPT1X3RYQNC5V4MLNSV3E54