integrating mod_pubsub into our application

So I'm working on a project where we decided we needed to get
event-driven, and so we needed some piece of message-oriented
middleware.  I initially identified Spread (spread.org) and xmlBlaster
(xmlblaster.org) as the leading candidates for the application, but
other folks on the team thought the advertising clause in the Spread
license was too onerous and xmlBlaster was too much of a PITA to
interface with (don't ask me why, I didn't look at xmlBlaster in that
depth myself) so we're stuck with mod_pubsub.  I'm partly happy, in
that all that work I put in back in 2000 is starting to yield results
in making something human beings like to use, and partly nervous
because I'm pretty sure nobody else has deployed mod_pubsub in
production software before.  (By "production software," I mean
"software where failures can cost you a lot of money.") That's why I
didn't recommend it in the first place.  I guess we get to be the
first!

I want to emphasize that we're not using mod_pubsub's browser clients
in our production product, because software that runs in a web browser
takes a lot of effort to build and is generally pretty shaky anyway.
We're just using it as a pub-sub messaging system inside our software,
which is written in Perl.  The browser clients (especially
'introspect') are really handy for seeing what's going on, though.

So I was doing an initial spike tonight to see how well things would
work.  We have a couple of daemons; they were both grabbing some data
every 15 seconds and doing some processing with it.  Since they were
both grabbing the same data --- over the network --- I thought it
would be good for only one of them to grab the data and post it to a
topic, while the other one would subscribe to the topic and get its
data that way.  The data is a Perl data structure, so I used YAML to
serialize it for transmission in the kn_payload.  Works great so far,
and while we had to do a lot of refactoring to get here (previously,
getting the data over the network happened deep down inside of some
methods, and we had to refactor all the implementations of those
methods), there are presently only about ten lines of code related to
publishing and subscribing.  And it was great to be able to see the
data streams with chat.plx.  Looked like about an event every 2
seconds (on average --- actually clumps of about 6 events every 15
seconds) and about 250 bytes per second of kn_payload data.  (We're
using one custom header, which had 1 byte of data in the cases under
test.)

I ran 'top', though, and got scared.  Handling half an event per
second, pubsub.py took about 10% of the CPU; the publisher daemon used
about 2%-5% (and it's doing a lot of other things besides publishing
events) and the subscriber daemon used 0.1%-1%.  These are both using
the Perl microserver implementation (PubSub::Client).  This is running
on a 2.4GHz beast, so multiply all these numbers by 2400000000.

Is this typical?  Is it just a matter of overhead (e.g. a reaper task)
or is it actually processing events?  In deployment, we'd like to
handle 5x greater loads than I was on that dev box, but we can't
afford to devote 50% of our CPU to message-passing, and we'd like to
drastically increase the number of things that use it.

(It does look like it continues to use 10% of the CPU even when I stop
the publisher for a while.)

Along similar lines, under this event load, it looks like the
pubsub.py process has grown from 8MB to 10.4MB in about 2.5 hours.
That won't continue indefinitely, the way it used to, will it?

FWIW I did have several extraneous subscriptions to the topic in
question for things like chat.plx at various times, and these old
connections still showed up in who.esp, so I suspect the server was
still routing things to old journals.

I ran into a bunch of other minor nigglies that will probably impede
other people in the future.  Please don't take these as complaints; I
think mod_pubsub is wonderful, and since I have commit access,
anything that's wrong with it is my fault.  That said, here are some
unpleasant surprises you might run into, and some workarounds for
them:

- there's no MakeMaker-based installation for the PubSub Perl modules,
  so for the time being, I'm just copying the cgi-bin/PubSub directory
  into our site-perl directory.  I'll probably build a MakeMaker-based
  install script for the Perl modules so they integrate more easily
  into our product.
- in Galeon built on  Mozilla 0.9, 'introspect' and 'chatroom' are 
  pretty slow, and under the event load described here, your browser
  can get pretty unresponsive.  I had to use chat.plx to get real-time
  visibility on the event stream in a useful form.
- to run chat.plx, you have to give it the server URI.  That's not
  documented as far as I can tell.  Also, the server URI to use is
  something like 'http://localhost:8202/kn', and I would have had a
  hard time coming up with the '/kn' part if I hadn't had previous
  experience with the software.  Also, chat.plx gives you unhelpful
  diagnostics if you e.g. omit the server URI or give it the wrong
  one.
- for our purposes, we'd like to have pubsub.py bind only to the
  loopback interface (except for development, when it would be nice to
  use 'introspect' from a different machine).  It doesn't seem to
  have that feature.
- I tried hitting https://servername:8202/kn_apps/ from a different
  machine by accident; like most HTTP servers, it wasn't smart enough
  to recognize an SSL initialization sequence when it saw one, so this
  hung for a long time before I figured out what was going on.
- since my kn_payloads were built with YAML, where newlines have
  semantic relevance, they weren't very readable in introspect or
  chatroom, which treat kn_payload as HTML.  chat.plx was fine though.
- PubSub::Client still doesn't integrate nicely with event loops ---
  it doesn't have a ->handle_events_forever() method, although it
  certainly could, and it doesn't have a well-defined way to get its
  fd so you can write your own fd event loop that incorporates it.
  For the time being I'm just doing 
  for (;;) {$client->handle_events(); sleep(1) } because a second of
  latency isn't critical in this task.
- PubSub::Client isn't documented in the usual place, i.e. pod.
  chat.plx is pretty good documentation.
- unlike most daemonish processes, pubsub.py prints stuff out when you
  run it, and not to a nice place like /var/log/pubsub --- to your
  terminal.  It also keeps a logfile in the current directory which is
  useful mostly for debugging, and rapidly becomes quite large.



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf