Re: integrating mod_pubsub into our application

Kragen,

So we have made minor improvements in performance since you last looked at the
Python server, almost all having to do with reaping events.  But we have a
couple of blocker problems right now that have prevented us from quickly
improving performance.

1) Sittler and Rifkin have been explicitly barred from checking in code that
improves performance, since KN figures that's one of their competitive
advantages.  I believe they can answer questions, have design discussions, and
fix bugs -- but no checkins.

2) We're dithering over switching from asyncore to Twisted, and as often
happens have wedged ourselves into a situation where it seems hardly worthwhile
to fix the old but way too much work to learn the new.

3) To effectively rid ourselves of dead topics and journals -- which I believe
accounts for almost all the memory bloat -- would probably require a change to
the object model.

The CPU thing could actually get better very quickly.  Right now our reaping
process happens every second, which is what pins your CPU at 10%.  You can
change that by changing the third argument in this line:

ServerSaver(self, poolfile, 1)

to some longer interval.  That would increase memory use, but mostly
temporarily.  Obviously it would be even better if we could override that
interval on the command line.

The other thing, reaping topics and possibly routes, is a bigger problem.  I
get the feeling that you'd rather reduce CPU at the expense of memory, as long
as the memory usage doesn't get out of hand -- and for your task, which has
lots of events but few topics and routes, that's not too hard.  So maybe it's
not a big enough pain point for you right now to look at it.  But if you'd like
to, Ben is the only one who can help you.

Hope this helps, thanks for the other feedback, JP


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf