Re: integrating mod-pubsub into our application

Joyce writes:
> So we have made minor improvements in performance since you last looked at the
> Python server, almost all having to do with reaping events.  But we have a
> couple of blocker problems right now that have prevented us from quickly
> improving performance.
> 
> 1) Sittler and Rifkin have been explicitly barred from checking in code that
> improves performance, since KN figures that's one of their competitive
> advantages.  I believe they can answer questions, have design discussions, and
> fix bugs -- but no checkins.

That's kind of silly --- KnowNow's server historically was only marginally
faster than pubsub.py, if that, but it supports clustering and has been
deployed in production by dozens of people for years now.  As long as
they keep developing and enhancing the technology, they have nothing to fear.

> 2) We're dithering over switching from asyncore to Twisted, and as often
> happens have wedged ourselves into a situation where it seems hardly worthwhile
> to fix the old but way too much work to learn the new.

Yeah, I like Twisted, and it might be worthwhile; and I understand how
it's possible to get wedged that way.  I mean, you have *no idea* how
much I procrastinate in the name of perfectionism.

> 3) To effectively rid ourselves of dead topics and journals -- which I believe
> accounts for almost all the memory bloat -- would probably require a change to
> the object model.

In our case, there shouldn't be any dead topics or journals --- the
subscribers are all long-lived, and there's only one other (non-journal)
topic at present.

> The CPU thing could actually get better very quickly.  Right now our reaping
> process happens every second, which is what pins your CPU at 10%.  You can
> change that by changing the third argument in this line:
> 
> ServerSaver(self, poolfile, 1)
> 
> to some longer interval.  That would increase memory use, but mostly
> temporarily.  Obviously it would be even better if we could override that
> interval on the command line.

Setting that to 10 seems to have reduced CPU usage significantly --- to
0.3% or so.  That's a little outrageous, but maybe something else has
changed --- e.g. today's messages are smaller.

> The other thing, reaping topics and possibly routes, is a bigger problem.  I
> get the feeling that you'd rather reduce CPU at the expense of memory, as long
> as the memory usage doesn't get out of hand -- and for your task, which has
> lots of events but few topics and routes, that's not too hard.  So maybe it's
> not a big enough pain point for you right now to look at it.  But if you'd like
> to, Ben is the only one who can help you.

Help me, Ben Kenobi, you're my only hope!  But actually I don't think
we'll have too much trouble with that.  At present we can probably
restart the server if it gets too big.

I'm not quite sure how we'll cope with failures in the server in
general.  For this particular application --- where the job is simply to
collect some data periodically and record it --- it's probably
acceptable to just lose messages due to temporary failures, as long as
it doesn't happen too often.  I think I've seen a worse problem, though,
when we restart pubsub.py.

We have a 'nanny' daemon that restarts daemons that have died.  When I
kill pubsub.py, the subscriber attached to it dies (as it probably
should, since the Perl microserver doesn't support tunnel restarts) and
then the nanny daemon comes along and restarts them both.  It looks like
the subscriber doesn't actually succeed in subscribing, though --- it's
not getting any events.  If I restart it again, it starts receiving
events again.  I'll investigate this further soon.

I've attached a patch that makes http://whatever/kn/server-status list
a little bit more detail about tunnels --- specifically what journal
topic they're attached to --- in order to facilitate debugging problems
like this.  (It was previously possible to look in the log file for this
information...) Now I just need some useful tool to list the destinations
of routes from some topic.

It also sets ServerSaver to happen less frequently, as you suggested.

I haven't committed this yet because I want to make sure my employer is
happy with the idea.

--- python_pubsub/pubsub.py	2003-07-23 11:13:35.000000000 -0700
+++ pubsub/pubsub.py	2003-09-14 16:26:01.000000000 -0700
@@ -633,18 +633,21 @@
     SimpleTunnel, JavaScriptTunnel, and (indirectly) FlashTunnel.
     """
 
-    def __init__(self, connection):
+    def __init__(self, connection, topicname='(no topic)'):
         Route.__init__(self,
                        {'kn_payload': str(connection),
                         'kn_expires': 'infinity',
                         'stale': '0'})
-        self.conn = connection
-        self.conn.report_status("tunnel")
+        self.conn, self.topicname = connection, topicname
+        self.report_status("open")
         self.header_sent = 0
         self.dead = 0
-        
-    def post(self, event):
 
+    def report_status(self, statusmsg):
+        self.conn.report_status('tunnel from %s: %s' %
+                                (self.topicname, statusmsg))
+
+    def post(self, event):
         # Tunnel's heartbeat mechanism.
         class TunnelTickler:
             def __init__(self, route, conn):
@@ -674,7 +677,7 @@
             self.conn.send(self.headerfrom(event))
             TunnelTickler(self, self.conn)
             self.header_sent = 1
-        self.conn.report_status("tunnel sending event %s" % event['kn_id'])
+        self.report_status("tunnel sending event %s" % event['kn_id'])
         self.conn.tickle_renderer = self.needs_tickling()
         self.conn.send(self.encode(event))
         return 1
@@ -763,8 +766,8 @@
         Format events in the kn_response_format=js format.
     """
 
-    def __init__(self, conn, watching):
-        Tunnel.__init__(self, conn)
+    def __init__(self, conn, topicname='(no topic)', watching=0):
+        Tunnel.__init__(self, conn, topicname)
         self.watching = watching
         self._jsTunnel_conn = conn
 
@@ -905,7 +908,7 @@
         self.verbose = verbose
         self.ignorePrologue = ignorePrologue
         self.root_topic = read_event_pool(poolfile)
-        ServerSaver(self, poolfile, 1)
+        ServerSaver(self, poolfile, 10)
         self.portnum = portnum
         self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
@@ -1029,20 +1032,19 @@
         def __call__(self):
             self.route.close()
             self.route.become_stale()
+    topic = route_get_topic(conn, uri, query)
     if query.has_key('kn_response_format'):
         if query['kn_response_format'][0] == 'simple':
-            route = SimpleTunnel(conn)
+            route = SimpleTunnel(conn, topic.getname())
         elif query['kn_response_format'][0] == 'flash':
-            route = FlashTunnel(conn)
+            route = FlashTunnel(conn, topic.getname())
         elif query['kn_response_format'][0] == 'js':
-            route = JavaScriptTunnel(conn, watching=1)
+            route = JavaScriptTunnel(conn, topic.getname(), watching=1)
         else:
             raise "Unsupported response format"
     else: # js is the default kn_response_format.
-        route = JavaScriptTunnel(conn, watching=1)
-    topic = route_get_topic(conn, uri, query)
+        route = JavaScriptTunnel(conn, topic.getname(), watching=1)
     topic.create_route(route)
-    conn.report_status('tunnel from %s' % topic.getname())
     route.post(status_event(query, '200 Watching topic',
                             'watching %s' % topic.getname()))
     # And DON'T finish sending.  Not until much later.  But there might be expiry:



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf