Re: Organizing a Quixote/Durus multiprocess application

"Mike Orr" <[email protected]> Thu, 31 Aug 2006 16:26:34 -0700

Newsgroups	gmane.comp.web.quixote.user,gmane.comp.python.durus
Message-ID	<[email protected]>

Here's what I've got so far.  Does it look reasonable?

I first divert sys.stdout and sys.stderr to a logfile unless envvar
"NO_LOG" is set.  This will be used by all subprocesses.  I don't want
some output sent to the console or swallowed up depending on where the
exception hits.  Then I fork and (in the child) start the Durus server
for the sessions db, and (in the parent) start the SCGI server.  I
catch KeyboardInterrupt in both cases to avoid a traceback in the log
file.  I'm using None for both the Quixote error log and the Durus
error log to prevent those packages from redirecting it further.

It took a while to get the file ownerships and umask correct, but I
finally got it working as user 'apache' from the command line.
But if I start it as a daemon and then stop it, it doesn't kill the
Durus server.  That must be because it's using SIGTERM instead of
SIGINT.  I added some code to kill it manually if it's still running,
but that doesn't help.

I may switch to a TCP socket if it gets too frustrating managing the
Unix socket, but I don't think that will help the current problem

Another issue is getting rid of Durus's "sys.stdout already customized.
sys.stderr already customized." messages in the log file.  I set
logginglevel to 101 but that doesn't help because the messages are
sent before that gets set.  I may replace durus.logging.direct_output
to get rid of them.

Another issue is in scgi_server.  I sometimes get a KeyError on
"self.reap_children(self.children[pid])".  I assume that means the
SCGI subprocess has already died, because one of them gave a
permission exception earlier.  I think scgi_server should silently
ignore this error?

I am happy that Quixote and Durus and scgi_server are simple enough
you can follow the code and usually fix problems yourself.  That has
helped me quite a bit in my last few applications.  I'm just getting
into new territory with Unix sockets and fork and how child processes
terminate.

-- 
Mike Orr <[email protected]>

_______________________________________________
Quixote-users mailing list
[email protected]
http://mail.mems-exchange.org/mailman/listinfo/quixote-users

cameo_server.py (text/x-python, 2.6 KB)

#!/usr/bin/env python
import codecs, os, sys
import init_app   # Must import before any cameo or hazweb modules.
import config

os.umask(02)  # Create all files/sockets mode "rw-rw-r--".
if not os.environ.get("NO_LOG"):
    _logstream = open(config.error_log, "ab", 1)
    sys.stdout = _logstream
    sys.stderr = _logstream
    del _logstream

import errno,  os, signal, sys, time
from durus.client_storage import ClientStorage
from durus.connection import Connection
from durus.run_durus import start_durus, stop_durus
from durus.storage_server import SocketAddress
import quixote
from quixote.publish import Publisher
from quixote.server import scgi_server
from session2.SessionManager import SessionManager
from session2.store.DurusSessionStore import DurusSessionStore
import init_app
import config
from cameo.controllers.root import RootDirectory
from cameo.logger import CameoLogger
from cameo.model.cameo_api import Cameo
from cameo.session import Session

# SCGI parameters
HOST = "127.0.0.1"
PORT = 3004
MAX_CHILDREN = 3

quixote.DEFAULT_CHARSET = config.output_encoding
def session_manager():
    """For the multiuser web server."""
    storage = ClientStorage(address=config.sessions_socket)
    conn = Connection(storage, cache_size=100)
    store = DurusSessionStore(conn)
    return SessionManager(store, Session)

def create_publisher():
    logger = CameoLogger(
        access_log=config.access_log,
        error_log=None,  # Already handled at top of this module.
        error_email=None)
    publisher = Publisher(RootDirectory(),
        display_exceptions=config.display_exceptions,
        session_cookie_name="CAMEO_Session", 
        session_manager=session_manager(),
        logger=logger)
    config.cameo = Cameo(config.db_file, readonly=True)
    config.cameo.precache()
    return publisher

def log_friendly(func, *args, **kw):
    try:
        func(*args, **kw)
    except KeyboardInterrupt:
        pass  # Don't put unnecessary traceback in the log file.

def main():
    pid = os.fork()  # Raises OSError.
    if pid == 0:  # Child process.
        address = SocketAddress.new(address=config.sessions_socket)
        #print "In Durus process", os.getpid()
        log_friendly(start_durus, logfile=None, logginglevel=99, 
            file=config.sessions_file,
            repair=False, readonly=False, address=address)
        return
    # Parent process
    log_friendly(scgi_server.run, create_publisher, host=HOST, port=PORT,
        max_children=MAX_CHILDREN)
    try:
        os.kill(pid, signal.SIGTERM)
    except OSError, e:
        if e.errno != errno.NOSCH:  # No such process.        
            raise

if __name__ == "__main__":  main()