Re: Pool Module Question

"Conan C. Albrecht" <conan-RXmsyT/Szrj2fBVCVOL8/[email protected]>
Newsgroups gmane.comp.python.spyce.general
Message-ID <[email protected]>
The "multiple instances of spyce, hence multiple instances of the  
pool"  problem is related to mod_python, not spyce.  Mod_python runs  
multiple instances of itself in an effort to increase load.  The side  
effect to this is you get multiple pools.

The solution to this is to use one of the following:

1. (not recommended from my experience with it) FastCGI  
installation.  There is a setting in the FastCGI httpd.conf arguments  
that ensures it only runs one copy of itself.  I don't remember what  
it is, but if you search the FastCGI docs you can find it.  I had  
problems with FastCGI restarting itself quite often, though, so I  
dropped this one.

2. Proxy server installation of Spyce. This is what i run on my  
installations.  It works very well, although I've never load tested  
it to extremely high loads.  You essentially run Spyce separately  
from Apache -- as a different process.  When Apache sees a .spy  
request come in, it passes the request to the Spyce server (running  
on port 8000 or something) for handling.  This has worked extremely  
well for me.  With this method you have total control.

I've attached an init.d script that starts the spyce server that I  
use to start Spyce.  I like it better than the one in the Spyce  
docs.  Here's the httpd.conf settings for me (I removed some settings):

<VirtualHost 207.71.8.73>
   ServerAdmin <removed>
   DocumentRoot /home/usiecr2/html/
   ServerName <removed>
   ServerAlias <removed>
   ErrorLog /home/usiecr2/logs/error.log
   CustomLog /home/usiecr2/logs/access.log combined
   ErrorDocument 404 /notfound.spy

   RewriteEngine On
   # this one is for the spyce engine
   RewriteRule ^(.*\.spy) http://localhost:8000$1 [p]
</VirtualHost>


____________________________________
Conan C. Albrecht, Ph.D.
Information Systems Department
Brigham Young University
Web: http://warp.byu.edu/
Email: conan-RXmsyT/Szrj2fBVCVOL8/[email protected]

What's the smime.p7s attachment?  It's a digital signature that tells  
your email client that this email was really from me.


On Apr 14, 2006, at Fri, Apr 14, 2006 1:52 PM, Ben Ringold wrote:

> Hi,
>
> I administer a fairly high traffic Spyce/mod_python/Apache/MySQL  
> application that's having trouble keeping up with its traffic  
> growth rate.
>
> I'm trying to improve the performance by making use of data caching  
> for objects that can persist between requests.
>
> I implemented an expiring data cache using Spyce's pool module to  
> store the cached objects.  It appears that there are multiple  
> instances of the pool dictionary on a single server, rather than  
> the single pool I anticipated.  This is fine for objects that can  
> persist through the life of the server instance (database  
> connections etc) but many of the objects I'd like to cache would  
> only be relevent for a few minutes, so multiple pools don't seem to  
> offer much processing relief.
>
> Does every client thread have it's own pool?  Is there anyway to  
> implement a more unified data cache?
>
> Thanks in advance for any help.
>
> Regards,
>
> Ben Ringold
> Hypothetical Software
>
>
>
>
>
>
>
>
> ________________________________________________________________
> Sent via the WebMail system at hypotheticalsoftware.com
>
>
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting  
> language
> that extends applications into web and mobile media. Attend the  
> live webcast
> and join the prime developer group breaking into this new coding  
> territory!
> http://sel.as-us.falkag.net/sel? 
> cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> Spyce-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/spyce-users
run_daemon.py (text/x-python-script, 832 B)
#!/usr/bin/python

import daemonize
import os, os.path, sys

LOG_DIR = '/var/log/usiecr2'
LOG_FILE = os.path.join(LOG_DIR, 'usiecr2.log')
ERR_FILE = os.path.join(LOG_DIR, 'usiecr2.err')
PID_FILE = '/var/run/usiecr2.pid'
CONF_FILE = '/home/usiecr2/spyceconf.py'  # this dir CANNOT have a space in it or the split will fail below
SPYCE_DIR = '/usr/share/spyce'

# ensure the correct directories exist
if not os.path.exists(LOG_DIR):
  os.makedirs(LOG_DIR)

# first fork this process to a daemon
daemonize.startstop(stdout=LOG_FILE, stderr=ERR_FILE, pidfile=PID_FILE)

# change to the spyce directory
sys.path.append(SPYCE_DIR)
os.chdir(SPYCE_DIR)

# next call the spyce web server
# change argv to the new command we normally call to start the server
sys.argv = ('spyceCmd.py -l --conf ' + CONF_FILE).split(' ')
execfile(sys.argv[0])
daemonize.py (text/x-python-script, 4.7 KB)
#!/usr/bin/python

'''
    This module is used to fork the current process into a daemon.
    Almost none of this is necessary (or advisable) if your daemon 
    is being started by inetd. In that case, stdin, stdout and stderr are 
    all set up for you to refer to the network connection, and the fork()s 
    and session manipulation should not be done (to avoid confusing inetd). 
    Only the chdir() and umask() steps remain as useful.
    References:
        UNIX Programming FAQ
            1.7 How do I get my program to act like a daemon?
                http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16
        Advanced Programming in the Unix Environment
            W. Richard Stevens, 1992, Addison-Wesley, ISBN 0-201-56317-7.

    History:
      2001/07/10 by Jurgen Hermann
      2002/08/28 by Noah Spurrier
      2003/02/24 by Clark Evans
      
      http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012
'''
import sys, os, time
from signal import SIGTERM

def deamonize(stdout='/dev/null', stderr=None, stdin='/dev/null',
              pidfile=None, startmsg = 'started with pid %s' ):
    '''
        This forks the current process into a daemon.
        The stdin, stdout, and stderr arguments are file names that
        will be opened and be used to replace the standard file descriptors
        in sys.stdin, sys.stdout, and sys.stderr.
        These arguments are optional and default to /dev/null.
        Note that stderr is opened unbuffered, so
        if it shares a file with stdout then interleaved output
        may not appear in the order that you expect.
    '''
    # Do first fork.
    try: 
        pid = os.fork() 
        if pid > 0: sys.exit(0) # Exit first parent.
    except OSError, e: 
        sys.stderr.write("fork #1 failed: (%d) %s\n" % (e.errno, e.strerror))
        sys.exit(1)
        
    # Decouple from parent environment.
    os.chdir("/") 
    os.umask(0) 
    os.setsid() 
    
    # Do second fork.
    try: 
        pid = os.fork() 
        if pid > 0: sys.exit(0) # Exit second parent.
    except OSError, e: 
        sys.stderr.write("fork #2 failed: (%d) %s\n" % (e.errno, e.strerror))
        sys.exit(1)
    
    # Open file descriptors and print start message
    if not stderr: stderr = stdout
    si = file(stdin, 'r')
    so = file(stdout, 'a+')
    se = file(stderr, 'a+', 0)
    pid = str(os.getpid())
    sys.stderr.write("\n%s\n" % startmsg % pid)
    sys.stderr.flush()
    if pidfile: file(pidfile,'w+').write("%s\n" % pid)
    
    # Redirect standard file descriptors.
    os.dup2(si.fileno(), sys.stdin.fileno())
    os.dup2(so.fileno(), sys.stdout.fileno())
    os.dup2(se.fileno(), sys.stderr.fileno())

def startstop(stdout='/dev/null', stderr=None, stdin='/dev/null',
              pidfile='pid.txt', startmsg = 'started with pid %s' ):
    if len(sys.argv) > 1:
        action = sys.argv[1]
        try:
            pf  = file(pidfile,'r')
            pid = int(pf.read().strip())
            pf.close()
        except IOError:
            pid = None
        if 'stop' == action or 'restart' == action:
            if not pid:
                mess = "Could not stop, pid file '%s' missing.\n"
                sys.stderr.write(mess % pidfile)
                if 'stop' == action:
                    sys.exit(1)
                action = 'start'
                pid = None
            else:
               try:
                  while 1:
                      os.kill(pid,SIGTERM)
                      time.sleep(1)
               except OSError, err:
                  err = str(err)
                  if err.find("No such process") > 0:
                      os.remove(pidfile)
                      if 'stop' == action:
                          sys.exit(0)
                      action = 'start'
                      pid = None
                  else:
                      print str(err)
                      sys.exit(1)
            time.sleep(5)

        if 'start' == action:
            if pid:
                mess = "Start aborded since pid file '%s' exists.\n"
                sys.stderr.write(mess % pidfile)
                sys.exit(1)
            deamonize(stdout,stderr,stdin,pidfile,startmsg)
            return
    print "usage: %s start|stop|restart" % sys.argv[0]
    sys.exit(2)

def test():
    '''
        This is an example main function run by the daemon.
        This prints a count and timestamp once per second.
    '''
    sys.stdout.write ('Message to stdout...')
    sys.stderr.write ('Message to stderr...')
    c = 0
    while 1:
        sys.stdout.write ('%d: %s\n' % (c, time.ctime(time.time())) )
        sys.stdout.flush()
        c = c + 1
        time.sleep(1)

if __name__ == "__main__":
    startstop(stdout='/tmp/deamonize.log',
              pidfile='/tmp/deamonize.pid')
    test()
smime.p7s (application/pkcs7-signature, 2.3 KB) - not displayed
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.