Re: Pool Module Question
"Conan C. Albrecht" <conan-RXmsyT/Szrj2fBVCVOL8/[email protected]>
| Newsgroups | gmane.comp.python.spyce.general |
|---|---|
| Message-ID | <[email protected]> |
The "multiple instances of spyce, hence multiple instances of the pool" problem is related to mod_python, not spyce. Mod_python runs multiple instances of itself in an effort to increase load. The side effect to this is you get multiple pools. The solution to this is to use one of the following: 1. (not recommended from my experience with it) FastCGI installation. There is a setting in the FastCGI httpd.conf arguments that ensures it only runs one copy of itself. I don't remember what it is, but if you search the FastCGI docs you can find it. I had problems with FastCGI restarting itself quite often, though, so I dropped this one. 2. Proxy server installation of Spyce. This is what i run on my installations. It works very well, although I've never load tested it to extremely high loads. You essentially run Spyce separately from Apache -- as a different process. When Apache sees a .spy request come in, it passes the request to the Spyce server (running on port 8000 or something) for handling. This has worked extremely well for me. With this method you have total control. I've attached an init.d script that starts the spyce server that I use to start Spyce. I like it better than the one in the Spyce docs. Here's the httpd.conf settings for me (I removed some settings): <VirtualHost 207.71.8.73> ServerAdmin <removed> DocumentRoot /home/usiecr2/html/ ServerName <removed> ServerAlias <removed> ErrorLog /home/usiecr2/logs/error.log CustomLog /home/usiecr2/logs/access.log combined ErrorDocument 404 /notfound.spy RewriteEngine On # this one is for the spyce engine RewriteRule ^(.*\.spy) http://localhost:8000$1 [p] </VirtualHost> ____________________________________ Conan C. Albrecht, Ph.D. Information Systems Department Brigham Young University Web: http://warp.byu.edu/ Email: conan-RXmsyT/Szrj2fBVCVOL8/[email protected] What's the smime.p7s attachment? It's a digital signature that tells your email client that this email was really from me. On Apr 14, 2006, at Fri, Apr 14, 2006 1:52 PM, Ben Ringold wrote: > Hi, > > I administer a fairly high traffic Spyce/mod_python/Apache/MySQL > application that's having trouble keeping up with its traffic > growth rate. > > I'm trying to improve the performance by making use of data caching > for objects that can persist between requests. > > I implemented an expiring data cache using Spyce's pool module to > store the cached objects. It appears that there are multiple > instances of the pool dictionary on a single server, rather than > the single pool I anticipated. This is fine for objects that can > persist through the life of the server instance (database > connections etc) but many of the objects I'd like to cache would > only be relevent for a few minutes, so multiple pools don't seem to > offer much processing relief. > > Does every client thread have it's own pool? Is there anyway to > implement a more unified data cache? > > Thanks in advance for any help. > > Regards, > > Ben Ringold > Hypothetical Software > > > > > > > > > ________________________________________________________________ > Sent via the WebMail system at hypotheticalsoftware.com > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Spyce-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/spyce-users
run_daemon.py
(text/x-python-script, 832 B)
#!/usr/bin/python
import daemonize
import os, os.path, sys
LOG_DIR = '/var/log/usiecr2'
LOG_FILE = os.path.join(LOG_DIR, 'usiecr2.log')
ERR_FILE = os.path.join(LOG_DIR, 'usiecr2.err')
PID_FILE = '/var/run/usiecr2.pid'
CONF_FILE = '/home/usiecr2/spyceconf.py' # this dir CANNOT have a space in it or the split will fail below
SPYCE_DIR = '/usr/share/spyce'
# ensure the correct directories exist
if not os.path.exists(LOG_DIR):
os.makedirs(LOG_DIR)
# first fork this process to a daemon
daemonize.startstop(stdout=LOG_FILE, stderr=ERR_FILE, pidfile=PID_FILE)
# change to the spyce directory
sys.path.append(SPYCE_DIR)
os.chdir(SPYCE_DIR)
# next call the spyce web server
# change argv to the new command we normally call to start the server
sys.argv = ('spyceCmd.py -l --conf ' + CONF_FILE).split(' ')
execfile(sys.argv[0])
daemonize.py
(text/x-python-script, 4.7 KB)
#!/usr/bin/python
'''
This module is used to fork the current process into a daemon.
Almost none of this is necessary (or advisable) if your daemon
is being started by inetd. In that case, stdin, stdout and stderr are
all set up for you to refer to the network connection, and the fork()s
and session manipulation should not be done (to avoid confusing inetd).
Only the chdir() and umask() steps remain as useful.
References:
UNIX Programming FAQ
1.7 How do I get my program to act like a daemon?
http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16
Advanced Programming in the Unix Environment
W. Richard Stevens, 1992, Addison-Wesley, ISBN 0-201-56317-7.
History:
2001/07/10 by Jurgen Hermann
2002/08/28 by Noah Spurrier
2003/02/24 by Clark Evans
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012
'''
import sys, os, time
from signal import SIGTERM
def deamonize(stdout='/dev/null', stderr=None, stdin='/dev/null',
pidfile=None, startmsg = 'started with pid %s' ):
'''
This forks the current process into a daemon.
The stdin, stdout, and stderr arguments are file names that
will be opened and be used to replace the standard file descriptors
in sys.stdin, sys.stdout, and sys.stderr.
These arguments are optional and default to /dev/null.
Note that stderr is opened unbuffered, so
if it shares a file with stdout then interleaved output
may not appear in the order that you expect.
'''
# Do first fork.
try:
pid = os.fork()
if pid > 0: sys.exit(0) # Exit first parent.
except OSError, e:
sys.stderr.write("fork #1 failed: (%d) %s\n" % (e.errno, e.strerror))
sys.exit(1)
# Decouple from parent environment.
os.chdir("/")
os.umask(0)
os.setsid()
# Do second fork.
try:
pid = os.fork()
if pid > 0: sys.exit(0) # Exit second parent.
except OSError, e:
sys.stderr.write("fork #2 failed: (%d) %s\n" % (e.errno, e.strerror))
sys.exit(1)
# Open file descriptors and print start message
if not stderr: stderr = stdout
si = file(stdin, 'r')
so = file(stdout, 'a+')
se = file(stderr, 'a+', 0)
pid = str(os.getpid())
sys.stderr.write("\n%s\n" % startmsg % pid)
sys.stderr.flush()
if pidfile: file(pidfile,'w+').write("%s\n" % pid)
# Redirect standard file descriptors.
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())
def startstop(stdout='/dev/null', stderr=None, stdin='/dev/null',
pidfile='pid.txt', startmsg = 'started with pid %s' ):
if len(sys.argv) > 1:
action = sys.argv[1]
try:
pf = file(pidfile,'r')
pid = int(pf.read().strip())
pf.close()
except IOError:
pid = None
if 'stop' == action or 'restart' == action:
if not pid:
mess = "Could not stop, pid file '%s' missing.\n"
sys.stderr.write(mess % pidfile)
if 'stop' == action:
sys.exit(1)
action = 'start'
pid = None
else:
try:
while 1:
os.kill(pid,SIGTERM)
time.sleep(1)
except OSError, err:
err = str(err)
if err.find("No such process") > 0:
os.remove(pidfile)
if 'stop' == action:
sys.exit(0)
action = 'start'
pid = None
else:
print str(err)
sys.exit(1)
time.sleep(5)
if 'start' == action:
if pid:
mess = "Start aborded since pid file '%s' exists.\n"
sys.stderr.write(mess % pidfile)
sys.exit(1)
deamonize(stdout,stderr,stdin,pidfile,startmsg)
return
print "usage: %s start|stop|restart" % sys.argv[0]
sys.exit(2)
def test():
'''
This is an example main function run by the daemon.
This prints a count and timestamp once per second.
'''
sys.stdout.write ('Message to stdout...')
sys.stderr.write ('Message to stderr...')
c = 0
while 1:
sys.stdout.write ('%d: %s\n' % (c, time.ctime(time.time())) )
sys.stdout.flush()
c = c + 1
time.sleep(1)
if __name__ == "__main__":
startstop(stdout='/tmp/deamonize.log',
pidfile='/tmp/deamonize.pid')
test()
smime.p7s
(application/pkcs7-signature, 2.3 KB) - not displayed