Re: Patch for simple walk caching

Ted,

I am indeed caching data between requests. Basically, my cache stores
the fileentry from a request in a pickle. For example, if you call
http://myserver/pyblosxom.cgi/2004/Nov , it caches the result of the
fileentry callback as '2004/Nov'. Since it's called around the
fileentry callback, it should (I hope) be caching the results of all
your plugins. Also, it should be thread-safe, since it locks the cache
files it uses before reading from/writing to them.

I have tested this plugin with a large number of entries that I
created with a script, but it hasn't been on anything nearing the
demands of a production server. I load tested it on my local machine
on both windows and linux w/Apache 2 on each, and it seemed to provide
a significant performance boost. The patch that is included was
created based on a recent cvs version of pyblosxom.py; if you want one
based on an older version I can generate that for you.

To install it, you'll first need to create a cache directory; it can
be anywhere on your server. Then apply the attached patch to
pyblosxom.py and put the filelist_cache.py file in your
Pyblosxom/cache directory. Last, set py['cache_dir'] equal to your
cache directory and py['cache_to'] to the value (in minutes) that
describes how long you want the caches to last. The cache_dir variable
is the switch that turns the cache on and off, so to disable it,
simply comment out that variable. The relevant portion of my
config.py:

###File Cache config
py['cache_dir'] = "/c/code/web/cleanblox/cache_dir"
#How soon, in minutes, the cache should expire
py['cache_to'] = 60

If you're willing to try it out, I would be tremendously appreciative.
My blog has so few entries that I can't make any real judgement about
it in a production environment. You can turn it off entirely at any
time by commenting out the cache_dir variable, and I've invested a
significant amount of time in it already, so I'd be eager to help you
fix any problems.

My big question is - how/when should the cache be refreshed? If you
use the config above, comments won't show up for an hour. Should it
not cache single entries? Should there be a form variable to refresh
it - i.e. http://myserver/mypage?refresh=true ?

Any feedback you have would be appreciated; sorry for the long email,
and the patch and file are attached.

Peace
Bill Mill
bill.mill at gmail.com

On Mon, 6 Dec 2004 12:53:46 -0800, Ted Leung <[email protected]> wrote:
> Bill,
> 
> What will said is right, but I'd be interesting in testing your cache
> changes if you are caching data between requests.
> 
> Ted
> 
> 
> 
> On Dec 6, 2004, at 8:23 AM, will guaraldi wrote:
> 
> >
> > Well, he's not trying to solve the request-to-request caching issue
> > but rather the issue where in a single pyblosxom request, there are
> > several calls to tools.Walk that walk the directory tree and bring
> > back results. So if you've got your blog running with pyarchives,
> > pycalendar, and pycategories, you do a tools.Walk on the directory
> > three 3 or 4 times.
> >
> > Caching that reduces the number of times down to 1.
> >
> > /will
> >
> >
> > On Mon, 6 Dec 2004, Bill Mill wrote:
> >>
> >> I've been working on a cache too. Are you using mod_python? If not,
> >> doesn't a new python process with a new memory segment get created on
> >> every request, meaning that an in-memory cache would never be read?
> >>
> >> Peace
> >> Bill Mill
> >> bill.mill at gmail.com
> >>
> >> On Mon, 6 Dec 2004 00:41:25 -0800, Ted Leung <[email protected]> wrote:
> >>> Hi all,
> >>>
> >>> I've attached a small patch to tools.py which caches the results of
> >>> calls to tools.Walk.  On my blog, I am using pycategories,
> >>> pyarchives,
> >>> and pycalendar, each of which do a tools.Walk of the entire datadir.
> >>> This was taking up a bunch of time, so I started looking for ways to
> >>> reduce this.
> >>> At the moment the patch caches according to the value of the root
> >>> passed to Walk, and assumes fixed values for the optional parameters.
> >>> This is good enough to reduce the number of calls related to
> >>> tools.Walk
> >>> dramatically.
> >>>
> >>> I could just commit the patch, but I wanted to get some feedback on
> >>> whether folks thought this was a good thing.
> >>>
> >>>
> >>>
> >>>
> >>> I also have some hotshot traces of running my blog with various
> >>> caching, etc turn on/off.
> >>> ----
> >>> Ted Leung                          Blog: <http://www.sauria.com/blog>
> >>> PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >> -------------------------------------------------------
> >> SF email is sponsored by - The IT Product Guide
> >> Read honest & candid reviews on hundreds of IT Products from real
> >> users.
> >> Discover which products truly live up to the hype. Start reading now.
> >> http://productguide.itmanagersjournal.com/
> >> _______________________________________________
> >> Pyblosxom-devel mailing list
> >> Pyblosxom-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> >> https://lists.sourceforge.net/lists/listinfo/pyblosxom-devel
> >>
> >
> > --
> > whatever it is, you can find it at http://www.bluesock.org/~willg/
> > except Will--you can only see him in real life.
> >
> >
> > -------------------------------------------------------
> > SF email is sponsored by - The IT Product Guide
> > Read honest & candid reviews on hundreds of IT Products from real
> > users.
> > Discover which products truly live up to the hype. Start reading now.
> > http://productguide.itmanagersjournal.com/
> > _______________________________________________
> > Pyblosxom-devel mailing list
> > Pyblosxom-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> > https://lists.sourceforge.net/lists/listinfo/pyblosxom-devel
> >
> ----
> Ted Leung                          Blog: <http://www.sauria.com/blog>
> PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42
> This message is:  [ ] bloggable   [x] ask first    [ ] private
> 
>

Index: pyblosxom.py
===================================================================
RCS file: /cvsroot/pyblosxom/pyblosxom/Pyblosxom/pyblosxom.py,v
retrieving revision 1.51
diff -r1.51 pyblosxom.py
387a388,405
> def cache_handler(request):
>     from cache import filelist_cache as fc
>     http = request.getHttp()
>     data = request.getData()
>     config = request.getConfiguration()
>     path = http.get('PATH_INFO', '').lstrip('/') or 'root'
>     f = fc.pathjoin(config.get("cache_dir", "/tmp"), path)
>     to = config.get('cache_to', 0)
>     c = fc.get_cache(f)
>     incache = c.get('filelist', None)
>     if incache and (time.time() - os.stat(f)[8]) / 60 < to: 
>         data["entry_list"] = incache
>     else:
>         data["entry_list"] = tools.run_callback("filelist",
>                            {"request": request},
>                            donefunc=lambda x:x != None,
>                            defaultfunc=blosxom_file_list_handler)
>         fc.put_cache({'filelist': data["entry_list"]}, f)
409c427
< 
---
>         
411,412c429,430
<         r = tools.importName("Pyblosxom.renderers", r)
< 
---
>         #r = tools.importName("Pyblosxom.renderers", r) #Why not work anymore?
>         r = tools.importName("renderers", r)
426,427c444,449
<     # call the filelist callback to generate a list of entries
<     data["entry_list"] = tools.run_callback("filelist",
---
>     res = 0
>     if config.get('cache_dir', ''):
>         cache_handler(request)
>     else:
>         # call the filelist callback to generate a list of entries
>         data["entry_list"] = tools.run_callback("filelist",

"""A caching system for pyblosxom. Works by storing the result of the fileentry
callback in cache_dir.

Configuration variables:
cache_dir: Directory in which to store cache files
cache_to:  Cache timeout in minutes"""

import cPickle, os
from Pyblosxom import tools

def pathjoin(*paths):
    return '/'.join(paths)

def get_cache(filename):
    """Load the cache of a file
    
    @param filename: The name of the file to check for a cache
    @type filename: String
    
    @returns: The associated cache Dictionary if it exists, else {}
    """
    try:
        f = file(filename, 'rb')
        tools.lock(f, tools.LOCK_EX)
        c = cPickle.load(f)
        f.close()
    except IOError: c = {}
    return c

def put_cache(cache, filename):
    """Write L{cache} to L{filename}

    @param cache: Dictionary to store in cache
    @type cache: Dictionary

    @param filename: name of file to put the cache in
    @type filename: String

    @returns: None
    """
    for c in cache['filelist']:
        c.getMetadata('')
        c._request = None
    try:
        f = file(filename, 'wb')
    except IOError:
        os.makedirs(os.path.split(filename)[0])
        f = file(filename, 'wb')
    tools.lock(f, tools.LOCK_EX)
    cb = cPickle.Pickler(f, -1)
    cb.dump(cache)
    f.close()