Re: Patch for simple walk caching

Bill,

This sounds interesting.  I'll try to take a look at this tonight and 
get back to you.  My "production" pyblosxom runs off what's in CVS, and 
I just brought everything up to date so your patch should be fine.

Ted

On Dec 6, 2004, at 1:42 PM, Bill Mill wrote:

> Ted,
>
> I am indeed caching data between requests. Basically, my cache stores
> the fileentry from a request in a pickle. For example, if you call
> http://myserver/pyblosxom.cgi/2004/Nov , it caches the result of the
> fileentry callback as '2004/Nov'. Since it's called around the
> fileentry callback, it should (I hope) be caching the results of all
> your plugins. Also, it should be thread-safe, since it locks the cache
> files it uses before reading from/writing to them.
>
> I have tested this plugin with a large number of entries that I
> created with a script, but it hasn't been on anything nearing the
> demands of a production server. I load tested it on my local machine
> on both windows and linux w/Apache 2 on each, and it seemed to provide
> a significant performance boost. The patch that is included was
> created based on a recent cvs version of pyblosxom.py; if you want one
> based on an older version I can generate that for you.
>
> To install it, you'll first need to create a cache directory; it can
> be anywhere on your server. Then apply the attached patch to
> pyblosxom.py and put the filelist_cache.py file in your
> Pyblosxom/cache directory. Last, set py['cache_dir'] equal to your
> cache directory and py['cache_to'] to the value (in minutes) that
> describes how long you want the caches to last. The cache_dir variable
> is the switch that turns the cache on and off, so to disable it,
> simply comment out that variable. The relevant portion of my
> config.py:
>
> ###File Cache config
> py['cache_dir'] = "/c/code/web/cleanblox/cache_dir"
> #How soon, in minutes, the cache should expire
> py['cache_to'] = 60
>
> If you're willing to try it out, I would be tremendously appreciative.
> My blog has so few entries that I can't make any real judgement about
> it in a production environment. You can turn it off entirely at any
> time by commenting out the cache_dir variable, and I've invested a
> significant amount of time in it already, so I'd be eager to help you
> fix any problems.
>
> My big question is - how/when should the cache be refreshed? If you
> use the config above, comments won't show up for an hour. Should it
> not cache single entries? Should there be a form variable to refresh
> it - i.e. http://myserver/mypage?refresh=true ?
>
> Any feedback you have would be appreciated; sorry for the long email,
> and the patch and file are attached.
>
> Peace
> Bill Mill
> bill.mill at gmail.com
>
> On Mon, 6 Dec 2004 12:53:46 -0800, Ted Leung <[email protected]> wrote:
>> Bill,
>>
>> What will said is right, but I'd be interesting in testing your cache
>> changes if you are caching data between requests.
>>
>> Ted
>>
>>
>>
>> On Dec 6, 2004, at 8:23 AM, will guaraldi wrote:
>>
>>>
>>> Well, he's not trying to solve the request-to-request caching issue
>>> but rather the issue where in a single pyblosxom request, there are
>>> several calls to tools.Walk that walk the directory tree and bring
>>> back results. So if you've got your blog running with pyarchives,
>>> pycalendar, and pycategories, you do a tools.Walk on the directory
>>> three 3 or 4 times.
>>>
>>> Caching that reduces the number of times down to 1.
>>>
>>> /will
>>>
>>>
>>> On Mon, 6 Dec 2004, Bill Mill wrote:
>>>>
>>>> I've been working on a cache too. Are you using mod_python? If not,
>>>> doesn't a new python process with a new memory segment get created 
>>>> on
>>>> every request, meaning that an in-memory cache would never be read?
>>>>
>>>> Peace
>>>> Bill Mill
>>>> bill.mill at gmail.com
>>>>
>>>> On Mon, 6 Dec 2004 00:41:25 -0800, Ted Leung <[email protected]> wrote:
>>>>> Hi all,
>>>>>
>>>>> I've attached a small patch to tools.py which caches the results of
>>>>> calls to tools.Walk.  On my blog, I am using pycategories,
>>>>> pyarchives,
>>>>> and pycalendar, each of which do a tools.Walk of the entire 
>>>>> datadir.
>>>>> This was taking up a bunch of time, so I started looking for ways 
>>>>> to
>>>>> reduce this.
>>>>> At the moment the patch caches according to the value of the root
>>>>> passed to Walk, and assumes fixed values for the optional 
>>>>> parameters.
>>>>> This is good enough to reduce the number of calls related to
>>>>> tools.Walk
>>>>> dramatically.
>>>>>
>>>>> I could just commit the patch, but I wanted to get some feedback on
>>>>> whether folks thought this was a good thing.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I also have some hotshot traces of running my blog with various
>>>>> caching, etc turn on/off.
>>>>> ----
>>>>> Ted Leung                          Blog: 
>>>>> <http://www.sauria.com/blog>
>>>>> PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> -------------------------------------------------------
>>>> SF email is sponsored by - The IT Product Guide
>>>> Read honest & candid reviews on hundreds of IT Products from real
>>>> users.
>>>> Discover which products truly live up to the hype. Start reading 
>>>> now.
>>>> http://productguide.itmanagersjournal.com/
>>>> _______________________________________________
>>>> Pyblosxom-devel mailing list
>>>> Pyblosxom-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>>>> https://lists.sourceforge.net/lists/listinfo/pyblosxom-devel
>>>>
>>>
>>> --
>>> whatever it is, you can find it at http://www.bluesock.org/~willg/
>>> except Will--you can only see him in real life.
>>>
>>>
>>> -------------------------------------------------------
>>> SF email is sponsored by - The IT Product Guide
>>> Read honest & candid reviews on hundreds of IT Products from real
>>> users.
>>> Discover which products truly live up to the hype. Start reading now.
>>> http://productguide.itmanagersjournal.com/
>>> _______________________________________________
>>> Pyblosxom-devel mailing list
>>> Pyblosxom-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>>> https://lists.sourceforge.net/lists/listinfo/pyblosxom-devel
>>>
>> ----
>> Ted Leung                          Blog: <http://www.sauria.com/blog>
>> PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42
>> This message is:  [ ] bloggable   [x] ask first    [ ] private
>>
>>
> <pyblosxom.py.diff><filelist_cache.py>
----
Ted Leung                          Blog: <http://www.sauria.com/blog>
PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42
This message is:  [ ] bloggable   [x] ask first    [ ] private



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/