Re: Patch for simple walk caching

On Dec 6, 2004, at 6:29 AM, will guaraldi wrote:

> On Mon, 6 Dec 2004, Ted Leung wrote:
>>
>> I've attached a small patch to tools.py which caches the results of 
>> calls to tools.Walk.  On my blog, I am using pycategories, 
>> pyarchives, and pycalendar, each of which do a tools.Walk of the 
>> entire datadir. This was taking up a bunch of time, so I started 
>> looking for ways to reduce this. At the moment the patch caches 
>> according to the value of the root passed to Walk, and assumes fixed 
>> values for the optional parameters.  This is good enough to reduce 
>> the number of calls related to tools.Walk dramatically.
>>
>> I could just commit the patch, but I wanted to get some feedback on 
>> whether folks thought this was a good thing.
>
> This is at least the third or fourth patch I've seen that attempts to 
> solve the index caching problem by creating a module-level cache for 
> tools.Walk results.  To date, the reason I've been giving that we 
> should reject these types of patches is that we didn't want anything 
> stored globally because if you run PyBlosxom in some CGI execution 
> environments (and I'm really vague on this--it was more of a Wari 
> thing), then there is the possibility that there are multiple threads 
> of execution involved.

Are those environments actually configurations that we want to support?

>
> So, my thinking is that if this fixes peoples' stuff, then they should 
> go ahead and apply the patch locally, but we shouldn't apply this 
> patch to the codebase because it doesn't fit our architectural 
> requirements.

Ok, I'll keep this local for now.  But I think that we need to look a a 
decent way of solving this problem.  If we have to have a thread safe 
cache for mod_python or whatever, that's fine by me.

My blog is up to almost 1200 entries and I've been getting complaints 
about the slowness.  I turned on entryshelve caching which helped some, 
and this cache patch helped a lot.  In my blog the bottleneck is now 
tools.walk_internal, and I'm thinking about implementing a 2.3 only 
version to see if os.walk is any better.

----
Ted Leung                          Blog: <http://www.sauria.com/blog>
PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/