Re: Patch for simple walk caching

You know, this sounds an awful lot like static rendering.  Especially 
considering I was trying to solve the exact same problems.

The difference being that static rendering caches the finally rendered 
page while your caching thing is caching the entry list.  Is there an 
advantage to caching the entry list as opposed to the finally rendered 
page?

/will


On Tue, 7 Dec 2004, Bill Mill wrote:
>
> That's excellent; I'm glad it's speeding things up. I knew that I
> still had issues with how to invalidate the cache, but I couldn't
> really resolve them until a) I knew the cache actually speeded things
> up in real life and b) I could talk with some other people about it.
>
> More comments below.
>
> On Tue, 7 Dec 2004 00:51:21 -0800, Ted Leung <[email protected]> wrote:
>> Hi Bill,
>>
>> I tried out your cache, and it appears to make a sizable difference.
>> I've attached 3 hotshot dumps that show the various combinations.
>>
>> pbls-nocache-prof.txt shows the results before I did any improvements
>> to the blog.
>> pbls-walk-cache-prof.txt shows the results after entryshelve caching
>> and the walk patch that I posted.
>> pbls-filecache.prof.txt shows the results after using your cache.
>>
>> You can't make much hay about the actual times because I ran these
>> profiles on my production blog and there were other processes running
>> at the same time, but the drop in function calls is appreciable, and
>> the impact on response time is noticeable.
>>
>> The big outstanding issue is how to invalidate the cache.  This is
>> symptomatic of one of the problems with pyblosxom's plugin
>> architecture: it is hard for plugins that know about each other to
>> co-operate with each other.  I would like see the filecache "pass on"
>> the cached data that it using to the rest of the plugin pipeline, so
>> that plugins later in the chain could invalidate the cache (presumably
>> by deleting the appropriate pickle -- this would be a function in
>> filelist_cache.py.   That way both the comment plugin and the
>> xmlrpc_metaweblog plugins could invalidate the cache and the next
>> request would cause the cache to get filled.
>>
>
> What if there was a function invalidate_cache(path) - and when a
> plugin received an update to a page, it would invalidate the cache for
> that path. From the client (plugin) side, ideally it would look
> something like:
>
> if config['cache_dir']:                        #if using cache
>    invalidate(path_to_changed_file)   #invalidate the path
>
> the problem is from the cache server side - how does it figure out
> which caches include that file? Does it open and search through all of
> them? Can you launch a background thread from a CGI script? If you
> can, then the user shouldn't experience too much delay, and his
> changes should be implemented soon but not instantly. If not, he's
> going to have to wait a while before his change is implemented.
>
> The other thought I have would be to maintain a listing of what files
> are in what caches. However, it's obviously really crucial to keep
> overhead on the cache as low as possible, or the fix becomes the
> problem.
>
> Also, as Wari said, permalinks don't even benefit from caching much.
> The question is how to determine that something is a permalink - I
> think I'll just set the cache up to ignore filelists with only one
> entry.
>
> Thanks for trying the patch out and for the ideas on how to improve
> it; I think optimization is all kinds of fun, so I'll play with it
> some today.
>
> Peace
> Bill Mill
> bill.mill at gmail.com
>

-- 
whatever it is, you can find it at http://www.bluesock.org/~willg/
except Will--you can only see him in real life.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/