Re: Patch for simple walk caching

On Dec 6, 2004, at 1:05 PM, will guaraldi wrote:

> On Mon, 6 Dec 2004, Ted Leung wrote:
>>
>> Are those environments actually configurations that we want to 
>> support?
>
> It's interesting you use "we".  I've been trying to minimize the 
> number of executive decisions I've been making because I'm not really 
> sure who "we" is.

Well, I use we lightly, since I haven't done much work to the core in 
quite some time.

>
> If the decision were up to me, I'd support Python by CGI and 
> mod_python and tell everyone else they should find another blog 
> system.  To be honest, I don't think we test with any other systems 
> anyhow.

 From where I sit, this would be fine.   I'm personally interested in 
seeing if Pyblosxom could run inside Twisted, but I don't see that as a 
goal for the project as a whole

>
> Last year sometime, I did a minor overhaul of the system which made it 
> more complex but which removed global caching.  So it's definitely 
> been a requirement to allow for multiple threads.
>
> Also, a while back you mentioned intertwining PyBlosxom with 
> Chandler--I'm not sure what that would entail.

It probably means doing some work on the storage layers, and getting it 
to run as a twisted servlet.  Again, this is a side project, 
independent of the pyblosxom core.

>
>
>> Ok, I'll keep this local for now.  But I think that we need to look a 
>> a decent way of solving this problem.  If we have to have a thread 
>> safe cache for mod_python or whatever, that's fine by me.
>>
>> My blog is up to almost 1200 entries and I've been getting complaints 
>> about the slowness.  I turned on entryshelve caching which helped 
>> some, and this cache patch helped a lot.  In my blog the bottleneck 
>> is now tools.walk_internal, and I'm thinking about implementing a 2.3 
>> only version to see if os.walk is any better.
>
> As near as I can see it, PyBlosxom's big issues are:
>
> 1) total inability to scale due to poor architecture in the realm of
>    storage (and the huge scope of the project)

Yep.  While I like the filesystem model, it makes a lot of things very 
hard.  This is one of the reasons that I am looking hard at Wordpress.  
At the same time, pyblosxom is doing what I need, except for 
performance.  I've got a decent amount of effort invested in comments 
and metaweblog, so I'm reluctant to just walk away.

>
> 2) half-assed implementation of static rendering (for which I take full
>    responsibility, but I don't use it, so I just wrote what seemed to
>    make sense at the time)

fixing this might help too.

>
> 3) configuration/testing/setup is difficult for newbies
>
> 4) mediocre to poor documentation
>
>
> I really do want to address issue 1 in pyblosxom 1.2 and it'll involve 
> a refactoring of a large chunk of how pyblosxom deals with entries.  
> Bill Mill and Robert Wall have expressed interest in this area as 
> well.  The plan is to build something that allows for index caching 
> and metadata caching as well as the ability to store mtimes in the 
> file itself.  All of that is a total pain in the ass with the current 
> architecture.  More thoughts on that at some point in the future.
>
> I'd like to address issue 2, but I'm not qualified to do it so I'm 
> going to defer until someone steps up to the plate.  At that point, 
> I'll let them know everything I know about it (assuming I remember) 
> and they can go off and fix the issues they see with the current 
> implementation.  It's possible issue 2 is related to issue 1--I have 
> no idea.
>
> Issues 3 and 4 will get fixed as time goes on assuming we keep up some 
> development momentum.
>
> Anyhow, so that's the plan as far as I know it.  I'm thinking that I'm 
> going to "take-over" in the full sense of the term.  If that does 
> happen, I'm going to seriously reduce the scope of the project from 
> meeting everyone's needs to meeting a specific set of requirements 
> which are explicitly laid out on the web-site.
>
> I'm still waffling about whether I want to put the hours in and take 
> over, though.  So that's where I lie on that.

I can understand that -- its a lot of effort.  The WordPress people are 
disappointing me in terms of their development speed, and I'm not 
really that excited about learning PHP, so I'm less excited about 
moving over there, except as a last resort.  I like your thinking for 1 
and (possibly) 2, above.  Right now, I'm willing to invest some more 
time in trying to get pyblosxom to work for me, as opposed to figuring 
out how to migrate off to WordPress.

>
> Getting there...  albeit extremely slowly right now.
>
> /will
>
----
Ted Leung                          Blog: <http://www.sauria.com/blog>
PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/