Re: VariableSchemaSupport unnecessary slow

On Wed, 2006-06-14 at 10:55 +0200, Anton Stonor wrote:
> Roché Compaan wrote:
> > Hi there
> > 
> > Yesterday some of my colleagues and I ran profiler while re-indexing
> > content types that subclass VariableSchemaSupport. We discovered that
> > the hash created when caching the schema is making things very very
> > slow. Here is the relevant code:
> > 
> >         # create a hash value out of the schema
> >         hash=sha.new(str([f.__dict__ for f in s.values()]) +
> >                      str(self.__class__)).hexdigest()
> > 
> > I understand that the above code ensures that we always have a fresh
> > schema and that nobody else has to do anything in their code, but the
> > price on has to pay in performance is simply to high. I would recommend
> > that we use a very simple mechanism where the schema is cached upon
> > access and invalidated when it changes.
> 
> And VariableSchemaSupport caching seems to be broken anyway.
> 
> Using Plonemall - that subclasses VariableSchemaSupport - we never got a 
> cache hit with VariableSchemaSupport. Instead it stored what should be 
> the same fields multiple times in the cache (the module level 
> schemadict) and introduced a severe memory leak. The ref counts and 
> memory usage just grew and grew.
> 
> Removing the cache fixed it for us.

We definitely need to cache the schema though - calling "generateClass"
each time an attribute is accessed, is a sure performance killer for
content types with lots of fields.

-- 
Roché Compaan
Upfront Systems                 http://www.upfrontsystems.co.za