Re: Schema-by-interface options

Hi Kapil,

> well, its not in production yet, it is being heavily utilized for a 
> system to be in production by plone conference. i just wanted to 
> introduce contentflavors, as it spoke to the conversation here on 
> thread, and is of general utility. it handles all the use cases 
> presented in this thread, the discussion to me seems to have shifted to 
> implementation details.

Yep.

 > after thinking about the global scenarios, i'm
> not really interested in making contentflavors deal with that scenario 
> via class monkey patching

Me neither.

>, i've outlined an alternative global 
> implementation using contentflavors and some brainstorming notes on 
> global scenarios below.

Cool.

> if your going to put schema delegation directly in AT, please do make 
> that delegation, 100% policy free and make adapters implement policy.

+1

>> Could you elaborate a bit more on how the "global" flavour thing may
>> look? For example, imagine a scenario where you have a "global"
>> 'tagging' flavour that adds a 'tag' field to any object marked with
>> ITaggable.
>>
>> Let's say that you then *also* want to add a specific 'article'
>> flavour or whatever.
>>
>> How would this look?
> 
> you would setup a local event adapter within a site folder for content 
> created, and apply flavors to the content on whatever basis ( in this 
> case implementing ITaggable).

That feels a bit strange, but I suppose it makes sense. The flavour's 
schema is still on-disk, just keyed by name (right?) so it's not like 
you're stuffing the schema of each instance in an event. I think I like 
it. :)

> i've been doing some more thinking about the global scenario, with 
> global implementations. part of the difficulty of using a global 
> implemetation are the opposing use cases, ie. applying global behavior 
> local to a site.
> 
> global behavior implementation based on class manipulation is a 
> radically different notion and is inherently global in nature. 
> additionally archetypes content have way too many responsibilities as 
> regards schema, which makes such monkey patching both fragile and 
> complex to support class weaving imho.

I'm not sure it has to be done with monkey-patching as such. My solution 
basically looked like this:

  BaseObject's Schema() does: schema = ISchemaComposer(self)()

  the SchemaComposer's __call__ method would look up adapters to 
ISchemaProvider - which could come from local or global site managers

  each ISchemaProvider would provide a part of a Schema object (still 
not attached to a class or instance)

  the final schema would be cached (keyed on __class__ in the current 
implementation, though I realise that may be too restrictive) and then 
returned

  subsequent lookups would use the cache until invalidated via an event

  you need to run class-gen to build out methods (on the class, in my 
case) for AT to work - the schema composer took care of this.

I wouldn't call this monkey patching. All it does is defer the decision 
of what Schema() should return until run-time (i.e. the first time it's 
called/after the cache is invalidated).

However, I'm agreeing with you that the subscriber-based version will 
work with the kind of decoupling/indirection that ContentFlavors favours 
(sorry), and that the above solution makes per-instance much more 
difficult unnecessarily.

> the sort of implicit global monkeying of arbitrary classes seems much 
> more error prone and which in imho is the opposite of what makes a good 
> framework.
> 
> also, notions of expected ordering application of schemas are also 
> harder on a purely global basis, as they require awareness of 
> configuration space ordering.

Yes. My solution had an 'order' attribute that specifies an int, and 
it's sorted by this. klass.schema was 0, and other could use numbers as 
they wished. But I never liked this. Note that this is how menus work in 
Zope 3...

> as before, i think the easiest ability to support global AT behavior 
> with current local flavors implementation is via an event listener.

The only negative I can think of here, really, is that if you uninstall 
the thing that provided the "global" (site-local) schema, you need to 
find all the content objects that have it attached. You can build some 
safety into that, of course - if the flavour can't be found, don't bomb. 
But it still seems a bit more tricky.

> both local flavors and application of global behavior are much simpler
> when you assume z3 schema weaving, formlib even makes it fun ;-), and 
> you basically get the behavior both explicitly ( via formlib view class 
> form fields definition ) with delegation to adapters for storage.
> 
> ie. in the case of z3 schemas/behaviors, you could already do this 
> without any additional infrastructure, using plain old adapters, of 
> which flavors are really just sugar for working with AT and some 
> additional behavior.
> 
> class IPloneTagMarker( Interface ): pass
> 
> class MyTagger( object ):
>     def __init__(self, context):
>         self.context = context
> 
>     def tag(self, tags):
>         # or any other alternative implementation
>         self.context.Schema()['mutator'].getMutator( self.context )( tags )

Assume you mean ['tags'] here?

>     def getTags(self):
>         return self.context.Schema()['tags'].getAccessor( self.context )()
> 
> <adapter
>   class="ITaggable"
>   for="IPloneTagMarker"
>   handler=".MyTagger"
>   />

btw, you can use implements() and adapter() in 2.9, and it's much nicer 
to read. :) The above is also not right, I assume you meant

<adapter
   factory=".tagger.MyTagger"
   provides=".interfaces.ITaggable'
   for=".interfaces.IPloneTagMarker'
/>

Anyway.

> <five:implements=".myContent.IPloneTagMarker"/>
 >
> to support the global case w/ a global implementation  cleanly (without 
> side effects) you have to delegate to some local notion but, making 
> behavior local while not affecting side effect global changes is hard, 
> you get to use a local adapter registry sure.. but you can't avoid the 
> side effects of global modification when doing class modification. to 
> get around global modifications you'd need, a new primitive is needed to 
> tie local configuration to global.

Yes. In my solution above, that'd mean that the schema cache would have 
to be in a local utility. I think it'd work otherwise, though - since 
when the cache was invalidated, all the adapters it would find would be 
local ones (and possibly some global ones).

> one solution, you could have a class provides which queried for a 
> provider utility, though it also creates dependencies on traversal for 
> site discovery. though testing frameworks that utilize would be 
> straightforward via implementations of the applied interfaces, the 
> runtime integration does bring up the importance of functional testing.
> 
> building on that primitive a marker utility for classes to support the 
> runtime persistent global case, ie. enabled via ui, in which case your 
> primary concern is persistability of declarations in a utility 
> implementation.
> 
> i took a look at it, its seems possible, though it might not be, there a 
> couple of invariants that need to be maintianed by interface 
> specifications and its fragile to constructs like classOnlyImplements 
> which replace class implements specifications. here's a possibly broken 
> prototype
> 
> from zope.declarations import Implements
> 
> class LocalClassImplements( Implements ):
>     """ we don't setup subscription of local bases
>         any setBases api should be called without local context.
>     """
> 
>     def setBases(self, bases):
>         # Register ourselves as a dependent of our old bases
>         for b in self.getBases()
>             b.unsubscribe(self)
> 
>         # Register ourselves as a dependent of our bases
>         self.__dict__['__local_bases__'] = bases
>         for b in bases:
>             b.subscribe(self)
> 
>         self.changed()
> 
>     def getBases( self ):
>         b = self.__dict__['__local_bases__']
>         marker = component.queryUtility( IClassMarkerUtility )
>         if marker is None:
>             return b
>         return
> 
>     __bases__ = property( getBases, setBases )

Now you've lost me - not sure what you're trying to achieve/demonstrate 
here...

>>> aye, variableschemasupport had some borked and expensive hash method 
>>> which
>>> always failed afaicr, volatile caches are compariatively fast and well
>>> known.
>>
>> True, but it also depends on the lifetime of that cache. The hash was
>> expensive (and did the opposite of what it was trying to do), but
>> running ClassGen when it's not necessary is expensive too.
>>
> 
> i added an additional cache layer, such that, class gen semantics (not 
> implementation) are never run when its not nescessary. the 
> implementation has some docs and details (trunk/schema.py)

Cool, great - will look.

>>> if the object is in memory its _v_schema is only going to be
>>> composed once till the object is either ghosted or mutated,
>>
>> Yeah - which may be reasonably often still, especially if it's 
>> per-instance.
>>
> 
> the new cache scheme uses a two layer cache, global by class, 
> composition combination and instance, it saves memory and never needs 
> extraneous composition calls.

I trust you to get things like caches right. :)

>> Also, I assume you generate methods on the instance and not on the 
>> __class__?
> 
> of course, mucking with a global like class multiply at runtime is way 
> too error prone and a horrible framework semantic.

Yes. As I said, the per-instance use case was never part of my original 
thinking. Agree with you that it's undesirable. I'd rather not have to 
generate methods at all, of course, but then we'd have to fix all sorts 
of things in AT (I think). I think it's an anti-pattern for things that 
use Flavors to rely on getMyField in any case. They should use 
Schema().getField('myField'). And of course, that should all be hidden 
behind adapters.

>>> incidentally,
>>> contentflavors already does instance cache invalidation via event, 
>>> though
>>> moving to the cache keys i proposed previously, you never need to
>>> invalidate a composed schema, just signal accessor/mutator gen on 
>>> instance.
>>
>> Why is it that you'd never need to invalidate?
> 
> i treat schemas as immutable post startup. as you mentioned before, 
> persistently storing schemas leads to all kinds of issues, once you get 
> away from that notion, you can compose schemas and save composition, 
> because for its definition combination its always valid. you can utilize 
> have mutable schemas if mutation triggers events that can be utilized 
> for cache invalidation.

Restarts are fine, I think. I've never been a fan of TTW schema editing. :)

>> An event that's fired on-schema-lookup may be another option .
> 
> our implementation viewpoints are very different, i don't want to modify 
> global variables like classes at all, it breaks any local semantic be 
> that instance or site.

They're converging - I think you're absolutely right.

>> My three main concerns then are:

(this was the summary bit)

>>  - How well does it handle the global per-class (possibly "global"
>> only to a Plone site, via a local adapter) use case (i.e. the ITagger
>> scenario above).
> 
> it does so by utilizing events, the global usage doesn't need to entail 
> global implementation, esp. since you've stated as a use case local 
> customization / installation to a plone site.

+1

>>  - Are we in the clear about performance of composing schemas and
>> possibly re-generating methods. Again, an explicitly invalidated cache
>> of (class, flavours-tuple) may make sense.
> 
> that cache works nicely w/ content flavors, and doing an implementation 
> of regenerating methods on the class is broken if you want local 
> behavior or customization without side affects.

+1

>>  - What is the best way of doing the delegation in AT to avoid the
>> mixin, but still keep ContentFlavours as an optional dependency for
>> third party products? In particular, if ContentFlavours ends up being
>> dependent on an overrides.zcml to override the Schem adapter on
>> IBaseObject, then it will be incompatible with any other generalised
>> solution that does the same. Not necessarily a huge problem, but it
>> would be nice to avoid it.
> 
> don't depend on additional policy beyond delegation would be best. the 
> incompatiblity is always an issue with multiple framework 
> implementations, esp. when their of the mythical variety.

hehe :)

So, specifically, this is what I'd like to do:

  1. Merge current trunk into 1.5 whitmo-schema-by-iface so that it's 
1.5 compatible. This contains more fun with interfaces and so on that 
helps all of this.

  2. This uses schema = ISchema(self). The adapter factory here is a 
class that does things that scares me - like 'self = 
ImplicitAquisitonWrapper(schema)' in __init__(). Yipes. :) We can do it 
better (imho) by using a function as a custom adapter factory - this is 
trivial.

  3. I also think the aq-wrapping in Schema() should stay in the 
Schema() method, so that it does:

   def Schema(self):
       schema = ISchema(self)
       return ImplicitAcquisitonWrapper(schema)

That way, at least we're not requiring all ISchema adapters to remember 
to aq-wrap (which may even lead to some funny semantics if the ISchema 
adapter is used elsewhere).

  4. We slap a big, scary note on the ISchema interface that people need 
to worry about generating methods, and caches.

  5. I update VariableSchemaSupport to get the cleaner VarClassGen 
that's on my branch atm. I don't propose we really use this, but in the 
case where people *do* want their own ISchema adapter for a specific 
class, they will be able to use it. Lots of caveats need to go on it, of 
course, but since the code is already there I see no problem putting it 
in for those who do want it.

  6. We let ContentFlavors mark BaseObject with IFlavorAware or 
something like that, and register our own ISchema adapter that has 
ContentFlavors semantics. That way, if someone wants to play with the 
marker interfaces to get different semantics again, they can, but by 
default, ContentFlavors is applicable to everything that wants it.

Just to be sure:

  - Do you see any problems (performance or otherwise) with having a 
ContentFlavors version of Schema() (via the ISchema adapter) in the 
cases where it's not actually used?

Oh - and can we eventually move this to AT svn?

Thanks again :)

Martin

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV