Re: too many database conflicts when using ZODB with int ids and indexes

Next problem in this case is that when I use catalog indexes then my 
database grows very fast. After packing database comes to acceptable size. 
When I inspect unpacked database I find large number of dead(un-referenced) 
objects of type "BTrees.LFBTree.LFSet" which covers 97% of database size.


On Thursday, 1 December 2016 11:26:35 UTC+5:30, Sanjay Rao wrote:
>
> Thanks a lot everyone for helping.
>
> On Friday, 18 November 2016 22:31:41 UTC+5:30, Jason Madden wrote:
>>
>>
>> > On Nov 18, 2016, at 05:34, Jim Fulton <[email protected]> wrote: 
>> > 
>> >> If i use ZC.intid in place of zope.intId then how to get rid of these 
>> subscribers in zope.intid ? You guys must have done this. 
>> >> 
>> >> 
>> >>   <subscriber handler=".removeIntIdSubscriber" /> 
>> >>   <subscriber handler=".addIntIdSubscriber" /> 
>> >> 
>> >> both the handlers are assuming that KeyReference based int id being 
>> used. and call getId of ZC.intId by passing a KeyReference object. 
>> KeyReference object is a wrapper object containing original object as an 
>> attribute 
>> > 
>> > You have to replace them with subscribers that behave differently. 
>>
>> While you *could* instead register a more specific IKeyReference adapter 
>> for your objects that just returns the object, I don't think that's a 
>> particularly good idea. The default KeyReferenceToPersistent has its uses. 
>>
>> Here's how my group configures zc.intid: 
>>
>>     <!-- configure.zcml --> 
>>     <!-- 
>>     If we load zope.intid, we get subscribers for the Object events 
>>     that ensure all ILocation objects are registered/unregistered when 
>>     they are added/removed, plus another set of events when they 
>>     get/lose intids. This second set of events is meant to update 
>>     zope.catalog. A consequence of this is that ILocation objects must 
>>     be adaptable to KeyReferences when they are ObjectAdded (for 
>>     purposes of zope.intid, which we don't care about, but this also 
>>     ensures that they have ZODB Connections, which is good). 
>>
>>     We cannot use these subscribers as-is due to the way the use 
>> IKeyReference 
>>     and try to register that. However, our subscribers *do* make sure 
>> that 
>>     the given objects can be adapted to IKeyReference because that's 
>> useful and 
>>     may be required by catalogs or other subscribers. 
>>     --> 
>>     <exclude package="zope.intid" file="subscribers.zcml" /> 
>>     <include package="zope.intid" /> 
>>
>>     <include package="zope.keyreference" /> 
>>
>>     <!-- 
>>     zc.intid fires a different set of events when objects gain/lose 
>>     intids. 
>>     --> 
>>     <include package="zc.intid" /> 
>>
>>     <!-- 
>>          This file has a bug (uses 'interface' instead of 
>>          'implements'), so we replicate its effect manually. 
>>     --> 
>>     <!-- 
>>     <include package="zc.intid" file="zope-intid.zcml" /> 
>>     --> 
>>     <class class="zc.intid.utility.IntIds"> 
>>         <implements interface="zope.intid.interfaces.IIntIds"/> 
>>     </class> 
>>
>>     <subscriber handler=".subscribers.intIdEventNotify" /> 
>>
>>     <subscriber handler=".subscribers.addIntIdSubscriber" /> 
>>     <subscriber handler=".subscribers.removeIntIdSubscriber" /> 
>>
>>     <subscriber handler=".subscribers.nti_intIdEventNotify" /> 
>>
>>
>> I'll be fixing the bug in zc.intid's zope-intid.zcml. 
>>
>> Here's what the subscribers.py looks like for us: 
>>
>> def _utilities_and_key(ob): 
>>     utilities = tuple(component.getAllUtilitiesRegisteredFor(IIntIds)) 
>>     return utilities, IKeyReference(ob, None) if utilities else None  # 
>> Don't even bother trying to adapt if no utilities 
>>
>> @adapter(ILocation, IObjectAddedEvent) 
>> def addintIdSubscriber(ob, event): 
>>     """ 
>>     Registers the object in all unique id utilities and fires 
>>     an event for the catalogs. Notice that each utility will 
>>     fire :class:`zc.intid.interfaces.IIntIdAddedEvent`; this subscriber 
>>     will then fire one single 
>> :class:`zope.intid.interfaces.IIntIdAddedEvent`, 
>>     followed by one single 
>> :class:`nti.intid.interfaces.IIntIdAddedEvent`; this 
>>     gives a guaranteed order such that :mod:`zope.catalog` and other Zope 
>>     event listeners will have fired. 
>>     """ 
>>     utilities, key = _utilities_and_key(ob) 
>>     if not utilities or key is None: 
>>         return 
>>
>>     idmap = {} 
>>     for utility in utilities: 
>>         idmap[utility] = utility.register(ob) 
>>
>>     # Notify the catalogs that this object was added. 
>>     notify(ZOPEIntIdAddedEvent(ob, event, idmap)) 
>>     notify(NTIIntIdAddedEvent(ob, event, idmap)) 
>>
>> @adapter(ILocation, IObjectRemovedEvent) 
>> def removeIntidSubscriber(ob, event): 
>>     """ 
>>     Removes the unique ids registered for the object in all the unique 
>>     id utilities. 
>>
>>     Just before this happens (for the first time), an 
>>     :class:`nti.intid.interfaces.IIntIdRemovedEvent` is fired, 
>>     followed by an :class:`zope.intid.interfaces.IIntIdRemovedEvent`. 
>>     Notice that this is fired before the id is actually removed from 
>>     any utility, giving other subscribers time to do their cleanup. 
>>     Before each utility removes its registration, it will fire 
>>     :class:`zc.intid.interfaces.IIntIdRemovedEvent`. This gives a 
>>     guaranteed order such that :mod:`zope.catalog` and other Zope 
>>     event listeners will have fired. 
>>     """ 
>>     utilities, key = _utilities_and_key(ob) 
>>     if not utilities or key is None: 
>>         return 
>>
>>     # Notify the catalogs that this object is about to be removed, 
>>     # if we actually find something to remove 
>>     fired_event = False 
>>
>>     for utility in utilities: 
>>         if not fired_event and utility.queryId(ob) is not None: 
>>             fired_event = True 
>>             notify(NTIIntIdRemovedEvent(ob, event)) 
>>             notify(ZOPEIntIdRemovedEvent(ob, event)) 
>>         try: 
>>             utility.unregister(ob) 
>>         except KeyError: 
>>             pass 
>>
>> @component.adapter(zope.intid.interfaces.IIntIdEvent) 
>> def intIdEventNotify(event): 
>>     """ 
>>     Event subscriber to dispatch IntIdEvent to interested adapters. 
>>     """ 
>>     handle(event.object, event) 
>>
>> @component.adapter(nti.intid.interfaces.IIntIdEvent) 
>> def nti_intIdEventNotify(event): 
>>     """ 
>>     Event subscriber to dispatch IntIdEvent to interested adapters. 
>>     """ 
>>     handle(event.object, event) 
>>
>> They're pretty simple, but maybe worth adding to zc.intid. 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.