Re: Catalogs causing pain? (was Re: too many database conflicts when using ZODB with int ids and indexes)
Sanjay Rao <[email protected]>
| Newsgroups | gmane.comp.web.zope.zodb |
|---|---|
| Message-ID | <[email protected]> |
Does anybody aware of any issue related to massive database size increments when we use 64 bit int ids in indexes. In this case we use 64-bit versions of BTree, Buckets, Set etc. On Friday, 24 February 2017 18:58:51 UTC+5:30, Jim Fulton wrote: > > On Fri, Feb 24, 2017 at 4:58 AM, Sanjay Rao <[email protected] > <javascript:>> wrote: > >> Next problem in this case is that when I use catalog indexes then my >> database grows very fast. After packing database comes to acceptable size. >> When I inspect unpacked database I find large number of >> dead(un-referenced) objects of type "BTrees.LFBTree.LFSet" which covers 97% >> of database size. >> > > Yes, catalogs can lead a lot of database churn (high rates of growth > between packs). > > The churn can be reduced by batching index updates, using catalog queues > (e.g. zc.catalogqueue, although you may need one find one compatible with > whatever catalog implementation you're using). > > If catalogs are causing you pain, you should try Newt DB, > http://www.newtdb.org. > > Implementing catalogs well is very hard. This is why I'm excited about > Newt DB. > > Newt DB is compatible with ZODB. It just augments ZODB and RelStorage. If > you can use *current* releases of ZODB and RelStorage, you can use Newt > DB. > > Jim > > > >> >> >> On Thursday, 1 December 2016 11:26:35 UTC+5:30, Sanjay Rao wrote: >>> >>> Thanks a lot everyone for helping. >>> >>> On Friday, 18 November 2016 22:31:41 UTC+5:30, Jason Madden wrote: >>>> >>>> >>>> > On Nov 18, 2016, at 05:34, Jim Fulton <[email protected]> wrote: >>>> > >>>> >> If i use ZC.intid in place of zope.intId then how to get rid of >>>> these subscribers in zope.intid ? You guys must have done this. >>>> >> >>>> >> >>>> >> <subscriber handler=".removeIntIdSubscriber" /> >>>> >> <subscriber handler=".addIntIdSubscriber" /> >>>> >> >>>> >> both the handlers are assuming that KeyReference based int id being >>>> used. and call getId of ZC.intId by passing a KeyReference object. >>>> KeyReference object is a wrapper object containing original object as an >>>> attribute >>>> > >>>> > You have to replace them with subscribers that behave differently. >>>> >>>> While you *could* instead register a more specific IKeyReference >>>> adapter for your objects that just returns the object, I don't think that's >>>> a particularly good idea. The default KeyReferenceToPersistent has its >>>> uses. >>>> >>>> Here's how my group configures zc.intid: >>>> >>>> <!-- configure.zcml --> >>>> <!-- >>>> If we load zope.intid, we get subscribers for the Object events >>>> that ensure all ILocation objects are registered/unregistered when >>>> they are added/removed, plus another set of events when they >>>> get/lose intids. This second set of events is meant to update >>>> zope.catalog. A consequence of this is that ILocation objects must >>>> be adaptable to KeyReferences when they are ObjectAdded (for >>>> purposes of zope.intid, which we don't care about, but this also >>>> ensures that they have ZODB Connections, which is good). >>>> >>>> We cannot use these subscribers as-is due to the way the use >>>> IKeyReference >>>> and try to register that. However, our subscribers *do* make sure >>>> that >>>> the given objects can be adapted to IKeyReference because that's >>>> useful and >>>> may be required by catalogs or other subscribers. >>>> --> >>>> <exclude package="zope.intid" file="subscribers.zcml" /> >>>> <include package="zope.intid" /> >>>> >>>> <include package="zope.keyreference" /> >>>> >>>> <!-- >>>> zc.intid fires a different set of events when objects gain/lose >>>> intids. >>>> --> >>>> <include package="zc.intid" /> >>>> >>>> <!-- >>>> This file has a bug (uses 'interface' instead of >>>> 'implements'), so we replicate its effect manually. >>>> --> >>>> <!-- >>>> <include package="zc.intid" file="zope-intid.zcml" /> >>>> --> >>>> <class class="zc.intid.utility.IntIds"> >>>> <implements interface="zope.intid.interfaces.IIntIds"/> >>>> </class> >>>> >>>> <subscriber handler=".subscribers.intIdEventNotify" /> >>>> >>>> <subscriber handler=".subscribers.addIntIdSubscriber" /> >>>> <subscriber handler=".subscribers.removeIntIdSubscriber" /> >>>> >>>> <subscriber handler=".subscribers.nti_intIdEventNotify" /> >>>> >>>> >>>> I'll be fixing the bug in zc.intid's zope-intid.zcml. >>>> >>>> Here's what the subscribers.py looks like for us: >>>> >>>> def _utilities_and_key(ob): >>>> utilities = tuple(component.getAllUtilitiesRegisteredFor(IIntIds)) >>>> return utilities, IKeyReference(ob, None) if utilities else None # >>>> Don't even bother trying to adapt if no utilities >>>> >>>> @adapter(ILocation, IObjectAddedEvent) >>>> def addintIdSubscriber(ob, event): >>>> """ >>>> Registers the object in all unique id utilities and fires >>>> an event for the catalogs. Notice that each utility will >>>> fire :class:`zc.intid.interfaces.IIntIdAddedEvent`; this subscriber >>>> will then fire one single >>>> :class:`zope.intid.interfaces.IIntIdAddedEvent`, >>>> followed by one single >>>> :class:`nti.intid.interfaces.IIntIdAddedEvent`; this >>>> gives a guaranteed order such that :mod:`zope.catalog` and other >>>> Zope >>>> event listeners will have fired. >>>> """ >>>> utilities, key = _utilities_and_key(ob) >>>> if not utilities or key is None: >>>> return >>>> >>>> idmap = {} >>>> for utility in utilities: >>>> idmap[utility] = utility.register(ob) >>>> >>>> # Notify the catalogs that this object was added. >>>> notify(ZOPEIntIdAddedEvent(ob, event, idmap)) >>>> notify(NTIIntIdAddedEvent(ob, event, idmap)) >>>> >>>> @adapter(ILocation, IObjectRemovedEvent) >>>> def removeIntidSubscriber(ob, event): >>>> """ >>>> Removes the unique ids registered for the object in all the unique >>>> id utilities. >>>> >>>> Just before this happens (for the first time), an >>>> :class:`nti.intid.interfaces.IIntIdRemovedEvent` is fired, >>>> followed by an :class:`zope.intid.interfaces.IIntIdRemovedEvent`. >>>> Notice that this is fired before the id is actually removed from >>>> any utility, giving other subscribers time to do their cleanup. >>>> Before each utility removes its registration, it will fire >>>> :class:`zc.intid.interfaces.IIntIdRemovedEvent`. This gives a >>>> guaranteed order such that :mod:`zope.catalog` and other Zope >>>> event listeners will have fired. >>>> """ >>>> utilities, key = _utilities_and_key(ob) >>>> if not utilities or key is None: >>>> return >>>> >>>> # Notify the catalogs that this object is about to be removed, >>>> # if we actually find something to remove >>>> fired_event = False >>>> >>>> for utility in utilities: >>>> if not fired_event and utility.queryId(ob) is not None: >>>> fired_event = True >>>> notify(NTIIntIdRemovedEvent(ob, event)) >>>> notify(ZOPEIntIdRemovedEvent(ob, event)) >>>> try: >>>> utility.unregister(ob) >>>> except KeyError: >>>> pass >>>> >>>> @component.adapter(zope.intid.interfaces.IIntIdEvent) >>>> def intIdEventNotify(event): >>>> """ >>>> Event subscriber to dispatch IntIdEvent to interested adapters. >>>> """ >>>> handle(event.object, event) >>>> >>>> @component.adapter(nti.intid.interfaces.IIntIdEvent) >>>> def nti_intIdEventNotify(event): >>>> """ >>>> Event subscriber to dispatch IntIdEvent to interested adapters. >>>> """ >>>> handle(event.object, event) >>>> >>>> They're pretty simple, but maybe worth adding to zc.intid. >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "zodb" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Jim Fulton > http://jimfulton.info > -- You received this message because you are subscribed to the Google Groups "zodb" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.