Re: Catalogs causing pain? (was Re: too many database conflicts when using ZODB with int ids and indexes)

On Fri, Feb 24, 2017 at 8:32 AM, Jason Madden <[email protected]>
wrote:

> If you can use RelStorage and don't require object history, using
> RelStorage in its history-free mode can also substantially reduce the churn
> without requiring any other application changes. You're still using
> catalogs so you still have their inherent downsides, but that will reduce
> the growth.
>

Yup, except in his application, his catalogs are also throwing off a lot of
garbage. (I meant to ask him how he figured that out. :))  So he'll still
have a lot of churn, although less with a history-free implementation.

+1 for using the history-free configuration. If you don't need time travel,
it's much faster. It's the default configuration when you use Newt.

But New DB makes a lot of this go away. You'll end up with a lot fewer
objects in your database, faster writes and wildly fewer conflicts.

Jim



>
> Jason
>
> > On Feb 24, 2017, at 07:28, Jim Fulton <[email protected]> wrote:
> >
> > On Fri, Feb 24, 2017 at 4:58 AM, Sanjay Rao <[email protected]>
> wrote:
> > Next problem in this case is that when I use catalog indexes then my
> database grows very fast. After packing database comes to acceptable size.
> > When I inspect unpacked database I find large number of
> dead(un-referenced) objects of type "BTrees.LFBTree.LFSet" which covers 97%
> of database size.
> >
> > Yes, catalogs can lead a lot of database churn (high rates of growth
> between packs).
> >
> > The churn can be reduced by batching index updates, using catalog queues
> (e.g. zc.catalogqueue, although you may need one find one compatible with
> whatever catalog implementation you're using).
> >
> > If catalogs are causing you pain, you should try Newt DB,
> http://www.newtdb.org.
> >
> > Implementing catalogs well is very hard. This is why I'm excited about
> Newt DB.
> >
> > Newt DB is compatible with ZODB. It just augments ZODB and RelStorage.
> If you can use current releases of ZODB and RelStorage, you can use Newt DB.
> >
> > Jim
> >
> >
> >
> >
> > On Thursday, 1 December 2016 11:26:35 UTC+5:30, Sanjay Rao wrote:
> > Thanks a lot everyone for helping.
> >
> > On Friday, 18 November 2016 22:31:41 UTC+5:30, Jason Madden wrote:
> >
> > > On Nov 18, 2016, at 05:34, Jim Fulton <[email protected]> wrote:
> > >
> > >> If i use ZC.intid in place of zope.intId then how to get rid of these
> subscribers in zope.intid ? You guys must have done this.
> > >>
> > >>
> > >>   <subscriber handler=".removeIntIdSubscriber" />
> > >>   <subscriber handler=".addIntIdSubscriber" />
> > >>
> > >> both the handlers are assuming that KeyReference based int id being
> used. and call getId of ZC.intId by passing a KeyReference object.
> KeyReference object is a wrapper object containing original object as an
> attribute
> > >
> > > You have to replace them with subscribers that behave differently.
> >
> > While you *could* instead register a more specific IKeyReference adapter
> for your objects that just returns the object, I don't think that's a
> particularly good idea. The default KeyReferenceToPersistent has its uses.
> >
> > Here's how my group configures zc.intid:
> >
> >     <!-- configure.zcml -->
> >     <!--
> >     If we load zope.intid, we get subscribers for the Object events
> >     that ensure all ILocation objects are registered/unregistered when
> >     they are added/removed, plus another set of events when they
> >     get/lose intids. This second set of events is meant to update
> >     zope.catalog. A consequence of this is that ILocation objects must
> >     be adaptable to KeyReferences when they are ObjectAdded (for
> >     purposes of zope.intid, which we don't care about, but this also
> >     ensures that they have ZODB Connections, which is good).
> >
> >     We cannot use these subscribers as-is due to the way the use
> IKeyReference
> >     and try to register that. However, our subscribers *do* make sure
> that
> >     the given objects can be adapted to IKeyReference because that's
> useful and
> >     may be required by catalogs or other subscribers.
> >     -->
> >     <exclude package="zope.intid" file="subscribers.zcml" />
> >     <include package="zope.intid" />
> >
> >     <include package="zope.keyreference" />
> >
> >     <!--
> >     zc.intid fires a different set of events when objects gain/lose
> >     intids.
> >     -->
> >     <include package="zc.intid" />
> >
> >     <!--
> >          This file has a bug (uses 'interface' instead of
> >          'implements'), so we replicate its effect manually.
> >     -->
> >     <!--
> >     <include package="zc.intid" file="zope-intid.zcml" />
> >     -->
> >     <class class="zc.intid.utility.IntIds">
> >         <implements interface="zope.intid.interfaces.IIntIds"/>
> >     </class>
> >
> >     <subscriber handler=".subscribers.intIdEventNotify" />
> >
> >     <subscriber handler=".subscribers.addIntIdSubscriber" />
> >     <subscriber handler=".subscribers.removeIntIdSubscriber" />
> >
> >     <subscriber handler=".subscribers.nti_intIdEventNotify" />
> >
> >
> > I'll be fixing the bug in zc.intid's zope-intid.zcml.
> >
> > Here's what the subscribers.py looks like for us:
> >
> > def _utilities_and_key(ob):
> >     utilities = tuple(component.getAllUtilitiesRegisteredFor(IIntIds))
> >     return utilities, IKeyReference(ob, None) if utilities else None  #
> Don't even bother trying to adapt if no utilities
> >
> > @adapter(ILocation, IObjectAddedEvent)
> > def addintIdSubscriber(ob, event):
> >     """
> >     Registers the object in all unique id utilities and fires
> >     an event for the catalogs. Notice that each utility will
> >     fire :class:`zc.intid.interfaces.IIntIdAddedEvent`; this subscriber
> >     will then fire one single :class:`zope.intid.interfaces.
> IIntIdAddedEvent`,
> >     followed by one single :class:`nti.intid.interfaces.IIntIdAddedEvent`;
> this
> >     gives a guaranteed order such that :mod:`zope.catalog` and other Zope
> >     event listeners will have fired.
> >     """
> >     utilities, key = _utilities_and_key(ob)
> >     if not utilities or key is None:
> >         return
> >
> >     idmap = {}
> >     for utility in utilities:
> >         idmap[utility] = utility.register(ob)
> >
> >     # Notify the catalogs that this object was added.
> >     notify(ZOPEIntIdAddedEvent(ob, event, idmap))
> >     notify(NTIIntIdAddedEvent(ob, event, idmap))
> >
> > @adapter(ILocation, IObjectRemovedEvent)
> > def removeIntidSubscriber(ob, event):
> >     """
> >     Removes the unique ids registered for the object in all the unique
> >     id utilities.
> >
> >     Just before this happens (for the first time), an
> >     :class:`nti.intid.interfaces.IIntIdRemovedEvent` is fired,
> >     followed by an :class:`zope.intid.interfaces.IIntIdRemovedEvent`.
> >     Notice that this is fired before the id is actually removed from
> >     any utility, giving other subscribers time to do their cleanup.
> >     Before each utility removes its registration, it will fire
> >     :class:`zc.intid.interfaces.IIntIdRemovedEvent`. This gives a
> >     guaranteed order such that :mod:`zope.catalog` and other Zope
> >     event listeners will have fired.
> >     """
> >     utilities, key = _utilities_and_key(ob)
> >     if not utilities or key is None:
> >         return
> >
> >     # Notify the catalogs that this object is about to be removed,
> >     # if we actually find something to remove
> >     fired_event = False
> >
> >     for utility in utilities:
> >         if not fired_event and utility.queryId(ob) is not None:
> >             fired_event = True
> >             notify(NTIIntIdRemovedEvent(ob, event))
> >             notify(ZOPEIntIdRemovedEvent(ob, event))
> >         try:
> >             utility.unregister(ob)
> >         except KeyError:
> >             pass
> >
> > @component.adapter(zope.intid.interfaces.IIntIdEvent)
> > def intIdEventNotify(event):
> >     """
> >     Event subscriber to dispatch IntIdEvent to interested adapters.
> >     """
> >     handle(event.object, event)
> >
> > @component.adapter(nti.intid.interfaces.IIntIdEvent)
> > def nti_intIdEventNotify(event):
> >     """
> >     Event subscriber to dispatch IntIdEvent to interested adapters.
> >     """
> >     handle(event.object, event)
> >
> > They're pretty simple, but maybe worth adding to zc.intid.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "zodb" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected].
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
> >
> > --
> > Jim Fulton
> > http://jimfulton.info
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "zodb" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected].
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "zodb" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.