Re: Catalogs causing pain? (was Re: too many database conflicts when using ZODB with int ids and indexes)

On Fri, Feb 24, 2017 at 8:59 AM, Jason Madden <[email protected]>
wrote:

>
> > On Feb 24, 2017, at 07:47, Jim Fulton <[email protected]> wrote:
> >
> > Yup, except in his application, his catalogs are also throwing off a lot
> of garbage. (I meant to ask him how he figured that out. :))  So he'll
> still have a lot of churn, although less with a history-free implementation.
>
> Yeah, I guess it wasn't clear to me whether the "dead" objects were
> actually garbage or simply the old states of reachable objects.
>
> If it was the old state of reachable objects, going history free will cut
> down on the number of object states in the database by a factor related to
> the rate at which buckets split.
>
> > +1 for using the history-free configuration. If you don't need time
> travel, it's much faster. It's the default configuration when you use Newt.
> >
> > But New DB makes a lot of this go away. You'll end up with a lot fewer
> objects in your database, faster writes and wildly fewer conflicts.
>
> It sure can (I like Newt DB!), but the caveat is that it doesn't make it
> go away magically on its own. You have to change your code to not use
> catalogs and query in the Newt way. Depending on how centralized
> (abstracted) your cataloging and querying are, this can be trivial or it
> can take some effort, especially if you have third-party dependencies that
> use catalogs.
>
>
> Not trying to rain on anybody's parade, just being
> "explicit-is-better-than-implicit",
>

np. In fact, this is covered in the fine print:

http://www.newtdb.org/en/latest/fine-print.html#learn-about-indexing-and-querying-postgresql

Instead of having to know about catalogs, you have to know about
PostgreSQL.  Sometimes, making searches go fast can take a lot of digging.
But at least there are a lot of resources available to help you.

As far as the existing catalog abstraction, it should be possible to
provide some sort of limited implementation on top of Newt DB, Newt DB can
already setup full-text search indexes for you.  It could be taught to set
up other indexes for you as well, at least as long as data extraction
doesn't require OO-style dispatch.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.