Re: High-level NewtDB questions

On Fri, Aug 24, 2018 at 12:05 AM, Sean Upton <[email protected]> wrote:

> I have a few conceptual questions about NewtDB, so I figured I would throw
> these out here:
>

Cool. Note that there's also a Newt DB mailing list (
https://groups.google.com/forum/#!forum/newtdb), although I suspect I
should retire it.


>
> 1. Is it intended that applications integrating NewtDB might add custom
> handlers to newt.db.jsonpickle.special_classes — I believe it is possible?
>

It is possible, but it's probably not flexible enough for general use.
It's intended as a quick way to handle common special cases.

The *preferred and documented way* to provide custom transformations is via
the custom transform API. :-)

http://www.newtdb.org/en/latest/topics/data-transformation.html




> 2. If so,


no


> would it be possible to serialize a objects of blacklisted type(s) to None
> or some sentinel value instead of storing an actual serialization in the
> JSON?


Yes, see http://www.newtdb.org/en/latest/topics/data-transformation.html



>   Or is it better to just monkey-patch newt.db.jsonpickle.Jsonifier.skip_class
> to this end?
>

At your own risk, but I think
http://www.newtdb.org/en/latest/topics/data-transformation.html
is better.


>
> 3. Would it be folly to think if #1 and (either path in) #2 are
> reasonable, I could store a Zope2 app in NewtDB, blacklisting any noisy
> ZCatalog objects that are not themselves in the BTrees package?
>

See http://www.newtdb.org/en/latest/topics/data-transformation.html :)


>
> 4. How does one make a query result from connection.where sensible in a
> situation where I want to get that object's original containment context
> (placeful parent object, path, or other such thing)?  Is this a
> non-starter, or is this sensible?
>

It's tricky to make graph-based queries.

   1. You can construct pretty complex indexes in Postgres by indexing the
   result of a stored procedure. A non-obvious limitation (to me, based on the
   PG documentation) if this is that the stored procedure can't use data from
   multiple rows. There's no error when you do this, but, if you do,
   eventually (quickly) the index will stop functioning properly and Postgres
   will report it's corrupted.
   2. You can do recursive queries, which themselves can leverage indexes,
   and are pretty fast, because they work on the set level, but aren't as fast
   as a direct index might be.
   3. You can index paths, but the paths have to be computed by the
   application (because 1).

I worked on a largish project where Newt was a big improvement for most
queries, but where the catalog was still a little faster for queries that
had a large graph component, but this was also an application where the
working set fit in ZODB's object cache.  (There might still have been a
benefit to converting this application to use Newt for all of it's indexes,
but it was a legacy application with a relatively short time horizon and
the development wasn't warranted.)

Part of the fine print in Newt is that to leverage it, you'll end up
needing Postgresql mastery:
http://www.newtdb.org/en/latest/fine-print.html#learn-about-indexing-and-querying-postgresql

For new projects, this means learning Postgresql esoteria rather than
Catalog esoteria. :)

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.