Re: Applause for ZODB! And a question: How do you get pending change info from a transaction object?

On Sun, Apr 29, 2018 at 8:44 AM, David Pesta <[email protected]> wrote:

> I have not spent much time looking at Newt DB beyond a quick glance.
>
> Since what I am starting is a completely new project, I want to use the
> latest and greatest approach and technology choices for this (ZODB)
> paradigm to ensure better compatibility with the future. It looks like Newt
> DB with the RelStorage Postgres backend embodies your latest vision and
> thrust going forward, am I correct?
>

Not really. A valuable feature of ZODB is it's pluggable storage
architecture, which provides more flexibility to meet a variety of needs,
than would a less flexible implementation.

The two most popular back ends for ZODB are ZEO and RelStorage.

RelStorage advantages include:

   - A bit faster, if you use the non-history-preserving option.  ZEO 5 has
   narrowed the gap a bit for Python 3.  The byteserver project,
   https://github.com/jimfulton/byteserver, will likely be faster, if I
   ever finish it.  I would love to see "zodbshootout" (a simple ZODB server
   benchmark) numbers for really large RDBMS deployments to get a better idea
   how far RelStorage can scale.

   Interestingly, at a Plone Conference a couple of years ago, no one in
   the audience of a talk I gave cared about server performance.

   - You can let someone else run your database server.  This is hugely
   valuable to me. I like that I can shove off responsibility for running
   stateful services to someone like Amazon or Google (or whoever knows what
   they're doing :-]).

   - Because invalidations are pulled rather than pushed, I suspect you can
   scale to more clients.

ZEO advantages include:

   - A little easier to set up, because you don't need to run an RDBMS
   database server, although running a single RDBMS server with a default
   configuration usually isn't that hard.
   - Faster (I think) in history-preserving mode.  Keeping some history is
   very valuable for use cases like time travel and 0-copy staging with
   production data.
   - Is a little more flexible with regard to blob handling. For example,
   you can store blobs in S3 with a local-disk cache.

Newt DB is essentially an add on to RelStorage with a Postgres backend that
keeps a secondary and lossy representation of your data in JSON format for
indexing and access without ZODB.  While reasonably mature, I don't think
there has been a lot of uptake. I certainly haven't heard of people using
it.  Newt DB has two modes for creating the JSON representation of your
data: synchronous and asynchronous. In synchronous, writes are slowed down
due to the extra computation. In asynchronous mode, writes aren't slowed,
but the JSON representation lags a bit.

The newt.db.follow API I mentioned in an earlier email can be used
independently of newt.db.  It can be used as long as you're using
RelStorage with a Postgres backend.



> And with Postgres as the fundamental basis of storage, it would be more
> palatable to the industry in conformity to their "good practice" (sells
> better).
>

Yup, at least if they don't look under the hood and see that all of their
data are Python pickles. :) Newt mitigates this a bit, because you can get
at data as JSON.


> Thus, for a completely new project that does not need to worry about being
> bulletproof for enterprise production (bleeding edge is safe for me), am I
> correct to assume that your recommendation would be to jump head first into
> Newt DB instead of traditional ZODB?
>

I don't think it matters that much.  It's fairly easy to change your mind
later.

You should use Newt if:

   - You want to leverage Postgres' powerful index and search capabilities.

   Some caveats:
   - You're comfortable with SQL, because you'll need to be good at it to
      leverage it well.  I've also found myself supplementing it with stored
      procedures.
      - You're searches use single-object attributes. Postgres doesn't let
      you use data from multiple objects when indexing. Well, it let's you, but
      if you do (via stored procedures), your index will eventually be
      corrupted.  This is especially acute when dealing with graph
relationships
      (e.g. hierarchies).
      - You lose some testing benefits of ZODB. Most notably, you can use
      in-memory databases to test ZODB applications, but if some of your tests
      rely on Postgres-provided search, then you'll need to deal with
a database
      server, which complicates and slows testing a lot.

      - You want to make your data available to non-ZODB applications.

Note that without using Postgres or some other external index (e.g.
elasticsearch), you will need to use BTrees or Catalogs to support search.
This means that indexing data structures are stored in your database and
can easily account for the majority of database objects.  Depending very
much on your application, these indexing mechanism can provide superior
performance, thanks to ZODB's caching machinery, especially if your queries
depend on hierarchy.  OTOH, for single-object queries, that don't depend on
on graph relationships, an external index like Postgres will likely be much
faster, and can allow you to get by with smaller object caches.  (You can
also mix approaches.)

The overall application development interface/experience/workflow is pretty
> much the same, right?
>

Yes, modulo using Postgres for Search.


> (After setting up the environment and low level development details.) Is
> it somewhat accurate to say that Newt DB is the next generation of ZODB?
>

Nope. It's just another storage option that provides some opportunities
that may or may not be of interest to you.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.