Re: ZODB with Lazy evaluation / Caching / Data Flow programming

On Wed, Apr 25, 2018 at 9:23 PM, Byron Wong <[email protected]> wrote:

> Dear all (and especially Jim),
>
> ZODB is an amazing tool.
>

Thanks!


> I used to work in an investment bank, and instead of SQL, their primary
> database is an in-house built Object database (they have their own globally
> distributed ZEO-sync-like service). Since then, I have fallen in love with
> the simplicity and flexibility of object databases. Now that I no longer
> work there, and I have always wanted to find an open-source alternative...
> gladly I bump into ZODB.
>
> The object database was only one part of it; they have tightly integrated
> concept of "Data Flow programming" into the Object Database implementation.
>
> For e.g.
>
> class MyClass(object):
>
>     def __init__(self):
>
>         self.a = varnode(default=10)
>
>         self.b = varnode(default=20)
>
>     @evalnode
>
>     def function_of_a_and_b(self):
>         return self.a() * self.b()
>
>     @evalnode
>
>     def example_node(self):
>         return self.function_of_a_and_b() * 5
>
>
>
> In the above, function_of_a_and_b() and example_node() are only evaluated
> lazily the first time it's being used. Sub-sequently, as long as the
> underlying dependency a() and b() are not modified or changed, when we
> invoke function_of_a_and_b() again it will return the results cached in
> memory. The concept ties very well with ZODB, of which you can think of the
> varnode() are usually stored in an object database and will only be invoked
> lazily if the "dependency evaluation graph" triggers that.
>
> As Jim Fulton mentioned in his Zodb-Tour presentation, "
> http://jimfulton.info/presentations/zodb-tour/#/step-1"
>
> *There are only two hard things in Computer Science: cache invalidation
> and naming things.* -- Phil Karlton
>
> ZODB object cache:
>
>    - In memory object graph
>    - Transactionally-consistent partial database replica.
>
> I am quite convinced that ZODB has a pretty good dependency tracking
> mechanism to ensure when we access an attribute, it's always up to date.
>

This isn't really dependency tracking, at least not in the sense of cached
computations.

The only "dependency" tracking is that provided by Python object
references.  When you access an object that hasn't been loaded from the
database, it's loaded automatically.

Cache invalidation isn't dependency based.  It's identity based. When an
object is modified, then invalidations for that object are sent to all
clients.



>
> Then when I looked into some of the Python open-source data flow paradigm,
> I found out that they are explicitly defining their dependency relationship
> and properly handle cache invalidation in memory.
> Two of such example projects are: PythonFlow (recently open sourced by
> Spotify) and MDF (open sourced by a number of folks who used to work in the
> Man group hedge funds)
> https://github.com/spotify/pythonflow
> https://github.com/manahl/mdf
>
So finally my question, is it possible to leverage these already open
> source technology to create an object-database paradigm... of which data
> are stored in ZODB, and when the objects are invoked, it's being handled
> in-memory with smart invalidation logic as in Data Flow programming.
>

I don't think so.  ZODB cache invalidation is largely a solved problem.

It's unclear how dataflow plays in here.  ZODB invalidation is
transactional.  If you modify object state, nothing is invalidated
unless/until data are committed.

Flipping this around though, I can see a dataflow-based application being
interested in when objects change in a distributed way, which might trigger
other (application-level) computations.  There are some experimental
mechanisms that could play into this. I've used these, for example, to
provide "real-time" UIs where UIs update automatically when data are
modified, even if modified on a remote client.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.