ZODB with Lazy evaluation / Caching / Data Flow programming

Dear all (and especially Jim),

ZODB is an amazing tool. I used to work in an investment bank, and instead 
of SQL, their primary database is an in-house built Object database (they 
have their own globally distributed ZEO-sync-like service). Since then, I 
have fallen in love with the simplicity and flexibility of object 
databases. Now that I no longer work there, and I have always wanted to 
find an open-source alternative... gladly I bump into ZODB.

The object database was only one part of it; they have tightly integrated 
concept of "Data Flow programming" into the Object Database implementation. 

For e.g.

class MyClass(object):

    def __init__(self):

        self.a = varnode(default=10)

        self.b = varnode(default=20)

    @evalnode

    def function_of_a_and_b(self):
        return self.a() * self.b()

    @evalnode

    def example_node(self):
        return self.function_of_a_and_b() * 5



In the above, function_of_a_and_b() and example_node() are only evaluated 
lazily the first time it's being used. Sub-sequently, as long as the 
underlying dependency a() and b() are not modified or changed, when we 
invoke function_of_a_and_b() again it will return the results cached in 
memory. The concept ties very well with ZODB, of which you can think of the 
varnode() are usually stored in an object database and will only be invoked 
lazily if the "dependency evaluation graph" triggers that.

As Jim Fulton mentioned in his Zodb-Tour presentation, 
"http://jimfulton.info/presentations/zodb-tour/#/step-1"

*There are only two hard things in Computer Science: cache invalidation and 
naming things.* -- Phil Karlton

ZODB object cache:

   - In memory object graph
   - Transactionally-consistent partial database replica.

I am quite convinced that ZODB has a pretty good dependency tracking 
mechanism to ensure when we access an attribute, it's always up to date.

Then when I looked into some of the Python open-source data flow paradigm, 
I found out that they are explicitly defining their dependency relationship 
and properly handle cache invalidation in memory.
Two of such example projects are: PythonFlow (recently open sourced by 
Spotify) and MDF (open sourced by a number of folks who used to work in the 
Man group hedge funds)
https://github.com/spotify/pythonflow
https://github.com/manahl/mdf

So finally my question, is it possible to leverage these already open 
source technology to create an object-database paradigm... of which data 
are stored in ZODB, and when the objects are invoked, it's being handled 
in-memory with smart invalidation logic as in Data Flow programming.

If you think about it, it's actually similar to how Excel is done today. We 
defined various different functional relationships among different cells, 
and the actual data is stored as "leaves" of the dependency graph.

In the MDF project, they actually have something quite similar, but 
currently only support dependency graph in Class() as opposed to Object 
attributes.
https://github.com/manahl/mdf/issues/23

to the ZODB community, any help or pointers are greatly appreciated!

Cheers,
Byron

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.