Re: ZODB with Lazy evaluation / Caching / Data Flow programming

Jim, Chris:

Thanks very much for your quick response. @Chris - thanks also to your
contribution to the ZODB community. I was listening to your YouTube talk
and the ZODB Demo app. It has helped me to learn to use ZODB better.

I am not building any top-secret project; Rather, Rather, I am trying to
use ZODB in a lot of places in my new job.

My experience perhaps it's a bit unique -- most people learn to just use
relational DB and get the job done. On the other hand, I actually have been
using Object Database in modelling and solving problem in my previous
company for ~8 years .... (not having to deal with the Object Relation
Impedance is awesome). However as I left the place, I have found myself
been trying to fight the productivity and efficiency I used to have (that's
before I bump into ZODB)

Now of course there are difference in how ZODB works and how the previous
in-house Object DB works.... and so I have been trying to find tools in the
Open Source community to replicate similar infrastructure. ZODB has solved
the object persistence problem, and the major missing piece now it's really
the whole Computational in-memory dependency graph caching/in-validation,
hence prompting me to ask the question here.

For my actual use case, I am building a Flask-based webApp which would have
controllers logic and models using ZODB. It's expected to run on multiple
computers. I am planning to use ZODB with RelStore backed by PostgresQL
(managed by Google Cloud SQL). Objects will be stored in the database, and
simple calculation will be Web service level. I will also be using Airflow
(the scheduler infrastructure written in Python) to schedule periodic
jobs.... that will query the database, pull objects, crunch some heavy
numbers, and write results back to Database.

With regards to the consistency across computers issues -- yes in my case,
it's fine that the read-only client are using a slightly stale data. In
fact, that was the model in my previous company... from time to time, they
will have to "refresh" the in-memory cache and ensure the state is
consistent with that's in the database. I saw the YouTube demo on 2 Tier
Kanban by Jim ... (this link
https://www.youtube.com/watch?v=6v1sIw5j4ss&t=11s)  and see how new feature
can automatically pop up after object being persisted in DB... that's seems
pretty awesome, but my guess is without some proper listener logic, it's
hard to achieve?

There are other projects that solved the cached dependency problems (like
the PythonFlow and MDF). However they are usually focusing on the
computational part -- and usually they don't design around the Object
Oriented notion... (they are usually functional in nature). As a result, I
often run into another kind of "impedance" problem -- in that when I am
modelling business logic and behavior... Object paradigm and ZODB are
perfect...... yet if I want to use the dependency tracking in-memory, I
have transform the object data into some functional representation.

In any event, glad to actually start posting my first post here in this
community. I also are experimenting the Newt.db thing.... seems a great
balance between the convenience in modelling and powerfulness of search.

Cheers,
Byron



On Fri, Apr 27, 2018 at 1:32 PM, Christopher Lozinski <
[email protected]> wrote:

> Thank you Jim for the excellent answer. The ability to
> update remote user interfaces sounds very interesting indeed.
>
>
> Baron, welcome to the ZODB community.  I am curious what you re building?
> What is the context for this question?  Or is it a top-secret startup?
>
> In particular what are your transactional requirements?  Is this
> application running on multiple computers?   Do you want to store the
> objects in the database, but have the calculations only be in RAM?  What
> about consistency across computers?
>
> Jim, please correct me if I am wrong, but here is how I believe that the
> ZODB works.  If both ZEO clients are reading a value, no problem.  If both
> are writing a value, one is first, its write completes, the other computer
> gets an error.  But if one is reading, while the other is writing, but
> before the cache is invalidated, the read goes through.  Even though the
> data is a bit out of date, it is not a problem in most cases.  Jim, is that
> a correct understanding?
>
> If that is the case, Baron do you need to make sure that no other server
> has recently updated a value?  How do the distributed dependency
> relationship graphs handle this problem?
>
> Warm Regards
> Chris
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.