Re: ZODB in IOT

On Sun, Apr 22, 2018 at 12:21 AM, Christopher Lozinski <
[email protected]> wrote:
...

> And I remember Jim once very wisely saying that IoT data collection
> usually does not need transactions.
>
You wound me with my own words! ;-)

This is a good point. When collecting lots and lots of data that are new
that can be inserted as individual (possibly complex, e.g. documents)
pieces of data. You don't need ACID, ___D will suffice.

Arguably ACID is overkill.  Still, I think what I'm suggesting has value
for that middle ground of applications that don't have truly big-data
needs.  I fear that people on this list aren't in that space.

...

Currently blobs are parts of transactions.   How about if we could have IoT
> blobs which are not always parts of transactions.  Create your IoT sensor,
> create the IoT blob and it goes ahead and collects data.  Occasionally when
> you need to rotate your log files, you execute a transaction, and one IoT
> blob file closes, and a new one opens.
>
So reduce the transaction burden by amortizing the cost over many records
by batching them; makes sense.

At that point, however, while batching data, ___D is on you.

You could use something like AWS Kinesis to do this buffering for you, at
least puting ___D on them (for a while), and generally requiring less
infrastructure.


> The ZODB is a tree of objects.  It would be very natural to have a similar
> tree of IoT blobs.  Okay, and here we get to the problem which I am facing.
> When I cut and paste branches of my ZODB model, I would love to execute a
> transaction on the files as well. So that I can move the whole  branch of
> IoT Blobs to the new location in the file system.  And everything just
> works.
>
Moving a subtree around has almost no cost. Depending on how you implement
the tree. You're either modifying a couple of container children
collections and/or a parent pointer.  The objects underneath are unaffected
(modulo indexing). You don't need any sort of transaction on the files,
because you don't need to move them in a physical file system, you're
moving them in the logical, ZODB-based file system.

Of course, physical file systems work the same way.  You can move a massive
directory and virtually zero cost.

...

There is also the question of big data.
>
That's not a problem space that ZODB should be used for, although it could
be useful for metadata.


>  When I give ZODB talks, people are always asking me about scalability.
>
Read scalability is very good. Write scalability is always going to be
limited, because of the GTL, but is enough for most applications. A lot
depends on application requirements, which people don't want to here,
because it means they have to think.


>   The core domain model may not be that big but the amount of data
> collected can be huge.
>
That's hard to reason about in the abstract.


> How about if we could create these IoT blobs on remote servers.  And then
> when we rotate them, or move them, we do a two phase commit across the
> network.    Say a IoT Blob server gets too full, we start up a new server,
> move something and keep working.
>
There's a ton of tools for something like this. This isn't a place where
ZODB can help.  We are not big data experts and shouldn't be designing
big-data solutions with ZODB.


> I am also mindful that Neo4J works on multiple computers.  This would be a
> different approach to data distribution than Neo4J uses.  Maybe this is
> similar to hosting blobs on Amazon s3.  The big difference is that those
> blobs can grow as data is collected.
>
> I need this.  Does anyone else need this?
>
If you're really interested in trying to scale much further with ZODB, you
should check out NEO https://neo.nexedi.com/ and Wendelin
https://lab.nexedi.com/nexedi/wendelin.core.

Heck, I should check them out, but the setup always seemed rather involved,
and I've been too lazy to.  Hm, this seems ripe for dockerizing.

Getting back to the idea I posted yesterday, there are non-big-data
problems where numpy/pandas/R work, where you can fit data in ram (or, with
memory mapping, local disk), where I hope ZODB can help. Wendelin seems to
be an existence proof.  I'm interested in doing something a bit simpler as
well.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.