ZODB in IOT

Jim fulton asked about ZODB for data science.  That is a brilliant target market.  The market is growing, there is lots of money there.  Good software is needed. 

Then there was this discussion of IoT.  And I remember Jim once very wisely saying that IoT data collection usually does not need transactions.  So here is what I would like to see. 

Currently blobs are parts of transactions.   How about if we could have IoT blobs which are not always parts of transactions.  Create your IoT sensor, create the IoT blob and it goes ahead and collects data.  Occasionally when you need to rotate your log files, you execute a transaction, and one IoT blob file closes, and a new one opens. 

The ZODB is a tree of objects.  It would be very natural to have a similar tree of IoT blobs.  Okay, and here we get to the problem which I am facing. When I cut and paste branches of my ZODB model, I would love to execute a transaction on the files as well. So that I can move the whole  branch of IoT Blobs to the new location in the file system.  And everything just works. 

Why do I want to do that?  Well because I want to serve those files with NGINX.  And if the url is parallel to the ZODB object URL, all of that is really easy to do. 

There is also the question of big data.  When I give ZODB talks, people are always asking me about scalability.  The core domain model may not be that big but the amount of data collected can be huge.  How about if we could create these IoT blobs on remote servers.  And then when we rotate them, or move them, we do a two phase commit across the network.    Say a IoT Blob server gets too full, we start up a new server, move something and keep working. 

I am also mindful that Neo4J works on multiple computers.  This would be a different approach to data distribution than Neo4J uses.  Maybe this is similar to hosting blobs on Amazon s3.  The big difference is that those blobs can grow as data is collected. 

I need this.  Does anyone else need this?


Warm Regards
Chris

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.