Re: Versioning your Model with NoSQL storage

> In one system I wrote the storage was serialized Java objects (blindingly
> fast read/write was objective), and the changes between 1.0 and 2.0 were
> substantial, including aggregate changes. What we did was to serialize the
> whole database to a single massive XML file, and then use a 2-pass SAX XSLT
> to transform it into new format, and then chop it up back into the new
> serialized format.

This works very well on small datasets. If you have a few billion
items in say riak (128 node cluster) this doesn't work so well :)

There is a secondary problem with this in that this is an "offline" migration.

On Tue, Apr 2, 2013 at 12:36 PM, Rickard Öberg <[email protected]> wrote:
> On 4/2/13 17:24 , Greg Young wrote:
>>
>> These are certainly issues with any storage system. There are a few
>> things you can look at.
>>
>> 1) use weak versioning. Make additive/deleting changes to documents
>> don't rename things etc. At that point a simple json serializer should
>> help. It will also allow any version to read any version
>> 2) use a view system of underlying documents. Transform them into the
>> way you want to see them.
>>
>> There are however some more sinister changes with these models. Namely
>> what happens when you have to repartition documents. As an example
>> consider Vehicle and Engine that used to be a single aggregate vehicle
>> but you realize that engines have lifecycles independent of vehicles
>> .... This would require separating the vehicle documents.
>>
>> These kinds of issues exist in all systems. NoSQL databases are strong
>> at handling some (as are SQL databases).
>
>
> In one system I wrote the storage was serialized Java objects (blindingly
> fast read/write was objective), and the changes between 1.0 and 2.0 were
> substantial, including aggregate changes. What we did was to serialize the
> whole database to a single massive XML file, and then use a 2-pass SAX XSLT
> to transform it into new format, and then chop it up back into the new
> serialized format.
>
> It was actually surprisingly pleasant to do it this way, and using XSLT for
> the transformations was fairly intuitive. Because we used SAX for parsing it
> scaled quite well to large databases, the only limitation being how much
> data we needed to keep in memory between first and second phase.
>
> Another app I worked on used Qi4j, which stores entities using JSON
> serialization and include app version nr in each object, and has subsystem
> for data migration rules either on the whole database, or fix-on-read.
>
> /Rickard
>



-- 
Le doute n'est pas une condition agréable, mais la certitude est absurde.


------------------------------------