Re: [dev] IStorage.lastOID() proposal (was Re: [dev] RFC IMVCCStorage everywhere (Move ZODB's MVCC to storage layer))

On Thu, Jun 16, 2016 at 5:04 PM, Julien Muchembled <[email protected]> wrote:
> Le 06/16/16 18:11, Jim Fulton a écrit :
>>> In order to migrate transparently several ZODB into a single NEO DB, we need the greatest committed oid of each source, so we easily stack oids range (then pickles are transformed on the fly to map oids). The oids from the first source DB don't change (except for mount points), those are other source DBs are shifted. The sum of all last_oid is the new last_oid, and new oids are allocated from there.
>>
>> Is anything else is writing to the destination database?
>
> The client.
>
> To make things clearer, I restart from the beginning. Let's 3 source DBs, whose oid ranges are 0..19, 0..29 and 0..39. The destination DB maps as follows:
> - DB1: 0..19 -> 0..19
> - DB2: 0..29 -> 20..49
> - DB3: 0..39 -> 50..89
> Which means that the destination DB starts with last_oid=89 and the master node delivers new oids starting from 90.

Interesting, so the destination database participates in this.

Your essentially reserving the oids, which you could (maybe do) do
with new_oid (or new_oids in ZEO).

I guess you're using lastOID on the source databases to determine the
reservations you need.

> Clients can commit without having to wait that the migration (any access to something that is not migrating yet is translated to an access to a source DB).
>
>> - ZEO (and I assume NEO and RelStorage) have to make server calls. ZEO
>> mitigates this by
>>   requesting multiple reservations in a single server call.
>
> I confirm for NEO.
>
>> And of course, sequentially allocating IODs makes merging storages difficult.
>
> Not that much. At least, the saved memory (by not mapping every oid) is worth it.

Because you only need to know the offset for each source.

>
>> An alternative I've been considering us to allocate OIDs randomly:
>>
>> - With 1<<64 possible values, collisions are wildly unlikely, even
>> when merging databases
>>
>> - Collisions are easily detected.  When store is called with a serial
>> of None or z64 (maybe we should pick one), then we know we're adding a
>> new object and we can check that the oid isn't being used.
>>
>> - When we detect a collision, we can raise a transient error causing
>> the request to be retried.
>
> We could not guarantee anymore that a specific transaction that create oids will always succeed, even with serialization or conflict resolution.
>
> We have applications that perform non-transaction tasks, like communications with external services. In this case, we already take care of isolating it inside a transaction that never fails. I have to think whether the inability to create oid is an issue.

This is an interesting approach. I doubt that you can have truly
failure free transactions. Network or servers can fail.  Presumably
the probability of failure is very low.  I doubt the risk of a random
oid collision would increase the chance of failure significantly. Even
if you had several billion objects in your database, the chance of a
collision would less than one in a billion (assuming you're adding a
single object).

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.