Re: [dev] IStorage.lastOID() proposal (was Re: [dev] RFC IMVCCStorage everywhere (Move ZODB's MVCC to storage layer))

Le 06/16/16 18:11, Jim Fulton a écrit :
>> In order to migrate transparently several ZODB into a single NEO DB, we need the greatest committed oid of each source, so we easily stack oids range (then pickles are transformed on the fly to map oids). The oids from the first source DB don't change (except for mount points), those are other source DBs are shifted. The sum of all last_oid is the new last_oid, and new oids are allocated from there.
> 
> Is anything else is writing to the destination database?

The client.

To make things clearer, I restart from the beginning. Let's 3 source DBs, whose oid ranges are 0..19, 0..29 and 0..39. The destination DB maps as follows:
- DB1: 0..19 -> 0..19
- DB2: 0..29 -> 20..49
- DB3: 0..39 -> 50..89
Which means that the destination DB starts with last_oid=89 and the master node delivers new oids starting from 90.
Clients can commit without having to wait that the migration (any access to something that is not migrating yet is translated to an access to a source DB).

> - ZEO (and I assume NEO and RelStorage) have to make server calls. ZEO
> mitigates this by
>   requesting multiple reservations in a single server call.

I confirm for NEO.

> And of course, sequentially allocating IODs makes merging storages difficult.

Not that much. At least, the saved memory (by not mapping every oid) is worth it.

> An alternative I've been considering us to allocate OIDs randomly:
> 
> - With 1<<64 possible values, collisions are wildly unlikely, even
> when merging databases
> 
> - Collisions are easily detected.  When store is called with a serial
> of None or z64 (maybe we should pick one), then we know we're adding a
> new object and we can check that the oid isn't being used.
> 
> - When we detect a collision, we can raise a transient error causing
> the request to be retried.

We could not guarantee anymore that a specific transaction that create oids will always succeed, even with serialization or conflict resolution.

We have applications that perform non-transaction tasks, like communications with external services. In this case, we already take care of isolating it inside a transaction that never fails. I have to think whether the inability to create oid is an issue.

About aesthetics, I can say that initially NEO used UUIDs to identify nodes and that was painful to debug.

Julien

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.