[dev] IStorage.lastOID() proposal (was Re: [dev] RFC IMVCCStorage everywhere (Move ZODB's MVCC to storage layer))

On Thu, Jun 16, 2016 at 10:43 AM, Julien Muchembled <[email protected]> wrote:
> Le 06/16/16 14:46, Jim Fulton a écrit :
...
>>> 2. IStorage.lastOID() ?
>>
>> You need to be more specific. This doesn't exist, so I guess you're
>> proposing this.
>
> Yes.
>
>>> We'd use it to migrate databases to NEO. Currently, we hack by calling new_oid(), without actually modifying the DB, which prevent us from opening a ZODB in read-only.
>>>
>>> I know you suggested a different way to implement transparent migration, but the memory cost is in O(number of oids) and it is quite a lot of non-trivial changes in NEO whereas the current implementation has already been well-tested.
>>>
>>> In ZODB, this would very little change, which I'm ready to do. Same for RelStorage (3 simple methods in adapters + 1 relstorage.py). For ZEO, a new simple rpc is needed.
>>
>> I don't know what you're proposing exactly or what problem it's
>> solving. You need to be more specific.
>
> In order to migrate transparently several ZODB into a single NEO DB, we need the greatest committed oid of each source, so we easily stack oids range (then pickles are transformed on the fly to map oids). The oids from the first source DB don't change (except for mount points), those are other source DBs are shifted. The sum of all last_oid is the new last_oid, and new oids are allocated from there.

Is anything else is writing to the destination database?

(Apologies, you explained this to me before, but it's been a while...)

>
> For example, the BaseStorage implementation (which applies to FileStorage) would be:
>
>   def lastOID(self): # or last_oid ?
>     return self._oid

So, the problem I have with  lastOID is that it assumes that OIDs are
allocated sequentially.  This is what is mostly done now, but it has
issues, in particular, clients have to reserve oids by calling
new_oid()

- ZEO (and I assume NEO and RelStorage) have to make server calls. ZEO
mitigates this by
  requesting multiple reservations in a single server call.

- If clients don't commit, the reserved oids are wasted. There are
1<<64 possible IDs, so this is OK.

And of course, sequentially allocating IODs makes merging storages difficult.

An alternative I've been considering us to allocate OIDs randomly:

- With 1<<64 possible values, collisions are wildly unlikely, even
when merging databases

- Collisions are easily detected.  When store is called with a serial
of None or z64 (maybe we should pick one), then we know we're adding a
new object and we can check that the oid isn't being used.

- When we detect a collision, we can raise a transient error causing
the request to be retried.

Storages actually do the collision check today by accident.  If you
create an object with an oid that's in use, the serial you pass won't
match the stored serial and you'll get a conflict error.  This could
be refined by providing a more specific exception.

This would greatly simplify your database merging problem as well as a
similar problem in DemoStorage.  It would allow some storages to stop
making server calls to allocate oids.  You might not even have to
reallocate oids, or reallocate oids only be exception. You could even
merge to a database that was being written to.

Ane issue might be with the restore method. It doesn't check for oid
collisions, even by accident and would need to if you were merging
multiple databases.

Man, I've 97% talked myself into allocating oids randomly in ZEO. :)

Jim

P.S. When I first started thinking about this some time ago, I was
thinking of using UUIDs, but given that detecting and handling
collisions is easy, I think we can get by with 64 bits. :)

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.