Re: [dev] RFC IMVCCStorage everywhere (Move ZODB's MVCC to storage layer)

Le 06/16/16 14:46, Jim Fulton a écrit :
>> I'm not enthusiast with so many lines of glue code and extra function calls.
> 
> I don't think there are more lines of code or more function calls. The
> MVCCAdapter work just moved some code around.

1. It's one layer of dummy function calls. For example, newTransaction called storage.sync() directly, and now it would call adapter.sync() which would in turn call storage.sync(). Same for load and other storage methods.

2. more lines of code, mainly to copy attributes, also some registerDB logic

Generally speaking, I find the whole result more complex than just seeing Connection code calling methods directly on the real storage, and if the storage does not overload, look at the inherited mixin.

>> I would understand this way of implementing things if you'd like not to break existing code but we're already discussing of a major release.
> 
> The main impact of ZODB5, having nothing to do with IMVCCAdapter, is
> on ZEO and NEO.  ZEO required meaningful, but minor changes. Given
> that the changes reflect the way NEO already worked, I'm curious what
> impact these changes have on NEO.  I would have thought none, or few.

You can see all the changes at
 https://lab.nexedi.com/nexedi/neoppod/commit/e7b648e6b9c10d7207ac19348a268d1374c3f8a0

Except for the monkey-patch that is specific to ZODB < 5, I could have implemented this way earlier, but I was not forced to do it.

For example, lastTransaction() must not ask the server anymore.

>> I'd rather merge IMVCCStorage into IStorage, and for ZODB implementations that don't implement IMVCCStorage, let them inherit a mixin (a mix of MVCCAdapter and MVCCAdapterInstance), the same way as it is done for ConflictResolvingStorage.
> 
> I don't consider ConflictResolvingStorage a success story. Generally,
> I don't like providing public inheritance APIs. Conflict resolution
> would have been better provided via composition.

(Taste and colours...)

I see there's also BlobStorageMixin.

> Of course, there's nothing preventing any storage implementation from
> implementing IMVCCStorage other than the fact that IMVCCStorage needs
> more thought.

And in the case NEO, that consists in duplicating code from mvccadapter.py

>> 2. IStorage.lastOID() ?
> 
> You need to be more specific. This doesn't exist, so I guess you're
> proposing this.

Yes.

>> We'd use it to migrate databases to NEO. Currently, we hack by calling new_oid(), without actually modifying the DB, which prevent us from opening a ZODB in read-only.
>>
>> I know you suggested a different way to implement transparent migration, but the memory cost is in O(number of oids) and it is quite a lot of non-trivial changes in NEO whereas the current implementation has already been well-tested.
>>
>> In ZODB, this would very little change, which I'm ready to do. Same for RelStorage (3 simple methods in adapters + 1 relstorage.py). For ZEO, a new simple rpc is needed.
> 
> I don't know what you're proposing exactly or what problem it's
> solving. You need to be more specific.

In order to migrate transparently several ZODB into a single NEO DB, we need the greatest committed oid of each source, so we easily stack oids range (then pickles are transformed on the fly to map oids). The oids from the first source DB don't change (except for mount points), those are other source DBs are shifted. The sum of all last_oid is the new last_oid, and new oids are allocated from there.

For example, the BaseStorage implementation (which applies to FileStorage) would be:

  def lastOID(self): # or last_oid ?
    return self._oid

> I think the idea of making all of the breaking changes at once has
> been proven to be a bad one by Python 3 and Zope 3.

OK. And since you'd like to rethink IMVCCStorage, the discussion on adapters can be postponed. I'd have to show draft implementation to be more convincing.

Julien

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.