[dev] Better tpc_finish (was Re: [dev] RFC IMVCCStorage everywhere (Move ZODB's MVCC to storage layer))

On Thu, Jun 16, 2016 at 4:09 PM, Julien Muchembled <[email protected]> wrote:
> Ahhh tpc_finish !
>
> Before today, I only tested NEO on the no-more-load branch before the adapters were merged. I have errors again.

Sorry!

> Some because tests do 'connection._storage.iterator', which I changed into 'connection.db().storage.iterator'
>
> But what annoys me is the monkey-patch we have for tpc_finish, which would become more complicated, mainly because of missing 'return' in adapters[1], but also because Connection.tpc_finish differs between ZODB5 and older versions[2].
>
> I thought I could leave this subject aside for ZODB5, but instead of wasting time inside NEO, I should rather work on pushing this ZODB before anything else.

Let's make this better. :)

> Hmm... not sure everyone remembers exactly what I am talking about. The monkey-patch was sent long time ago on this mailing-list (early 2012). So, in NEO, we don't want to serialize of the second phase of 2PC, which means:
> - the real tid is only known during tpc_finish
> - before tpc_finish, NEO works with a temporary tid
> - NEOStorage.tpc_vote only returns oids for which a conflict happened
> - NEOSTorage.tpc_finish returns the committed tid
> - and what tpc_vote usually does for other ZODB implementation has to be done at the end of Connection.tpc_finish
>
> You can find the monkey-patch at
>   https://lab.nexedi.com/nexedi/neoppod/blob/v1.6.3/neo/client/__init__.py#L35
>
> https://lab.nexedi.com/nexedi/neoppod/blob/v1.6.3/ZODB.patch
> contains other changes:
> - update of IStorage.tpc_finish doc string
> - update of existing unit tests

This is the 21st Century. You should start submitting pull requests. :)

I'm with you.  I'm planning to make ZEO5 like NEO in this regard and
there's even a new comment in ZODB that alludes to this.

>
> TODO:
> - 2 lines are removed in tpc_vote, instead of being reworded
> - write a new ZODB test to check this feature
> - it's been a long time since the whole ZODB test suite was run with this patch

Let's change ZODB5 to make this easier/saner for NEO and future ZEO.

Let's think of how we **want** this to work and work out a transition
plan.  I'd like to do this in a way that doesn't break existing
storages if practical.

Here's a straw man:

- store, undo, and tpc_vote are no longer required to return serials,
except when they want to signal oids who's conflicts have been
resolved.

- tpc_finish may optionally return a tid. If it does, then that tid
should be applied to modified objects who's without resolved
conflicts.

- If tpc_finish doesn't return a tid, then store, undo, and tpc_vote
must return serials for all stored objects.

- There are some special cases where undo or tpc_vote should return
serials for without conflicts, because objects were modified via unto.
The serials in this case may be None, as long as tpc_finish returns a
tid.

Eventually, it will be expected for tpc_finish to return a tid.

Thoughts?

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.