Re: Re: query about after commit hook

On Fri, May 20, 2016 at 4:01 AM, Thierry Florac <[email protected]> wrote:
> Hi,
> I often use "after commit hooks" to handle actions which are not
> "transaction aware" (for example, to do an update on a file system or on a
> remote server).

Is it OK if these actions fail?

> If I want to make these updates only if my transaction is committed, how can
> I do it without using these hooks ?

So, first of all, it's difficult to mix transactional and
non-transactional code.  Maybe there should be a mechanism that makes
this easier, but I don't think after commit hooks do this well. They
*seem* to do what you want, but leave applications open to subtle
bugs.

A better approach, IMO, is to transactionally record the actions you
want to take persistently as part of the transaction. Then try to do
the action in the after commit hook and have a fallback mechanism to
handle queued actions that can't be handled immediately.

Ironically, getting queueing right is hard too.

zc.async implements a queue in ZODB.  I like this idea a lot, but the
implementation was extremely complex and caused lots of operational
pain. I'm not sure is this was due to implementation choices, or
because the problem is hard.

We ended up using Amazon's SQS, but even then, there were issues.
Actually calling SQS can fail. For us, this was extremely rare, but
left us in a bad place when errors did occur. So we implemented a very
simple ZODB-based queue that we used to record the intention to submit
SQS jobs.  We'd add a job to this queue, then submit the job to SQS in
an after-commit hood.  If that failed, a separate process would handle
jobs left in the ZODB queue too long.

Then there's the fact that the after-commit hook implementation is
quirky. For example, see:

https://groups.google.com/forum/#!topic/python-transaction/rwB4RI3nMr8

Another idea I've had is to have a min-commit hook that runs between
tpc_vote and tpc_finish.  If it failed, then the transaction would be
rolled back, and if it succeeded, then the transaction would be
guaranteed ;) to complete. A major disadvantage of this approach is
that you really don't want to delay after vote because locks are held
at that point. This might be OK if the action was really fast, or if
vote got find-grained (object-level) locks.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.