Parallel serialization and writes ? (Was: ACIDity of Prevayler 2.6)

I've been pouring through the code (version 2.3), both to understand in more details, and see if it is possible to serialize and write transactions completely in parallel with execution. I *think* I understand what is happening, and want to offer a possible explanation and implementation

It seems to me that as a general principle, if you can order writes to the journal in the same order as transaction execution, there is no reason why you cannot do the two in parallel. It would mean that the system state could get ahead of the log by multiple transaction executions, but as long as this is bounded, it doesn't violate the persistence invariants (I'm trying to stay away from the term 'ACID' :-) - the system would still restore properly up to the point of last write. The only transactions lost would be the ones waiting to be written, which would by definition be bounded - probably by a system parameter or some kind of heuristic

So how would you do that? As I understand it now, Prevayler orders journal serialization and write first, using the Guide's Turn,then use that ordering to drive transaction execution (and interleaving queries with that). So how would it be if that were reversed? Transaction would arrive and be dispatched to a parallel journal-writing thread, and then go on to execute, ordered as is done now. The writing thread would prepare the transaction for writing (create the Capsule), then wait for a signal from the transaction execution before actually writing it (actually, that would be the publish operation, so that other publishers (e.g ClientPublish) would get the same ordering).

All that is required to do this is a reference in the transaction execution that references the to-be-created Capsule. When transactions executes, their references are queued in the same order, so that the writing thread can examine that queue each time it is done with preparation - it then decides if it has the Capsule referred to by the head of the queue ready for writing. If it does, it removes the reference and writes the Capsule, if not it goes back to preparing more capsules. You might even prepare each Capsule on it's own thread, and match Capsule creation and queue head each time one finishes. You would not have more threads than there are CPU cores, with an amount left over for user threads.

It seems that this would remove the journal write and serialization latency from transactions in the same way as is now done for queries, while still maintaining the correct ordering and persistence guarantees.

What did I miss ?

-pb


On Nov 17, 2012, at 2:44 AM, Justin T. Sampson wrote:

> Arnaud -- You've got things a bit backwards. Yes, larger latencies are acceptable for transactions than for queries, but that's exactly how Prevayler behaves: Queries execute and return immediately whereas transactions have to wait for writing to disk before executing and returning. If transactions were written to disk after executing, then both queries and transactions would execute immediately but both would wait for writing to disk before returning. Throughput would be similar, but query latency would be worse than it is now.
> 
> Transactions are already serialized concurrently (in memory) without holding any locks, so don't worry about that. And the journal is written as a continuous stream, not individual files, so moving/renaming is not an option (and would be significantly slower).
> 
> 
> On Fri, Nov 16, 2012 at 2:06 AM, Arnaud Masson <[email protected]> wrote:
> Many systems have much more queries than transactions,
> so transaction latency wouldn't be so critical.
> 
> Moreover a transaction could be physically pre-serialized (or maybe in
> parallel thread) on the disk, in some temporary location, before the
> lock scope.
> After tx successful execution, just a quick "file move/rename" operation
> would be required to commit it to the journal, not a full serialization.
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
> To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org

-pb

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov

_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org