Report from using Kafka as Prevalence journal

Perhaps some of you find this interesting.


For a little while now we've been running a Prevalance pattern that use Kafka as journal to handle multiple load balanced and availability nodes of the same data store. Transactions have unique identities and when executed by the user on a local node they are sent to Kafka, waited upon to bounce back from Kafka and then executed on the local node using a CountDownLatch to make sure the transaction has been distributed. Calls on a local node are kept in order other using a ConcurrentReadWriteLock. It's very simplistic, some 200 lines of code. 


The obvious drawbacks is that:

* Write speed is crippled, bound to the Kafka polling rate. This will of course also affect the read speed at the local node on which the transaction was created, as the node will be locked down for further queries and transaction until the waited upon transaction has been received and executed.

* The nodes are out of sync at a level equal to the network latency. Queries executed on different nodes can for a short time return different results but there is eventual consistency. One can however often implement transactions and queries in a manner so that this never occurs (e.g. never request data that has not been created or is about to be updated), or simply by accepting this caveat as a system limitation.

For us, none of the above seems to be a problem at this point. 


Well implemented transactions with validation before broadcasting a transaction and the same validation before executing the transaction seems to handle when multiple nodes simultaneously attempt to broadcast incompatible transactions, then only the first to arrive to Kafka will succeed and the others will receive a validation exception. This does however mean that the journal might contain transactions that indeed will fail.

I suppose it would be possible to use Kafka as an inter-node write-lock if one is OK with adding even more latency, and thus removing a few of the caveats. But I'm not going there voluntarily.


All in all it seems to be working great. 

So far there are no snapshots, but the idea is to schedule starting up a new service node to which no clients will connect, take a snapshot and then shut it down again.


		k
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org