Re: Report from using Kafka as Prevalence journal
Ralph Johnson <[email protected]> Tue, 14 Mar 2017 16:13:20 -0500
| Newsgroups | gmane.comp.java.prevayler |
|---|---|
| Message-ID | <CAB_aMcv8yZnvjdEUpxFFqD5twiKg04fsrG2r_UD=JWxeS8gKzg@mail.gmail.com> |
--===============7068811949128726953== Content-Type: multipart/alternative; boundary=f403045e91a4280c01054ab74c66 --f403045e91a4280c01054ab74c66 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable i was thinking of doing something similar, i.e. using a messaging system to keep processes using Prevayler in synch. But the systems I was looking at would ensure that transactions would arrive in the same order for all processes. It sounds like Kafka doesn't do that. Or did I not understand= ? I looked up Kafka after reading your message. I've never looked at it before. It looks well-designed. You say "write speed is crippled". How many transactions per second can it deliver? -Ralph Johnson On Tue, Mar 14, 2017 at 4:02 AM, Karl Wettin <[email protected]> wrote= : > [image: Boxbe] <https://www.boxbe.com/overview> You chose to allow Open > discussion about the Prevayler project. (prevayler-discussion@lists. > sourceforge.net) even though this message failed authentication > Click to disallow > <https://www.boxbe.com/anno?tc_serial=3D29310543226&tc_rand=3D1341927045&= utm_source=3Dstf&utm_medium=3Demail&utm_campaign=3DANNO_AFA&utm_content=3D0= 01&action=3Dauthfail&set=3Dfalse&key=3DuC3uVkXGuUtcik6vMPkTsS2nWCSgICUoD8Mb= mnjcqxE%3D&token=3D9g%2F5SLD81HsuGVR2emB%2Fb8Z0p17qbKMSZwt%2BSOrggii4%2BFX%= 2FDDF2XX4XpbUu3%2FAt> > This message is eligible for Automatic Cleanup! ([email protected]) = Add > cleanup rule > <https://www.boxbe.com/popup?url=3Dhttps%3A%2F%2Fwww.boxbe.com%2Fcleanup%= 3Fkey%3DuC3uVkXGuUv4Wb99Bjk%252BGPzxrK8ieOUyl1gov9ijHh0%253D%26token%3DSD1v= QhkW0iDi7vbXrjcW75E90Fv8TPUuQv%252F8Bp%252FfxG%252BdxXeDTckFZPjXSccVRfu6syL= PHQ7ZbxY4WpnqFAZOtsDWbcqZb7TIcB6%252FDYZQScYaHrBGiLBgzuf4GVNlIGNmPhb0qMB82O= BQKgZg7pSWHw%253D%253D&tc_serial=3D29310543233&tc_rand=3D130196421&utm_sour= ce=3Dstf&utm_medium=3Demail&utm_campaign=3DANNO_CLEANUP_ADD&utm_content=3D0= 01> > | More info > <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=3D293105= 43233&tc_rand=3D130196421&utm_source=3Dstf&utm_medium=3Demail&utm_campaign= =3DANNO_CLEANUP_ADD&utm_content=3D001> > > Perhaps some of you find this interesting. > > > For a little while now we've been running a Prevalance pattern that use > Kafka as journal to handle multiple load balanced and availability nodes = of > the same data store. Transactions have unique identities and when execute= d > by the user on a local node they are sent to Kafka, waited upon to bounce > back from Kafka and then executed on the local node using a CountDownLatc= h > to make sure the transaction has been distributed. Calls on a local node > are kept in order other using a ConcurrentReadWriteLock. It's very > simplistic, some 200 lines of code. > > > The obvious drawbacks is that: > > * Write speed is crippled, bound to the Kafka polling rate. This will of > course also affect the read speed at the local node on which the > transaction was created, as the node will be locked down for further > queries and transaction until the waited upon transaction has been receiv= ed > and executed. > > * The nodes are out of sync at a level equal to the network latency. > Queries executed on different nodes can for a short time return different > results but there is eventual consistency. One can however often implemen= t > transactions and queries in a manner so that this never occurs (e.g. neve= r > request data that has not been created or is about to be updated), or > simply by accepting this caveat as a system limitation. > > For us, none of the above seems to be a problem at this point. > > > Well implemented transactions with validation before broadcasting a > transaction and the same validation before executing the transaction seem= s > to handle when multiple nodes simultaneously attempt to broadcast > incompatible transactions, then only the first to arrive to Kafka will > succeed and the others will receive a validation exception. This does > however mean that the journal might contain transactions that indeed will > fail. > > I suppose it would be possible to use Kafka as an inter-node write-lock i= f > one is OK with adding even more latency, and thus removing a few of the > caveats. But I'm not going there voluntarily. > > > All in all it seems to be working great. > > So far there are no snapshots, but the idea is to schedule starting up a > new service node to which no clients will connect, take a snapshot and th= en > shut it down again. > > > k > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > To unsubscribe go to the end of this page: http://lists.sourceforge.net/ > lists/listinfo/prevayler-discussion > _______________________________________________ > "Databases in Memoriam" -- http://www.prevayler.org > > --f403045e91a4280c01054ab74c66 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">i was thinking of doing something similar, i.e. using a me= ssaging system to keep processes using Prevayler in synch. =C2=A0 =C2=A0But= the systems I was looking at would ensure that transactions would arrive i= n the same order for all processes. =C2=A0 It sounds like Kafka doesn't= do that.=C2=A0 Or did I not understand?<div><br></div><div>I looked up Kaf= ka after reading your message.=C2=A0 I've never looked at it before. = =C2=A0 It looks well-designed. =C2=A0 You say "write speed is crippled= ".=C2=A0 How many transactions per second can it deliver?</div><div><b= r></div><div>-Ralph Johnson</div></div><div class=3D"gmail_extra"><br><div = class=3D"gmail_quote">On Tue, Mar 14, 2017 at 4:02 AM, Karl Wettin <span di= r=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_blank">k= [email protected]</a>></span> wrote:<br><blockquote class=3D"gmail_q= uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e= x"> <div style=3D"font-size:12px;color:#777;font-family:'Lucida Grande'= ,Helvetica,Arial,sans-serif;background-color:#fff;padding:4px"> <a href=3D"https://www.boxbe.com/overview" style=3D"text-decoration:none;co= lor:#5e96ea" target=3D"_blank"><img alt=3D"Boxbe" src=3D"http://www.boxbe.c= om/images/logo_dark_small.png" width=3D"64px" style=3D"margin-left:0px;bord= er:none"></a> <img src=3D"http://www.boxbe.com/stfopen?tc_serial=3D29310543226&tc_ran= d=3D1341927045&utm_source=3Dstf&utm_medium=3Demail&utm_campaign= =3DANNO_AFA&utm_content=3D001"> You chose to allow Open discussion about the Prevayler project. (<a href=3D= "mailto:[email protected]" target=3D"_blank">preva= yler-discussion@lists.<wbr>sourceforge.net</a>) even though this message fa= iled authentication <br><a style=3D"text-decoration:none;color:#5e96ea" href=3D"https://www.box= be.com/anno?tc_serial=3D29310543226&tc_rand=3D1341927045&utm_source= =3Dstf&utm_medium=3Demail&utm_campaign=3DANNO_AFA&utm_content= =3D001&action=3Dauthfail&set=3Dfalse&key=3DuC3uVkXGuUtcik6vMPkT= sS2nWCSgICUoD8MbmnjcqxE%3D&token=3D9g%2F5SLD81HsuGVR2emB%2Fb8Z0p17qbKMS= Zwt%2BSOrggii4%2BFX%2FDDF2XX4XpbUu3%2FAt" target=3D"_blank">Click to disall= ow</a> <br> This message is eligible for Automatic Cleanup! (<a href=3D"mailto:karl.wet= [email protected]" target=3D"_blank">[email protected]</a>)=20 <a style=3D"text-decoration:none;color:#5e96ea" href=3D"https://www.boxbe.c= om/popup?url=3Dhttps%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DuC3uVkXGuUv4Wb= 99Bjk%252BGPzxrK8ieOUyl1gov9ijHh0%253D%26token%3DSD1vQhkW0iDi7vbXrjcW75E90F= v8TPUuQv%252F8Bp%252FfxG%252BdxXeDTckFZPjXSccVRfu6syLPHQ7ZbxY4WpnqFAZOtsDWb= cqZb7TIcB6%252FDYZQScYaHrBGiLBgzuf4GVNlIGNmPhb0qMB82OBQKgZg7pSWHw%253D%253D= &tc_serial=3D29310543233&tc_rand=3D130196421&utm_source=3Dstf&a= mp;utm_medium=3Demail&utm_campaign=3DANNO_CLEANUP_ADD&utm_content= =3D001" title=3D"Add a new automatic cleanup rule" target=3D"_blank">Add cl= eanup rule</a> | <a style=3D"text-decoration:none;color:#5e96ea" href=3D"http://blog.boxb= e.com/general/boxbe-automatic-cleanup?tc_serial=3D29310543233&tc_rand= =3D130196421&utm_source=3Dstf&utm_medium=3Demail&utm_campaign= =3DANNO_CLEANUP_ADD&utm_content=3D001" title=3D"Get info on automatic c= leanup" target=3D"_blank">More info</a> <br> </div> <br>Perhaps some of you find this interesting.<br> <br> <br> For a little while now we've been running a Prevalance pattern that use= Kafka as journal to handle multiple load balanced and availability nodes o= f the same data store. Transactions have unique identities and when execute= d by the user on a local node they are sent to Kafka, waited upon to bounce= back from Kafka and then executed on the local node using a CountDownLatch= to make sure the transaction has been distributed. Calls on a local node a= re kept in order other using a ConcurrentReadWriteLock. It's very simpl= istic, some 200 lines of code.<br> <br> <br> The obvious drawbacks is that:<br> <br> * Write speed is crippled, bound to the Kafka polling rate. This will of co= urse also affect the read speed at the local node on which the transaction = was created, as the node will be locked down for further queries and transa= ction until the waited upon transaction has been received and executed.<br> <br> * The nodes are out of sync at a level equal to the network latency. Querie= s executed on different nodes can for a short time return different results= but there is eventual consistency. One can however often implement transac= tions and queries in a manner so that this never occurs (e.g. never request= data that has not been created or is about to be updated), or simply by ac= cepting this caveat as a system limitation.<br> <br> For us, none of the above seems to be a problem at this point.<br> <br> <br> Well implemented transactions with validation before broadcasting a transac= tion and the same validation before executing the transaction seems to hand= le when multiple nodes simultaneously attempt to broadcast incompatible tra= nsactions, then only the first to arrive to Kafka will succeed and the othe= rs will receive a validation exception. This does however mean that the jou= rnal might contain transactions that indeed will fail.<br> <br> I suppose it would be possible to use Kafka as an inter-node write-lock if = one is OK with adding even more latency, and thus removing a few of the cav= eats. But I'm not going there voluntarily.<br> <br> <br> All in all it seems to be working great.<br> <br> So far there are no snapshots, but the idea is to schedule starting up a ne= w service node to which no clients will connect, take a snapshot and then s= hut it down again.<br> <br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 k<br> ------------------------------<wbr>------------------------------<wbr>-----= -------------<br> Check out the vibrant tech community on one of the world's most<br> engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot" rel= =3D"noreferrer" target=3D"_blank">http://sdm.link/slashdot</a><br> ______________________________<wbr>_________________<br> To unsubscribe go to the end of this page: <a href=3D"http://lists.sourcefo= rge.net/lists/listinfo/prevayler-discussion" rel=3D"noreferrer" target=3D"_= blank">http://lists.sourceforge.net/<wbr>lists/listinfo/prevayler-<wbr>disc= ussion</a><br> ______________________________<wbr>_________________<br> "Databases in Memoriam" -- <a href=3D"http://www.prevayler.org" r= el=3D"noreferrer" target=3D"_blank">http://www.prevayler.org</a><br> <br></blockquote></div><br></div> --f403045e91a4280c01054ab74c66-- --===============7068811949128726953== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot --===============7068811949128726953== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion _______________________________________________ "Databases in Memoriam" -- http://www.prevayler.org --===============7068811949128726953==--