Re: Report from using Kafka as Prevalence journal
Marcel Oerlemans <[email protected]> Mon, 27 Mar 2017 15:00:57 +0200
| Newsgroups | gmane.comp.java.prevayler |
|---|---|
| Message-ID | <CAKE0621-ouf3BAj9oQP0=Mr4SNeZt2cFixquXHwZbU7g6pfR1A@mail.gmail.com> |
--===============6259968846844080697== Content-Type: multipart/alternative; boundary=001a114c6cc437a6cd054bb5ef0d --001a114c6cc437a6cd054bb5ef0d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sometimes I feel we are like wizards, with our own Arcane incantations ;) Much thanks for the Docker line though! On Mon, Mar 27, 2017 at 2:52 PM, Karl Wettin <[email protected]> wrote= : > Kafka doesn't have to be that tricky, at least not to get started with it= : > > https://github.com/spotify/docker-kafka/ > > docker run -p 2181:2181 -p 9092:9092 --rm --env > ADVERTISED_HOST=3D10.40.1.52 --env ADVERTISED_PORT=3D9092 --name kafka -h= kafka > spotify/kafka > > // kafka properties: > config.put("auto.offset.reset", "earliest"); > config.put("group.id", UUID.randomUUID().toString()); > > That's pretty much all configuration you need. > > > kalle > > > On 14 Mar 2017, at 23:42, Robert Friberg <[email protected]> > wrote: > > > > Hi, > > > > We=E2=80=99ve been using this approach as well with OrigoDB. A key stre= ngth not > mentioned by Karl is that additional consumers of the transaction stream > can generate arbitrary read models, either in memory or on disk, in real > time or on demand. > > > > Kafka is very mature and popular but it requires a LOT of knowledge to > install, configure and operation. Ask me how I know =F0=9F=98=8A > > To understand the kafka model, it=E2=80=99s easy to think of a persiste= nt > pub/sub message bus. A producer publishes messages to a topic and consume= rs > subscribe to topics. > > > > Yes, message ordering is preserved and the delivery guarantee is at > least once, so you should probably have a deduplication step in the > pipeline. > > > > Kafka is relatively slow if you write a single message using a producer > and wait for the message using a consumer. You can configure the poll > interval to be more aggressive but sending a single transaction over the > wire, flushing to disk on the kafka broker(s) and then reading it back wi= ll > be slow. Guessing TPS is in the hundreds, but haven=E2=80=99t measured. Y= ou could > scale a bit by accepting writes at multiple nodes but better to go async > and have multiple threads write to kafka using a single producer (yes, > producers are thread safe) and then correlate with the transaction when i= t > arrives at the consumer. > > > > This async approach using EventStore instead of Kafka, we can easily do > 100K write transactions per second. The EventStore client batches messag= es > on the write side and uses push subscriptions on the read side, so there = is > no extra latency due to polling. Here is the jvm client for EventStore: > https://github.com/EventStore/EventStore.JVM > > > > One trick to avoid stale reads is to pass a sequential version number > with a query. The query will wait for the prevalent system to be at least > the required version before executing. > > > > Note that there are degrees of eventual consistency, and the > inconsistency here is only temporal. Any given state is consistent in > itself, it just might not be the most recent consistent state. > > > > -- > > Robert Friberg > > Building OrigoDB at Devrex Labs > > +46733839080 > > > > From: Ralph Johnson [mailto:[email protected]] > > Sent: den 14 mars 2017 22:13 > > To: Open discussion about the Prevayler project. < > [email protected]> > > Subject: Re: [Prevayler-discussion] Report from using Kafka as > Prevalence journal > > > > i was thinking of doing something similar, i.e. using a messaging syste= m > to keep processes using Prevayler in synch. But the systems I was > looking at would ensure that transactions would arrive in the same order > for all processes. It sounds like Kafka doesn't do that. Or did I not > understand? > > > > I looked up Kafka after reading your message. I've never looked at it > before. It looks well-designed. You say "write speed is crippled". H= ow > many transactions per second can it deliver? > > > > -Ralph Johnson > > > > On Tue, Mar 14, 2017 at 4:02 AM, Karl Wettin <[email protected]> > wrote: > > You chose to allow Open discussion about the Prevayler project. ( > [email protected]) even though this message > failed authentication > > Click to disallow > > This message is eligible for Automatic Cleanup! ([email protected]= ) > Add cleanup rule | More info > > > > Perhaps some of you find this interesting. > > > > > > For a little while now we've been running a Prevalance pattern that use > Kafka as journal to handle multiple load balanced and availability nodes = of > the same data store. Transactions have unique identities and when execute= d > by the user on a local node they are sent to Kafka, waited upon to bounce > back from Kafka and then executed on the local node using a CountDownLatc= h > to make sure the transaction has been distributed. Calls on a local node > are kept in order other using a ConcurrentReadWriteLock. It's very > simplistic, some 200 lines of code. > > > > > > The obvious drawbacks is that: > > > > * Write speed is crippled, bound to the Kafka polling rate. This will o= f > course also affect the read speed at the local node on which the > transaction was created, as the node will be locked down for further > queries and transaction until the waited upon transaction has been receiv= ed > and executed. > > > > * The nodes are out of sync at a level equal to the network latency. > Queries executed on different nodes can for a short time return different > results but there is eventual consistency. One can however often implemen= t > transactions and queries in a manner so that this never occurs (e.g. neve= r > request data that has not been created or is about to be updated), or > simply by accepting this caveat as a system limitation. > > > > For us, none of the above seems to be a problem at this point. > > > > > > Well implemented transactions with validation before broadcasting a > transaction and the same validation before executing the transaction seem= s > to handle when multiple nodes simultaneously attempt to broadcast > incompatible transactions, then only the first to arrive to Kafka will > succeed and the others will receive a validation exception. This does > however mean that the journal might contain transactions that indeed will > fail. > > > > I suppose it would be possible to use Kafka as an inter-node write-lock > if one is OK with adding even more latency, and thus removing a few of th= e > caveats. But I'm not going there voluntarily. > > > > > > All in all it seems to be working great. > > > > So far there are no snapshots, but the idea is to schedule starting up = a > new service node to which no clients will connect, take a snapshot and th= en > shut it down again. > > > > > > k > > ------------------------------------------------------------ > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > _______________________________________________ > > To unsubscribe go to the end of this page: http://lists.sourceforge.net= / > lists/listinfo/prevayler-discussion > > _______________________________________________ > > "Databases in Memoriam" -- http://www.prevayler.org > > > > > > ------------------------------------------------------------ > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot______ > _________________________________________ > > To unsubscribe go to the end of this page: http://lists.sourceforge.net= / > lists/listinfo/prevayler-discussion > > _______________________________________________ > > "Databases in Memoriam" -- http://www.prevayler.org > > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > To unsubscribe go to the end of this page: http://lists.sourceforge.net/ > lists/listinfo/prevayler-discussion > _______________________________________________ > "Databases in Memoriam" -- http://www.prevayler.org > --001a114c6cc437a6cd054bb5ef0d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Sometimes I feel we are like wizards, with our own Arcane = incantations ;)<div><br></div><div>Much thanks for the Docker line though!<= /div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon= , Mar 27, 2017 at 2:52 PM, Karl Wettin <span dir=3D"ltr"><<a href=3D"mai= lto:[email protected]" target=3D"_blank">[email protected]</a>>= ;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex">Kafka doesn't have to= be that tricky, at least not to get started with it:<br> <br> <a href=3D"https://github.com/spotify/docker-kafka/" rel=3D"noreferrer" tar= get=3D"_blank">https://github.com/spotify/<wbr>docker-kafka/</a><br> <br> docker run -p 2181:2181 -p 9092:9092 --rm=C2=A0 --env ADVERTISED_HOST=3D10.= 40.1.52 --env ADVERTISED_PORT=3D9092 --name kafka -h kafka spotify/kafka<br= > <br> // kafka properties:<br> config.put("auto.offset.reset"<wbr>, "earliest");<br> config.put("<a href=3D"http://group.id" rel=3D"noreferrer" target=3D"_= blank">group.id</a>", UUID.randomUUID().toString());<br> <br> That's pretty much all configuration you need.<br> <br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 kalle<br> <span class=3D"im HOEnZb"><br> > On 14 Mar 2017, at 23:42, Robert Friberg <<a href=3D"mailto:robert.= [email protected]">[email protected]</a>> wrote:<br> ><br> > Hi,<br> ><br> > We=E2=80=99ve been using this approach as well with OrigoDB. A key str= ength not mentioned by Karl is that additional consumers of the transaction= stream can generate arbitrary read models, either in memory or on disk, in= real time or on demand.<br> ><br> > Kafka is very mature and popular but it requires a LOT of knowledge to= install, configure and operation. Ask me how I know =F0=9F=98=8A<br> > To understand the kafka model, it=E2=80=99s easy to think of a persist= ent pub/sub message bus. A producer publishes messages to a topic and consu= mers subscribe to topics.<br> ><br> > Yes, message ordering is preserved and the delivery guarantee is at le= ast once, so you should probably have a deduplication step in the pipeline.= <br> ><br> > Kafka is relatively slow if you write a single message using a produce= r and wait for the message using a consumer. You can configure the poll int= erval to be more aggressive but sending a single transaction over the wire,= flushing to disk on the kafka broker(s) and then reading it back will be s= low. Guessing TPS is in the hundreds, but haven=E2=80=99t measured. You cou= ld scale a bit by accepting writes at multiple nodes but better to go async= and have multiple threads write to kafka using a single producer (yes, pro= ducers are thread safe) and then correlate with the transaction when it arr= ives at the consumer.<br> ><br> > This async approach using EventStore instead of Kafka, we can easily d= o 100K write transactions per second. The EventStore=C2=A0 client batches m= essages on the write side and uses push subscriptions on the read side, so = there is no extra latency due to polling. Here is the jvm client for EventS= tore:<a href=3D"https://github.com/EventStore/EventStore.JVM" rel=3D"norefe= rrer" target=3D"_blank">https://github.com/<wbr>EventStore/EventStore.JVM</= a><br> ><br> > One trick to avoid stale reads is to pass a sequential version number = with a query. The query will wait for the prevalent system to be at least t= he required version before executing.<br> ><br> > Note that there are degrees of eventual consistency, and the inconsist= ency here is only temporal. Any given state is consistent in itself, it jus= t might not be the most recent consistent state.<br> ><br> > --<br> > Robert Friberg<br> > Building OrigoDB at Devrex Labs<br> > <a href=3D"tel:%2B46733839080" value=3D"+46733839080">+46733839080</a>= <br> ><br> > From: Ralph Johnson [mailto:<a href=3D"mailto:[email protected]">joh= [email protected]</a>]<br> > Sent: den 14 mars 2017 22:13<br> > To: Open discussion about the Prevayler project. <<a href=3D"mailto= :[email protected]">prevayler-discussion@lists.<wb= r>sourceforge.net</a>><br> > Subject: Re: [Prevayler-discussion] Report from using Kafka as Prevale= nce journal<br> ><br> > i was thinking of doing something similar, i.e. using a messaging syst= em to keep processes using Prevayler in synch.=C2=A0 =C2=A0 But the systems= I was looking at would ensure that transactions would arrive in the same o= rder for all processes.=C2=A0 =C2=A0It sounds like Kafka doesn't do tha= t.=C2=A0 Or did I not understand?<br> ><br> > I looked up Kafka after reading your message.=C2=A0 I've never loo= ked at it before.=C2=A0 =C2=A0It looks well-designed.=C2=A0 =C2=A0You say &= quot;write speed is crippled".=C2=A0 How many transactions per second = can it deliver?<br> ><br> > -Ralph Johnson<br> ><br> > On Tue, Mar 14, 2017 at 4:02 AM, Karl Wettin <<a href=3D"mailto:kar= [email protected]">[email protected]</a>> wrote:<br> </span><div class=3D"HOEnZb"><div class=3D"h5">> You chose to allow Open= discussion about the Prevayler project. (<a href=3D"mailto:prevayler-discu= [email protected]">prevayler-discussion@lists.<wbr>sourceforge.ne= t</a>) even though this message failed authentication<br> > Click to disallow<br> > This message is eligible for Automatic Cleanup! (<a href=3D"mailto:kar= [email protected]">[email protected]</a>) Add cleanup rule | More in= fo<br> ><br> > Perhaps some of you find this interesting.<br> ><br> ><br> > For a little while now we've been running a Prevalance pattern tha= t use Kafka as journal to handle multiple load balanced and availability no= des of the same data store. Transactions have unique identities and when ex= ecuted by the user on a local node they are sent to Kafka, waited upon to b= ounce back from Kafka and then executed on the local node using a CountDown= Latch to make sure the transaction has been distributed. Calls on a local n= ode are kept in order other using a ConcurrentReadWriteLock. It's very = simplistic, some 200 lines of code.<br> ><br> ><br> > The obvious drawbacks is that:<br> ><br> > * Write speed is crippled, bound to the Kafka polling rate. This will = of course also affect the read speed at the local node on which the transac= tion was created, as the node will be locked down for further queries and t= ransaction until the waited upon transaction has been received and executed= .<br> ><br> > * The nodes are out of sync at a level equal to the network latency. Q= ueries executed on different nodes can for a short time return different re= sults but there is eventual consistency. One can however often implement tr= ansactions and queries in a manner so that this never occurs (e.g. never re= quest data that has not been created or is about to be updated), or simply = by accepting this caveat as a system limitation.<br> ><br> > For us, none of the above seems to be a problem at this point.<br> ><br> ><br> > Well implemented transactions with validation before broadcasting a tr= ansaction and the same validation before executing the transaction seems to= handle when multiple nodes simultaneously attempt to broadcast incompatibl= e transactions, then only the first to arrive to Kafka will succeed and the= others will receive a validation exception. This does however mean that th= e journal might contain transactions that indeed will fail.<br> ><br> > I suppose it would be possible to use Kafka as an inter-node write-loc= k if one is OK with adding even more latency, and thus removing a few of th= e caveats. But I'm not going there voluntarily.<br> ><br> ><br> > All in all it seems to be working great.<br> ><br> > So far there are no snapshots, but the idea is to schedule starting up= a new service node to which no clients will connect, take a snapshot and t= hen shut it down again.<br> ><br> ><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0k<br> > ------------------------------<wbr>------------------------------<wbr>= ------------------<br> > Check out the vibrant tech community on one of the world's most<br= > > engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot= " rel=3D"noreferrer" target=3D"_blank">http://sdm.link/slashdot</a><br> > ______________________________<wbr>_________________<br> > To unsubscribe go to the end of this page: <a href=3D"http://lists.sou= rceforge.net/lists/listinfo/prevayler-discussion" rel=3D"noreferrer" target= =3D"_blank">http://lists.sourceforge.net/<wbr>lists/listinfo/prevayler-<wbr= >discussion</a><br> > ______________________________<wbr>_________________<br> > "Databases in Memoriam" -- <a href=3D"http://www.prevayler.o= rg" rel=3D"noreferrer" target=3D"_blank">http://www.prevayler.org</a><br> ><br> ><br> > ------------------------------<wbr>------------------------------<wbr>= ------------------<br> > Check out the vibrant tech community on one of the world's most<br= > > engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot= _______________________________________________" rel=3D"noreferrer" target= =3D"_blank">http://sdm.link/slashdot______<wbr>____________________________= __<wbr>___________</a><br> > To unsubscribe go to the end of this page: <a href=3D"http://lists.sou= rceforge.net/lists/listinfo/prevayler-discussion" rel=3D"noreferrer" target= =3D"_blank">http://lists.sourceforge.net/<wbr>lists/listinfo/prevayler-<wbr= >discussion</a><br> > ______________________________<wbr>_________________<br> > "Databases in Memoriam" -- <a href=3D"http://www.prevayler.o= rg" rel=3D"noreferrer" target=3D"_blank">http://www.prevayler.org</a><br> <br> <br> ------------------------------<wbr>------------------------------<wbr>-----= -------------<br> Check out the vibrant tech community on one of the world's most<br> engaging tech sites, Slashdot.org! <a href=3D"http://sdm.link/slashdot" rel= =3D"noreferrer" target=3D"_blank">http://sdm.link/slashdot</a><br> ______________________________<wbr>_________________<br> To unsubscribe go to the end of this page: <a href=3D"http://lists.sourcefo= rge.net/lists/listinfo/prevayler-discussion" rel=3D"noreferrer" target=3D"_= blank">http://lists.sourceforge.net/<wbr>lists/listinfo/prevayler-<wbr>disc= ussion</a><br> ______________________________<wbr>_________________<br> "Databases in Memoriam" -- <a href=3D"http://www.prevayler.org" r= el=3D"noreferrer" target=3D"_blank">http://www.prevayler.org</a><br> </div></div></blockquote></div><br></div> --001a114c6cc437a6cd054bb5ef0d-- --===============6259968846844080697== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot --===============6259968846844080697== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion _______________________________________________ "Databases in Memoriam" -- http://www.prevayler.org --===============6259968846844080697==--