Prevayler 3
"Flavio W. Brasil" <[email protected]> Sat, 11 Mar 2017 23:23:24 -0800
| Newsgroups | gmane.comp.java.prevayler |
|---|---|
| Message-ID | <[email protected]> |
--===============0372784688785776358== Content-Type: multipart/alternative; boundary="----=_NextPart_45411499.235038689783" ------=_NextPart_45411499.235038689783 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi! I've asked Klaus on Twitter if there are plans for a new major version of P= revayler and it seems that there's some interest! :) https://twitter.com/klauswuestefeld/status/840698354585763840 [https://twit= ter.com/klauswuestefeld/status/840698354585763840] I don't know if you already have are plans and ideas for it, but I'd like t= o start the discussion with some thoughts: Memory mapped files A common approach to deal with persistence and in-memory data nowadays is u= sing=C2=A0memory mapped files [http://howtodoinjava.com/java-7/nio/java-nio= -2-0-memory-mapped-files-mappedbytebuffer-tutorial/]. They have interesting= =C2=A0characteristics: - The operating system is responsible for managing the page loads from the = disk, and automatically handles page faults in an optimized manner - The OS does the syncing to disk, so even if the JVM dies, the contents ar= e sill flushed to disk. It doesn't guarantee consistency if the machine/OS = dies, though. Most libs consider that the OS/machine won't fail and claim t= hat writing to the memory mapped file is enough for consistency. - =C2=A0The memory mapped files can be used as an in-memory cache that avoi= ds garbage collection pressure since the data is kept off-heap. Serialization Maybe we could use protobuf to define the prevalent system and manage it wi= th the=C2=A0protostuff [http://www.protostuff.io/]=C2=A0lib. It has much be= tter performance for serialization/deserialization and handles schema migra= tion. It'd introduce more moving pieces and make using Prevayler less strai= ghtforward, so I'm not sure it's worth it. If we don't use protobuf, we could make kryo the default serialization, tha= t is faster than java's serialization. I think it'd be important to build a= mechanism to help the user with migrations as well. Finer lock granularity I imagine most transactions touch only a handful values of the prevalent sy= stem. We could try to build a mechanism that doesn't lock the entire preval= ent system to run transactions. Some ideas: - Create a sub-system concept that allows the user to isolate transactions = to specific parts of the prevalent system. - Read the transaction bytecode to determine the fields used by the transac= tion (including "transitive" usages if it calls methods that mutate fields)= , and use the information to lock only parts of the system. - Implement a=C2=A0Software Transactional Memory [https://en.wikipedia.org/= wiki/Software_transactional_memory]=C2=A0(STM) to handle mutations and conc= urrency at the field level. It's considerably more complex,=C2=A0but it'd m= ake it much easier to scale vertically with prevayler. Also, rollbacks woul= d be supported out-of-the-box. Log only mutations Instead of serializing the transaction and replaying it for recovery, we co= uld try to isolate only the actual mutations that it performs. It'd require= a way to isolate changes, maybe using bytecode manipulation or even the ST= M log if we decide to use it. It could produce larger log files since trans= actions can mutate more values that the size of its inputs, but it has some= benefits: - Transactions wouldn't be serialized, allowing the users to change their i= mplementations without worrying about schema evolution and how they'll beha= ve for recovery. - Recovering from the transaction log would be faster since it wouldn't exe= cute the user code, only apply the mutations. Distributed prevalent system We could use=C2=A0chronicle [https://github.com/OpenHFT/Chronicle-Queue]=C2= =A0to implement a low-latency distributed prevalent system. Also, we could = provide some sort of sharding mechanism to allow the users to expand a prev= alent system without having to have machines/JVMs that can hold the entire = dataset in memory. Conclusion I'm leaning more to a solution using STM + protobuf. We could customize the= protobuf code generation to efficiently track the STM state and even read/= write directly to memory mapped files, avoiding the JVM heap entirely.=C2= =A0I haven't given much thought to it, so please feel free to say if this i= s all nonsense :) Maybe it should be a new open source project, not a Preva= yler 3. Best, Flavio ------=_NextPart_45411499.235038689783 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <div id=3D"__MailbirdStyleContent" style=3D"font-size: 10pt;font-family: ar= ial;color: #000000"><div>Hi!</div><div><br></div><div>I've asked Klaus on T= witter if there are plans for a new major version of Prevayler and it seems= that there's some interest! :)</div><div><br></div><div><a href=3D"https:/= /twitter.com/klauswuestefeld/status/840698354585763840">https://twitter.com= /klauswuestefeld/status/840698354585763840</a></div><div><br></div><div>I d= on't know if you already have are plans and ideas for it, but I'd like to s= tart the discussion with some thoughts:</div><div><br></div><div><b>Memory = mapped files</b></div><div><br></div><div>A common approach to deal with pe= rsistence and in-memory data nowadays is using <a href=3D"http://howto= doinjava.com/java-7/nio/java-nio-2-0-memory-mapped-files-mappedbytebuffer-t= utorial/"><span style=3D"font-size: 10pt;color: rgb(0, 0, 0)">memory mapped= files</span></a>. They have interesting characteristics:</div><div><b= r></div><div>- The operating system is responsible for managing the page lo= ads from the disk, and automatically handles page faults in an optimized ma= nner</div><div>- The OS does the syncing to disk, so even if the JVM dies, = the contents are sill flushed to disk. It doesn't guarantee consistency if = the machine/OS dies, though. Most libs consider that the OS/machine won't f= ail and claim that writing to the memory mapped file is enough for consiste= ncy.</div><div>- The memory mapped files can be used as an in-memory = cache that avoids garbage collection pressure since the data is kept off-he= ap.</div><div><br></div><div><b>Serialization</b></div><div><br></div><div>= Maybe we could use protobuf to define the prevalent system and manage it wi= th the <a href=3D"http://www.protostuff.io/"><span style=3D"font-size:= 10pt;color: rgb(0, 0, 0)">protostuff</span></a> lib. It has much bett= er performance for serialization/deserialization and handles schema migrati= on. It'd introduce more moving pieces and make using Prevayler less straigh= tforward, so I'm not sure it's worth it.</div><div><br></div><div>If we don= 't use protobuf, we could make kryo the default serialization, that is fast= er than java's serialization. I think it'd be important to build a mechanis= m to help the user with migrations as well.</div><div><br></div><div><b>Fin= er lock granularity</b></div><div><br></div><div>I imagine most transaction= s touch only a handful values of the prevalent system. We could try to buil= d a mechanism that doesn't lock the entire prevalent system to run transact= ions. Some ideas:</div><div><br></div><div>- Create a sub-system concept th= at allows the user to isolate transactions to specific parts of the prevale= nt system.</div><div><br></div><div>- Read the transaction bytecode to dete= rmine the fields used by the transaction (including "transitive" usages if = it calls methods that mutate fields), and use the information to lock only = parts of the system.</div><div><br></div><div>- Implement a <a href=3D= "https://en.wikipedia.org/wiki/Software_transactional_memory"><span style= =3D"font-size: 10pt;color: rgb(0, 0, 0)">Software Transactional Memory</spa= n></a> (STM) to handle mutations and concurrency at the field level. I= t's considerably more complex, but it'd make it much easier to scale v= ertically with prevayler. Also, rollbacks would be supported out-of-the-box= .</div><div><br></div><div><b>Log only mutations</b></div><div><br></div><d= iv>Instead of serializing the transaction and replaying it for recovery, we= could try to isolate only the actual mutations that it performs. It'd requ= ire a way to isolate changes, maybe using bytecode manipulation or even the= STM log if we decide to use it. It could produce larger log files since tr= ansactions can mutate more values that the size of its inputs, but it has s= ome benefits:</div><div><br></div><div>- Transactions wouldn't be serialize= d, allowing the users to change their implementations without worrying abou= t schema evolution and how they'll behave for recovery.</div><div>- Recover= ing from the transaction log would be faster since it wouldn't execute the = user code, only apply the mutations.</div><div><br></div><div><b>Distribute= d prevalent system</b></div><div><br></div><div>We could use <a href= =3D"https://github.com/OpenHFT/Chronicle-Queue"><span style=3D"font-size: 1= 0pt;color: rgb(0, 0, 0)">chronicle</span></a> to implement a low-laten= cy distributed prevalent system. Also, we could provide some sort of shardi= ng mechanism to allow the users to expand a prevalent system without having= to have machines/JVMs that can hold the entire dataset in memory.</div><di= v><br></div><div><b>Conclusion</b></div><div><br></div><div>I'm leaning mor= e to a solution using STM + protobuf. We could customize the protobuf code = generation to efficiently track the STM state and even read/write directly = to memory mapped files, avoiding the JVM heap entirely. <span style=3D= "font-size: 10pt;line-height: 1.5">I haven't given much thought to it, so p= lease feel free to say if this is all nonsense :) Maybe it should be a new = open source project, not a Prevayler 3.</span></div><div><span style=3D"fon= t-size: 10pt;line-height: 1.5"><br></span></div><div><span style=3D"font-si= ze: 10pt;line-height: 1.5">Best,</span></div><div><span style=3D"font-size:= 10pt;line-height: 1.5"><br></span></div><div><span style=3D"font-size: 10p= t;line-height: 1.5">Flavio</span></div><div class=3D"mb_sig"></div></div> ------=_NextPart_45411499.235038689783-- --===============0372784688785776358== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Announcing the Oxford Dictionaries API! The API offers world-renowned dictionary content that is easy and intuitive to access. Sign up for an account today to start using our lexical data to power your apps and projects. Get started today and enter our developer competition. http://sdm.link/oxford --===============0372784688785776358== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion _______________________________________________ "Databases in Memoriam" -- http://www.prevayler.org --===============0372784688785776358==--