Re: ACIDity of Prevayler 2.6

PS. We have implemented an sql backing store for #liveDB, which helps keep the
slow adopters happy. 

Robert Friberg
http://about.me/robertfriberg

From: Rick Ross [mailto:[email protected]] 
Sent: den 1 december 2012 23:43
To: Open discussion about the Prevayler project.
Subject: Re: [Prevayler-discussion] ACIDity of Prevayler 2.6

"It seems very demanding on the prevalent system code to require it to be rollbackable."

Well, yes.  That would be the point.   I'm sure that standard DBs would much faster if they didn't have to conform to ACID rules, but the ACID rules aren't there to slow down DBs, they are there to ensure data integrity.   Slowing down Prevayler to achieve the same levels of data integrity that DBs offer (and lord knows they are not bulletproof) would make an important statement about Prevayler as a serious substitute.   

I spent well over six months building my model in prevayler.  It is a deeply complex model and very difficult to construct in standard DBs.   Of course, as objects in a prevalent system, it was much simpler and indeed faster.   Sadly, that product did not make it to market and a portion of the reason was that the decision makers were not comfortable with an entirely new data system that did not have the same level of commitment to data protection that RDBMSs claim.  

I'm not trying to throw firebombs here, but rather point out the things that matter when it comes to adoption.

R


On Dec 1, 2012, at 1:36 PM, Klaus Wuestefeld <[email protected]> wrote:


It seems very demanding on the prevalent system code to require it to be rollbackable.

I like the idea of a meaningful prevalentSystem.hashCode() very much. It is useful for checking schema migrations too.

Klaus

On Fri, Nov 23, 2012 at 7:46 AM, Arnaud Masson <[email protected]> wrote:
Hi

I am currently using this alternate pattern for a system that has many
reads and few writes, with strong consistency requirements.
Any single-point-in-time snapshot of the journal (at disk level) is correct.


prepareCommandOnJournal(command); // minimize IO during lock scope

writeLock.lock();
try {

  try {

    if (command.prepareExecution()) {

      oldHash = model.hashCode();
      command.execute();

      commitCommandToJournal(command); // might cause some latency
                                       // in concurrent queries
    }
   } catch (Exception re) { // execution or journal IO problem

     try {
       command.rollback(); //Maybe it would be easier to just reload
                           //whole system, throw some Exception

       // optional: check that rollback has really restored the old state
       int newHash = model.hashCode();
       if (newHash != oldHash)
           throw RuntimeException("Rollback was not correct", re);

       // no need to fix the journal

     } catch (Exception fatalEx) {
       System.exit(-1); //Or another way to reload the whole system
     }
   }
}
finaly {

   writeLock.unlock();
}



On 23/11/2012 03:42, Alexandre Nodari wrote:
> Uau!
>
> Considering from past e-mails:
>
> - Atomicity and Consistency have some overlap
> - When we say Exceptions, we are usually talking about unexpected
> events. Normally either the Commands were written without throwing
> Exceptions or there are expected Exceptions that are catched and dealt
> within the Command.
> - The Royal Food Taster provides Rollback with 2 steps:
>     - Avoiding a Command that threw an Exception (unexpected event) to
> be written to the journal
>     - Avoiding partial changes by discarding temporary data used to
> check the Commands
>
> We have then two grades (or levels) of Atomicity+Consistency:
>
> A) Without Royal Food Taster: Exceptions (expected and unexpected)
> happen and we let the business code deal with them. Advantages: simple
> and fast. Disadvantage: being some Exceptions unexpected events, maybe
> it is not possible to guarantee that they are deterministic, and not
> being deterministic they can really cause trouble in the system based on
> Prevaylers definition.
>
> B) With Royal Food Taster: Exceptions (unexpected) cause Rollback.
> Advantage: "better" Atomicity+Consistency. Disadvantages: less
> performance and more memory needed.
>
> My opinion:
>
> I think the choice should be given to the developers, offering both options.
>
> And I like the idea of trying to do better than Royal Food Taster. I
> will give a draft of another idea of Rollback to help with the
> discussion, trying to address the 2 steps described above:
>
> Another type of Command called RollbackableCommand with 3 methods:
>
> boolean prepareExecution(); // Prepare to execute, validate
> pre-conditions, save data for rollback
> void execute(); //Execute the changes, catch and deal with all expected
> Exceptions
> void rollback(); //Rollback possible changes
>
> And Prevayler would do something like this:
>
> try {
>     if (command.prepareExection()) {
>        writeCommandToJournal();
>        command.execute();
>     }
> } catch (RuntimeException re) {
>     try {
>        removeCommandFromJournal();
>        command.rollback(); //Maybe it would be easier to just reload
> whole system, throw some Exception
>     } catch (Exception e) {
>        System.exit(-1); //Or another way to reload the whole system
>     }
> }
>
> Did not think about all the caveats possible. Just beginning a new idea.
> Probably for Prevayler 3 :-)
>
> See you,
>
>
> 2012/11/17 Paul Bennett <[email protected] <mailto:[email protected]>>
>
>     Justin, thanks very much for such clear explanation. Some comments
>     below.
>
>     On Nov 16, 2012, at 2:55 AM, Justin T. Sampson wrote:
>
>>     It's exciting to see such a passionate discussion. :) And also
>>     good that it's coming around to some more practical questions.
>>
>>     First of all, the ACID properties are ill-defined, incomplete, and
>>     redundant, so ultimately we shouldn't take them too seriously.
>>     Yes, A and C have some overlap, as do A and I, C and D, etc. They
>>     are a nice set of reminders of things to talk about regarding
>>     transactions in a persistence layer, though. And I still maintain
>>     that rollback is just one potential mechanism to partially address
>>     some of these properties, not a fundamental requirement for any of
>>     them.
>
>     Ah, OK - then we are in violent agreement :-). I misunderstood your
>     (and Klaus's) previous statements to mean that rollback was always
>     irrelevant, and never ever useful. That was what I was arguing against.
>
>>
>>     As for journaling transactions after executing them, that would
>>     cause queries to have the same latency as transactions, which is a
>>     pretty big departure from Prevayler's normal performance profile.
>>     It would hardly even be Prevayler anymore. ;)
>>
>>     Paul wrote:
>>
>>     > Klaus, you haven't answered my question. As far as I can see,
>>     > without something to maintain consistency in the face of an
>>     > unexpected exceptions, once that happens, the system is
>>     > inconsistent and cannot proceed *for all users*. You must
>>     > either a) shut down and fix the bug or b) remove the txn from
>>     > the journal.
>>     >
>>     > So unlike an ordinary db, which would isolate the exception
>>     > to a single user, and let others continue to use the
>>     > non-problemmatic parts of the system, an exception kills the
>>     > whole thing. That's unacceptable to me.
>>
>>     Why does it kill the whole thing?
>
>     Yes, OK, that was a little extreme :-) What I was really trying to
>     say was the above - that rollback can be useful in dealing with
>     certain cases (unpredictable but repeatable exceptions thrown
>     because of coding errors), in a certain context. In our system,
>     (which is different from a (normal, conventional ?) prevalent system
>     where transactions and queries are written as specific classes), it
>     is pretty easy to generically generate the inverse of each
>     transaction, which makes implementing rollback on exceptions very
>     do-able. The user has the same experience as a 'regular' database
>     system. But - as you say above - it is not a complete solution for
>     handling inconsistency due to any and all errors. It just reduces
>     the number of situations where some kind of outside intervention is
>     needed.
>
>>
>>     The point that Klaus and I are making is really that you have to
>>     worry about the kind of situation you're describing even if you do
>>     have a rollback mechanism. I've seen plenty of bugs in relational
>>     database applications that caused inconsistent data without
>>     throwing exceptions. At some point a user notices something wonky,
>>     or some feature starts breaking or throwing exceptions later due
>>     to the bad data.
>
>     Agreed. I was not arguing against that - apologies if I was not clear
>
>>
>>     When such inconsistencies happen, it doesn't typically mean that
>>     the system is suddenly unusable for everyone. Maybe one entity is
>>     inaccessible, or some calculations display incorrectly, or one
>>     user can't log in. Bugs that take down an entire application are
>>     pretty rare, in my experience, but of course they do occasionally
>>     happen -- again, even in systems with rollbacks.
>>
>>     I haven't so far addressed the question of what to do in a
>>     Prevayler-based application when bad data creeps in; I've just
>>     clarified that you need to be prepared for bad data regardless of
>>     rollbacks. In the worst case, whatever your persistence layer,
>>     there's always the option to restart the system from a nightly
>>     backup. For smaller fixes, a relational database does provide the
>>     convenience of a built-in scripting language (some variant of SQL)
>>     with an interactive interpreter; in a similar vein, some Prevayler
>>     users have talked about using something like BeanShell or some
>>     other JVM-based scripting language as a superuser feature in an
>>     application, allowing an arbitrary script to be submitted as a
>>     transaction in a running system. The purer approach would be to
>>     write a new fix-up transaction in Java (possibly even with unit
>>     tests!) and either redeploy or hotswap the app in order to execute
>>     it on the running system.
>
>     That's very useful, and a few things I hadn't thought of. Thanks!
>
>>
>>     As for the history of dropping the food taster, it happened in
>>     this thread last November, with not nearly as much excitement as
>>     in our present thread:
>>
>>     http://sourceforge.net/mailarchive/forum.php?thread_name=CAGs7EcU_X1%2BTE3O-CjCje2gsSoxxjnGTm8UZWb6ZkORE7%3Do77A%40mail.gmail.com&forum_name=prevayler-discussion
>>
>>     ------------------------------------------------------------------------------
>>     Monitor your physical, virtual and cloud infrastructure from a single
>>     web console. Get in-depth insight into apps, servers, databases,
>>     vmware,
>>     SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>>     Pricing starts from $795 for 25 servers or applications!
>>     http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>>     To unsubscribe go to the end of this page:
>>     http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>>     _______________________________________________
>>     "Databases in Memoriam" -- http://www.prevayler.org
>
>     -pb
>
>
>
>     ------------------------------------------------------------------------------
>     Monitor your physical, virtual and cloud infrastructure from a single
>     web console. Get in-depth insight into apps, servers, databases, vmware,
>     SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>     Pricing starts from $795 for 25 servers or applications!
>     http://p.sf.net/sfu/zoho_dev2dev_nov
>     _______________________________________________
>     To unsubscribe go to the end of this page:
>     http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
>     _______________________________________________
>     "Databases in Memoriam" -- http://www.prevayler.org
>
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
>
>
>
> _______________________________________________
> To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
> _______________________________________________
> "Databases in Memoriam" -- http://www.prevayler.org
>


------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org




-- 
Valeu, Klaus.
------------------------------------------------------------------------------
Keep yourself connected to Go Parallel: 
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org



------------------------------------------------------------------------------
Keep yourself connected to Go Parallel: 
BUILD Helping you discover the best ways to construct your parallel projects.
http://goparallel.sourceforge.net
_______________________________________________
To unsubscribe go to the end of this page: http://lists.sourceforge.net/lists/listinfo/prevayler-discussion
_______________________________________________
"Databases in Memoriam" -- http://www.prevayler.org