Re: [jgroups-dev] [javagroups-users] Scope of JGroups compared to Corosync

Hi,

On 3/25/10, Bela Ban <[email protected]> wrote:
>
>
> Alexander T wrote:
>>> This is the first time I've heard about Corosync, I'm vaguely
>>> familiar with the OpenAIS effort.
>>
>> Seems like they just separated OpenAIS into interface and impl. Impl
>> is corosync, and interface+impl is called OpenAIS. That is what my
>> first impression is anyway.
>>
>>>> It seems to me that the scope of the two products is very similar.
>>> Yes, although I'd have to look at OpenAIS / Corosync in more detail.
>>> OpenAIS (and Corosync) are written in C, therefore platform dependent,
>>> compared to JGroups in Java.
>>
>> True. Although I don't really think that is a major. I think that
>> argument risks undermining the real benefits of a good product like
>> JGroups: that it works better! But that's just my 2 cents :)
>
> Well, I do think that being in Java and as such platform independent is
> a big plus ! If you write your impl in C (like Spread or Corosync), then
> you either need some JNI code, or a pipe with a protocol and a separate
> daemon, to provide Java clients, and that's a hassle. JNI is a hassle
> because it involves copying of data between the C and Java heaps (slow)
> and a separate demon is an issue because it requires ad additional
> process to manager, configure, restart and so on.

I'm of the opinion that a lot of Java software goes unnoticed because
they profile themselves as 'X but reimplemented in Java', whereas it
would be much better to say 'X but with these additional features:'.
In fact, Java has a bit of a 'not invented here' thing going on, and
most people outside of Java circles frown upon the community use this
argument to discard Java software as Java-only solutions reinventing
the wheel. I don't know the programming language used for most of the
services (libs, deamons) I use. I only know that it provides me with a
Service I need, and I think this should remain the most important
selling point.

It is of course easier to integrate if you are using the same language
as the service self, but I don't think that should be the main selling
point.

I see where you're coming from though from reading you intro in the
docs, but I would rather try to focus on the fact that it is a good
product, also for non-java clients!

>
>
>>>> Are there any integration plans, or are these products in fact
>>>> complimentary?
>>> No, there are no plans. Looks like OpenAIS is a totem protocol
>>> implementation, so it is closer to Spread than JGroups...
>>
>> Hmm, I got the impression that some sort of ring protocol was
>> necessary in order to guarantee virtual synchrony.
>
> As a matter of fact, a totem ring protocol is used to provide *total
> order*, *not* virtual synchrony. It is simple though to implement
> virtual synchrony on top of a totem ring.

So having a total order makes it easy to figure out the right message
dropping and get virtual synchrony?

>
>> I therefore assumed that JGroups contained totem or some sort of ring
>> protocol if you turn on FLUSH.
>
> No. JGroups with FLUSH provides virtual synchrony, but not total
> ordering. If you want total ordering, use SEQUENCER, which uses a
> sequencer based approach to total ordering. I do like the totem
> protocol, but it requires constant token rotation, and IMO is not
> scalable as the time the token spends on each member is a function of
> the ring size.

So FLUSH can provide virtual synchrony without having total order as a
pre-requisite? That's cool.

>
>> I know that JGroups is much more flexible than that, but
>> could you elaborate on what you mean when you say that JGroups doesn't
>> use totem?
>
> We use a sequencer based approach to provide total ordering. I might
> provide the totem protocol at some point in the future (we used to have
> an impl a long time ago)...
>
>> I gave up on spread because it seems to require a ring protocol which
>> is not scalable and uses a 'many agents per deamon' model which
>> doesn't scale to many deamons, so there is a pretty hard limit on the
>> max number of machines that you can have with spread.
>
> OK
>
>> I've had much
>> better luck with JGroups there. But I never tried scaling the JGroups
>> FLUSH, do you think (know) it will scale better than Spread?
>
> Depends on what you compare it with. A totem based protocol will
> certainly scale better than a sequencer based protocol if you want total
> ordering. OTOH, I haven't yet found one application that requires total
> ordering. FIFO or causal is good enough in most cases.
>
> JGroups and FLUSH will *NOT* scale ! It is in the nature of virtual
> synchrony that it doesn't scale to (say) 20 members. Ken Birman has
> written a bunch of papers on that, google's your friend. FLUSH is like a
> 2PC protocol, run before every view change.

Now I'm confused. You say that you didn't chose for totem since it
doesn't scale, yet you say that virtual synchrony (and thereby FLUSH)
intrinsically doesn't scale? What is the benefit from using SEQUENCER
then? Am I interpreting this correctly as:

* Total order through sequencer performs better than ring-based total
order (more resilient to partial failures (lost token) and not
sensitive to ring roundtrip times etc)
but
* If you want virtual synchrony you are equally bad off with JGroups
FLUSH as with virtual synchrony above a ring (i.e Spread). This is a
fundamental limitation in this concept due to the fact that you need
to await confirmation from all members before proceeding.

You also mention that you only run 2PC synch on view change. How does
a member know that it hasn't already missed a view change f it
progresses without participating in a synch? I'm asking since I expect
infrequent but big view changes, perhaps it will scale reasonably then
anyhow? Does synching take O(n^2) messages or more?

>
>
>>>> If similar, is JGroups more mature that Corosync?
>>> I hope so ! :-) JGroups has been around for 12 years, whereas OpenAIS /
>>> Corosync for 2-4 years (?)
>>
>> I don't know, found some info indicating that they started in 2002,
>> virtual synchrony in 2004.
>>
>>>> I hope that I'm not stepping on anyones toes with this question.
>>> No, sure not. At least not on mine :-)
>>
>> Great. You never know :)
>>
>>>> Information and comparisons between group messaging systems is just
>>>> very scarce on the inet.
>>> Do you volunteer ? :-)
>>
>> Absolutely. I just need to get some experience first, then I will
>> start writing a bit about it I think. Funny that there is not more
>> data available, now when distributed systems are the latest buzzword.
>
> Good, let me know if you need more info,
> cheers,

Always nice to know more. I have googled some and got some books as
well, but still takes time to get into the game.

>
> --
> Bela Ban
> Lead JGroups / Clustering Team
> JBoss
>

Best Regards,

Alexander

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Javagroups-development mailing list