Re: [jgroups-dev] Muti node crash

On 25/03/15 02:42, Sundararajan, Mukund wrote:
> All,
>
> We are facing a unique problem. I have a 30 node cluster, 15 nodes in
> one data center and the rest in another.

Using RELAY2 ?

> The coordinator is in the second data center.

Ah, ok, not RELAY2 then... what's your config ?

> When the first data center loses power, thereby
> resulting in the abnormal shutdown of the 15 nodes, the following happens  -

Note that FD_HOST [1] might come in handy here, to detect crashes of 
multiple nodes on a box faster.


> 1.The coordinator puts out several view changes, seconds apart from each
> other. The total number of view changes is 15. This is good.

If you use FD_ALL, you should not get 15 view changes (1 for each 
member), but rather 1-2 view changes. If you use FD, then - yes - you'll 
probably get 15 view changes.

> 2.While the view changes are being broadcasted, MERGE2 on the
> coordinator starts finding all views.

Use MERGE3; it handles more edge cases and MERGE2 has been deprecated 
some time ago.

What do you mean by this ? Is the first data center restarted and 
merging between the 2nd and first DC are starting to happen ?

> This sends a GET_MBRS_REQ to go out. This goes out with View ID = X.
>
> 3.The coordinator gets a response to GET_MBRS_REQ from all members.
> However different members report different views. By the time the
> response reaches the coordinator, it itself has installed a new View Y
> because the view changes as a result of step 1 are happening in parallel.
>
> 4.The coordinator then initiates a merge. There are four sub groups to
> merge. All four sub groups have the same coordinator ( just different
> view IDs). As a net result the coordinator ends up creating a final view
> with just one member, itself. It never recovers from this condition.

Try MERGE3; asymmetric merges are not handled well by MERGE2.

> Does anyone have any idea why GET_MBRS_REQ triggered by MERGE2 and the
> view changes are not synchronized? Is my configuration missing
> something? Why does not the coordinator recover, rediscover members and
> install and full view of 15 members after it incorrectly merges with
> itself in step 4 and reduces the view size to 1?
>
> Thanks
>
> Mukund

[1] http://www.jgroups.org/manual/index.html#FD_HOST

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Javagroups-development mailing list