Re: [jgroups-users] JGroups failing to give up on crashed coordinator
Questions/problems related to using JGroups <[email protected]> Mon, 12 Dec 2016 16:19:20 -0700
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <CAPdvCBTLdL+NK_tSESfa9kPc7MAVj+GjaoH1ZfOhQ1Q2GJ2UrA@mail.gmail.com> |
Hi Bela,
We made some improvements to the network and hardware and things have been
much more reliable. 3.6.11.Final seemed to really help with some of the
merging problems that we were still having. However we still manage to get
a condition where a node just won't give up on a view with a coordinator
who has moved on to a new view which does not include it. In this case the
coordinator (node_1) is installing a new view that does not include node_9
which it rejects and then it seems to be stuck forever (or at least way
longer than I'm waiting for it to fix itself). Here is a log from node_9
with some commentary inline:
12-09 21:03:15.900 I/MessagingManager( 1869): Configuring JGroups to use
address /fd00:0:b1:3:219:0:b801:ed26%6%6 on interface wlan0
12-09 21:03:15.900 I/MessagingManager( 1869): Configuring JGroups to use
multicast port 53436
12-09 21:03:15.960 I/LargestWinningPolicy( 1869): Initialized.
12-09 21:03:16.020 D/Configurator( 1869): set property UDP.mcast_group_addr
to default value /ff0e::8:8:8
12-09 21:03:16.020 D/Configurator( 1869): set property UDP.diagnostics_addr
to default value /ff0e::75:75
12-09 21:03:16.610 D/UDP ( 1869): sockets will use interface
fd00:0:b1:3:219:0:b801:ed26%6
12-09 21:03:16.650 D/UDP ( 1869): socket information:
12-09 21:03:16.650 D/UDP ( 1869): mcast_addr=ff0e::8:8:8:53436,
bind_addr=/fd00:0:b1:3:219:0:b801:ed26%6%6, ttl=8
12-09 21:03:16.650 D/UDP ( 1869): sock: bound to
fd00:0:b1:3:219:0:b801:ed26:45083, receive buffer size=522240, send buffer
size=522240
12-09 21:03:16.650 D/UDP ( 1869): mcast_sock: bound to
fd00:0:b1:3:219:0:b801:ed26%6:53436, send buffer size=522240, receive
buffer size=522240
12-09 21:03:16.680 W/UDP ( 1869): failed to join /ff0e::75:75:7500 on
eth0: java.net.SocketException: setsockopt failed: EADDRNOTAVAIL (Cannot
assign requested address)
12-09 21:03:16.710 W/MessagingManager( 1869): Node 9 is blocked.
12-09 21:03:16.710 W/MessagingManager( 1869): Node 9 is blocked.
12-09 21:03:16.710 I/System.out( 1869): ------------------------------
-------------------------------------
12-09 21:03:16.710 I/System.out( 1869): GMS: address=node_9,
cluster=MessagingManager, physical address=fd00:0:b1:3:219:0:b801:ed26:45083
12-09 21:03:16.710 I/System.out( 1869): ------------------------------
-------------------------------------
12-09 21:03:32.230 D/GMS ( 1869): node_9: sending JOIN(node_9) to node_1
12-09 21:03:38.110 D/NAKACK2 ( 1869):
12-09 21:03:38.110 D/NAKACK2 ( 1869): [node_9 setDigest()]
12-09 21:03:38.110 D/NAKACK2 ( 1869): existing digest: []
12-09 21:03:38.110 D/NAKACK2 ( 1869): new digest: node_1: [1403
(1403)], node_5: [37 (37)], node_14: [913 (913)], node_3: [1901 (1901)],
node_12: [6 (6)], node_9: [0 (0)]
12-09 21:03:38.110 D/NAKACK2 ( 1869): resulting digest: node_1: [1403
(1403)], node_5: [37 (37)], node_14: [913 (913)], node_3: [1901 (1901)],
node_12: [6 (6)], node_9: [0 (0)]
12-09 21:03:38.110 D/GMS ( 1869): node_9: installing view [node_1|476]
(6) [node_1, node_5, node_14, node_3, node_12, node_9]
12-09 21:03:38.120 I/MessagingManager( 1869): viewChange: [node_1|476] (6)
[node_1, node_5, node_14, node_3, node_12, node_9]
12-09 21:03:38.120 D/FLUSH ( 1869): node_9: installing view [node_1|476]
(6) [node_1, node_5, node_14, node_3, node_12, node_9]
12-09 21:03:38.130 W/NAKACK2 ( 1869): JGRP000011: node_9: dropped message
5694 from non-member node_8 (view=[node_1|476] (6) [node_1, node_5,
node_14, node_3, node_12, node_9])
12-09 21:03:38.130 W/NAKACK2 ( 1869): JGRP000011: node_9: dropped message
64 from non-member node_7 (view=[node_1|476] (6) [node_1, node_5, node_14,
node_3, node_12, node_9])
12-09 21:03:38.130 D/FD_SOCK ( 1869): node_9: ping_dest is node_1,
pingable_mbrs=[node_1, node_5, node_14, node_3, node_12, node_9]
12-09 21:03:38.130 I/MessagingManager( 1869): Node 9 has connected to
channel: node_9
12-09 21:03:38.130 D/STATE ( 1869): node_9: asking node_1 for state
12-09 21:03:38.130 D/FLUSH ( 1869): node_9: blocking for 8000ms
12-09 21:03:38.140 D/STATE ( 1869): node_9: setting the state in the
aplication
12-09 21:03:38.140 I/MessagingManager( 1869): setState() called on node 9
12-09 21:03:39.550 D/FLUSH ( 1869): node_9: received RESUME, sending
STOP_FLUSH to all
12-09 21:03:39.550 I/MessagingManager( 1869): Main Channel is connected to
MessagingManager using address b79f7432-bfaf-ca76-e1d3-5cdc9709f563
12-09 21:03:39.550 D/FLUSH ( 1869): node_9: received STOP_FLUSH,
unblocking FLUSH.down() and sending UNBLOCK up
12-09 21:03:39.550 W/MessagingManager( 1869): Node 9 has unblocked.
12-09 21:03:39.550 W/MessagingManager( 1869): Node 9 has unblocked.
12-09 21:03:43.779 W/MessagingManager( 1869): Node 9 is blocked.
12-09 21:03:43.779 W/MessagingManager( 1869): Node 9 is blocked.
12-09 21:03:43.779 D/FLUSH ( 1869): node_9: received START_FLUSH,
responded with FLUSH_COMPLETED to node_1
12-09 21:03:43.809 D/GMS ( 1869): node_9: installing view [node_1|477]
(7) [node_1, node_5, node_14, node_3, node_12, node_9, node_4]
12-09 21:03:43.819 D/FD_SOCK ( 1869): node_9: socket to node_1 was closed
gracefully
12-09 21:03:43.819 D/FD_SOCK ( 1869): node_9: ping_dest is node_4,
pingable_mbrs=[node_1, node_5, node_14, node_3, node_12, node_9, node_4]
# This is the last view that we accept
12-09 21:03:43.819 I/MessagingManager( 1869): viewChange: [node_1|477] (7)
[node_1, node_5, node_14, node_3, node_12, node_9, node_4]
# On every view change the coordinator also multicasts the view as a string
which I use for detecting this fault
12-09 21:03:43.829 D/FLUSH ( 1869): node_9: installing view [node_1|477]
(7) [node_1, node_5, node_14, node_3, node_12, node_9, node_4]
12-09 21:03:43.829 D/FLUSH ( 1869): node_9: blocking for 8000ms
12-09 21:03:45.329 D/FLUSH ( 1869): node_9: blocking for 8000ms
12-09 21:03:46.409 D/FLUSH ( 1869): node_9: blocking for 8000ms
12-09 21:03:46.409 D/FLUSH ( 1869): node_9: blocking for 8000ms
12-09 21:03:46.449 D/FLUSH ( 1869): node_9: blocking for 8000ms
12-09 21:03:46.499 D/FLUSH ( 1869): node_9: blocking for 8000ms
12-09 21:03:51.829 W/FLUSH ( 1869): node_9: unblocking after 8000ms
12-09 21:03:53.059 D/FLUSH ( 1869): node_9: received STOP_FLUSH,
unblocking FLUSH.down() and sending UNBLOCK up
12-09 21:03:53.059 W/MessagingManager( 1869): Node 9 has unblocked.
12-09 21:03:53.059 W/MessagingManager( 1869): Node 9 has unblocked.
12-09 21:03:56.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 10008 ms, adding it to suspect list
12-09 21:03:58.149 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
12-09 21:03:58.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 12012 ms, adding it to suspect list
12-09 21:04:00.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 14013 ms, adding it to suspect list
12-09 21:04:00.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 10010 ms, adding it to suspect list
12-09 21:04:02.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 16014 ms, adding it to suspect list
12-09 21:04:02.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 12011 ms, adding it to suspect list
12-09 21:04:04.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 18015 ms, adding it to suspect list
12-09 21:04:04.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 14012 ms, adding it to suspect list
12-09 21:04:06.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 19540 ms, adding it to suspect list
12-09 21:04:06.799 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 15537 ms, adding it to suspect list
12-09 21:04:07.869 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
12-09 21:04:08.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 11030 ms, adding it to suspect list
12-09 21:04:08.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 21541 ms, adding it to suspect list
12-09 21:04:08.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_3 for 11532 ms, adding it to suspect list
12-09 21:04:08.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 17538 ms, adding it to suspect list
12-09 21:04:10.419 D/FLUSH ( 1869): node_9: received STOP_FLUSH,
unblocking FLUSH.down() and sending UNBLOCK up
12-09 21:04:10.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 13031 ms, adding it to suspect list
12-09 21:04:10.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 23543 ms, adding it to suspect list
12-09 21:04:10.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_3 for 13534 ms, adding it to suspect list
12-09 21:04:10.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 19539 ms, adding it to suspect list
12-09 21:04:11.729 D/FLUSH ( 1869): node_9: received STOP_FLUSH,
unblocking FLUSH.down() and sending UNBLOCK up
12-09 21:04:11.729 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
12-09 21:04:12.669 D/FLUSH ( 1869): node_9: FLUSH_COMPLETED from node_1,
completed false, flushMembers [], flushCompleted [node_1]
12-09 21:04:12.679 D/FLUSH ( 1869): node_9: FLUSH_COMPLETED from node_1,
completed false, flushMembers [], flushCompleted [node_1]
12-09 21:04:12.689 D/FLUSH ( 1869): node_9: FLUSH_COMPLETED from node_1,
completed false, flushMembers [], flushCompleted [node_1]
12-09 21:04:12.689 D/FLUSH ( 1869): node_9: FLUSH_COMPLETED from node_1,
completed false, flushMembers [], flushCompleted [node_1]
12-09 21:04:12.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 15033 ms, adding it to suspect list
12-09 21:04:12.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 25544 ms, adding it to suspect list
12-09 21:04:12.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_3 for 15535 ms, adding it to suspect list
12-09 21:04:12.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 21541 ms, adding it to suspect list
12-09 21:04:13.449 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
12-09 21:04:13.449 D/FLUSH ( 1869): node_9: received FLUSH_RECONCILE,
passing digest to NAKACK node_1: [1481 (1481)], node_5: [40 (40)], node_14:
[916 (916)], node_3: [1950 (1950)], node_12: [37 (37)], node_9: [45 (45)],
node_4: [10 (10)]
12-09 21:04:13.449 D/FLUSH ( 1869): node_9: returned from
FLUSH_RECONCILE, sending RECONCILE_OK to node_1
# Apparently this is the point where we go horribly wrong. Our coordinator
-- node_1 has installed a view that we are
# not a part of and for some reason instead of installing our own view with
us as coordinator we assume our
# view 477 of node_1's cluster is better than the coordinator's view 478.
What is the expected recovery from this situation?
12-09 21:04:14.489 W/GMS ( 1869): node_9: not member of view
[node_1|478]; discarding it
# from this point on I think we are recieving and accepting multicast
traffic from nodes in view 477 but none of them
# have us in their view so we are being shunned. I believe we will stay
stuck this way forever.
12-09 21:04:14.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 17034 ms, adding it to suspect list
12-09 21:04:14.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 27546 ms, adding it to suspect list
12-09 21:04:14.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_3 for 17537 ms, adding it to suspect list
12-09 21:04:14.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 23542 ms, adding it to suspect list
12-09 21:04:16.679 D/FLUSH ( 1869): node_9: received STOP_FLUSH,
unblocking FLUSH.down() and sending UNBLOCK up
# Right here is where node_1 has installed view 479 but we reject it too.
Interestingly we still receive the multicast view string from node_1
12-09 21:04:16.689 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
12-09 21:04:16.709 W/GMS ( 1869): node_9: failed to create view from
delta-view; dropping view: java.lang.IllegalStateException: the view-id of
the delta view ([node_1|478]) doesn't match the current view-id
([node_1|477]); discarding delta view [node_1|479], ref-view=[node_1|478],
left=[node_3]
12-09 21:04:16.729 D/FLUSH ( 1869): node_9: received STOP_FLUSH,
unblocking FLUSH.down() and sending UNBLOCK up
12-09 21:04:16.729 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
12-09 21:04:16.759 D/FLUSH ( 1869): node_9: received FLUSH_RECONCILE,
passing digest to NAKACK node_1: [1493 (1493)], node_12: [40 (40)], node_4:
[13 (13)]
12-09 21:04:16.759 D/FLUSH ( 1869): node_9: returned from
FLUSH_RECONCILE, sending RECONCILE_OK to node_1
12-09 21:04:16.769 D/STABLE ( 1869): suspending message garbage collection
12-09 21:04:16.769 D/STABLE ( 1869): node_9: resume task started,
max_suspend_time=220000
12-09 21:04:16.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 19035 ms, adding it to suspect list
12-09 21:04:16.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 29547 ms, adding it to suspect list
12-09 21:04:16.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_3 for 19538 ms, adding it to suspect list
12-09 21:04:16.809 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 25543 ms, adding it to suspect list
# trimmed a bunch more of the same heartbeat messages
12-09 21:04:44.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 47074 ms, adding it to suspect list
12-09 21:04:44.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 57585 ms, adding it to suspect list
12-09 21:04:44.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_3 for 47577 ms, adding it to suspect list
12-09 21:04:44.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 53582 ms, adding it to suspect list
12-09 21:04:44.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_7 for 14512 ms, adding it to suspect list
# I put this in as an attempt to recover from our shunned state. We make
use of the view string that was multicast by node_1
# Our software figures out that something is wrong because have not managed
to install the view that we received a copy of in
# a broadcast message 30 seconds ago so we begin the process of shutting
down here.
# If we didn't do this we would stay stuck this way forever
12-09 21:04:46.749 E/MessagingManager( 1869): Received broadcast view
[node_1|479] (3) [node_1, node_12, node_4] (viewId=479) from 30 seconds ago
is ahead of our channel view id =477 -- our cluster is messed up
12-09 21:04:46.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 49075 ms, adding it to suspect list
12-09 21:04:46.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 59587 ms, adding it to suspect list
12-09 21:04:46.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_3 for 49578 ms, adding it to suspect list
12-09 21:04:46.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 55584 ms, adding it to suspect list
12-09 21:04:46.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_7 for 16514 ms, adding it to suspect list
12-09 21:04:47.459 W/GMS ( 1869): node_9: not member of view
[node_1|480]; discarding it
12-09 21:04:47.959 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
12-09 21:04:48.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 51078 ms, adding it to suspect list
12-09 21:04:48.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_5 for 61589 ms, adding it to suspect list
12-09 21:04:48.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_3 for 51581 ms, adding it to suspect list
12-09 21:04:48.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_14 for 57586 ms, adding it to suspect list
12-09 21:04:48.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_7 for 18516 ms, adding it to suspect list
12-09 21:04:49.459 W/GMS ( 1869): node_9: failed to create view from
delta-view; dropping view: java.lang.IllegalStateException: the view-id of
the delta view ([node_1|480]) doesn't match the current view-id
([node_1|477]); discarding delta view [node_1|481], ref-view=[node_1|480],
joined=[node_2, node_11]
12-09 21:04:49.459 D/FLUSH ( 1869): node_9: received STOP_FLUSH,
unblocking FLUSH.down() and sending UNBLOCK up
12-09 21:04:50.839 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 53079 ms, adding it to suspect list
# Trimmed more of the same heartbeat messages
12-09 21:04:54.849 D/FD_ALL ( 1869): haven't received a heartbeat from
node_7 for 24520 ms, adding it to suspect list
12-09 21:04:56.209 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
12-09 21:04:56.849 D/FD_ALL ( 1869): haven't received a heartbeat from
node_8 for 59083 ms, adding it to suspect list
# Trimmed more of the same heartbeat messages
12-09 21:04:58.849 D/FD_ALL ( 1869): haven't received a heartbeat from
node_7 for 28523 ms, adding it to suspect list
12-09 21:04:59.580 D/FLUSH ( 1869): node_9: received START_FLUSH but I'm
not flush participant, not responding
# ... We eventually shut down
I still can't reproduce this on demand so this is really all I have. Here
is my protocol stack:
stack.addProtocol(new UDP().setValue("enable_diagnostics",
true)
.setValue("ip_mcast", true)
.setValue("ip_ttl", 8)
.setValue("max_bundle_size", 1400)
.setValue("max_bundle_timeout",
25)
.setValue("mcast_port",
multicastPortNum)
.setValue("bind_addr",
localInetAddress)
.setValue("bind_interface_str",
interfaceName)
.setValue("bind_port", 0)
// Sizes are in bytes
.setValue("mcast_recv_buf_size",
5 * 1024 * 1024)
.setValue("mcast_send_buf_size",
5 * 1024 * 1024)
.setValue("ucast_recv_buf_size",
5 * 1024 * 1024)
.setValue("ucast_send_buf_size",
5 * 1024 * 1024)
.setValue("thread_naming_pattern",
"pcl")
//
.setValue("thread_pool_enabled",
true)
.setValue("thread_pool_keep_alive_time",
30000)
.setValue("thread_pool_max_threads",
16)
.setValue("thread_pool_min_threads",
16)
// Turns out new threads are not
added till the queue is full
.setValue("thread_pool_queue_enabled",
true)
.setValue("thread_pool_queue_max_size",
51)
.setValue("thread_pool_rejection_policy",
rejectionPolicy)
//
.setValue("oob_thread_pool_enabled",
true)
.setValue("oob_thread_pool_keep_alive_time",
30000)
.setValue("oob_thread_pool_max_threads",
32)
.setValue("oob_thread_pool_min_threads",
32)
.setValue("oob_thread_pool_queue_enabled",
true)
.setValue("oob_thread_pool_queue_max_size",
52)
.setValue("oob_thread_pool_rejection_policy",
rejectionPolicy)
//
.setValue("internal_thread_pool_enabled",
true)
.setValue("internal_thread_pool_keep_alive_time",
30000)
.setValue("internal_thread_pool_max_threads",
16)
.setValue("internal_thread_pool_min_threads",
16)
.setValue("internal_thread_pool_queue_enabled",
true)
.setValue("internal_thread_pool_queue_max_size",
50)
.setValue("internal_thread_pool_rejection_policy",
rejectionPolicy)
//
.setValue("timer_keep_alive_time",
3000)
.setValue("timer_max_threads", 8)
.setValue("timer_min_threads", 8)
.setValue("timer_queue_max_size",
50)
.setValue("timer_rejection_policy",
rejectionPolicy)
.setValue("timer_type",
"new3").setValue("tos", 8))
.addProtocol(new PING())
// .addProtocol(new TRACE())
.addProtocol(new MERGE3().setValue("min_interval",
2500).setValue("max_interval", 5000)
.setValue("check_interval",
9000))
// Was 20k and 10k
.addProtocol(new FD_SOCK().setValue("bind_addr",
localInetAddress)
.setValue("bind_interface_str",
interfaceName))
.addProtocol(new FD_ALL().setValue("interval",
3000).setValue("timeout", 10000)
.setValue("msg_counts_as_heartbeat",
false))
.addProtocol(new
VERIFY_SUSPECT().setValue("use_mcast_rsps",
false)
.setValue("timeout", 3000)
.setValue("bind_addr", localInetAddress)
.setValue("bind_interface_str",
interfaceName))
.addProtocol(new BARRIER())
.addProtocol(new
NAKACK2().setValue("discard_delivered_msgs",
true)
.setValue("use_mcast_xmit", false)
.setValue("use_mcast_xmit_req",
false)
.setValue("xmit_from_random_member",
false)
.setValue("xmit_interval", 500)
.setValue("discard_delivered_msgs",
true)
.setValue("xmit_table_max_compaction_time",
30000)
.setValue("xmit_table_msgs_per_row",
2048)
.setValue("xmit_table_num_rows",
100))
.addProtocol(new UNICAST3()
.setValue("conn_expiry_timeout", 0)
.setValue("xmit_interval", 500)
.setValue("max_retransmit_time",
60000)
.setValue("xmit_table_max_compaction_time",
60000)
.setValue("xmit_table_msgs_per_row",
2048)
.setValue("xmit_table_num_rows",
100))
// According to Bela Ban we should have a frag
protocol higher in the stack than UNICAST3/NAKACK2
.addProtocol(new FRAG2().setValue("frag_size",
1250))
// .addProtocol(new RSVP().setValue("ack_on_delivery",
false).setValue("resend_interval", 250)
//
.setValue("throw_exception_on_timeout", true).setValue("timeout", 5000))
.addProtocol(new
STABLE().setValue("desired_avg_gossip",
15000).setValue("max_bytes", 4000000)
.setValue("send_stable_msgs_to_coord_only",
false)
.setValue("stability_delay",
1000))
.addProtocol(new GMS().setValue("join_timeout",
15000) // This is the discovery timeout as well
.setValue("leave_timeout",
15000)
.setValue("print_local_addr",
true)
.setValue("view_bundling",
true)
.setValue("merge_timeout",
30000)
.setValue("view_ack_collection_timeout",
7500)
.setValue("max_join_attempts",
100000) // A huge number so we never give up
// Use the largest sub-group
not the oldest coordinator
.setValue("membership_change_policy",
new LargestWinningPolicy()) //
).addProtocol(new STATE().setValue("buffer_size",
10000)
.setValue("max_pool", 5)
.setValue("pool_thread_keep_alive", 10000)) //;
.addProtocol(new FLUSH()); // Pauses traffic while
state transfers are occurring
I'm wondering what condition has to occur for the coordinator to drop
node_9 from the view and node_9 being oblivious. Is there a setting that I
could change that will cause this to recover faster, or at all? Or should
JGroups be checking to see if the rejected view is coming from our
coordinator and do something other than just dropping it?
Thanks
JT
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users