Re: [jgroups-dev] The message from Kuwon Kang about problem.
Bela Ban <[email protected]>
| Newsgroups | gmane.comp.java.javagroups.devel |
|---|---|
| Message-ID | <[email protected]> |
What did you do to run into this issue ? Restart 7801 ?
To avoid reincarnation:
* Wait a bit after killing a node before restarting it
* Don't use TCP.start_port, so we pick an ephemeral port rather than
7800 and thus reduce the chances of reincarnation
* Upgrade to 2.10
Kuwon Kang wrote:
> On Thu, Aug 19, 2010 at 6:19 PM, Kuwon Kang <[email protected]> wrote:
>
>
>> Thanks.
>>
>> But unfortunately we can not test or upgrade right now because we are
>> operating in production environment.
>>
>> Is it possible(ok) that we configured jgroups config with different ip to
>> avoid this problem?
>>
>> I read your post and release note but this problem might not same with the
>> reincarnation problem ? Am I right?
>>
>> The article does not seem to tell about our problem.
>>
>> below xml is our entirely configuration.
>>
>>
>>
>> - <config>
>> <TCP start_port="*7800*" end_port="*7801*" loopback="*true*"recv_buf_size
>> ="*20000000*" send_buf_size="*640000*" discard_incompatible_packets="*true
>> *" max_bundle_size="*64000*" max_bundle_timeout="*30*" enable_bundling="*
>> true*" use_send_queues="*false*" sock_conn_timeout="*300*"skip_suspected_members
>> ="*true*" thread_pool.enabled="*true*" thread_pool.min_threads="*1*"thread_pool.max_threads
>> ="*200*" thread_pool.keep_alive_time="*5000*" thread_pool.queue_enabled="*
>> false*" thread_pool.queue_max_size="*100*" thread_pool.rejection_policy="*
>> run*" oob_thread_pool.enabled="*true*" oob_thread_pool.min_threads="*1*"oob_thread_pool.max_threads
>> ="*8*" oob_thread_pool.keep_alive_time="*5000*"oob_thread_pool.queue_enabled
>> ="*false*" oob_thread_pool.queue_max_size="*100*"oob_thread_pool.rejection_policy
>> ="*run*" />
>> <TCPPING timeout="*3000*" initial_hosts="*
>> ${jgroups.tcpping.initial_hosts:100.254.161.55[7800]}*" port_range="*2*"num_initial_members
>> ="*2*" />
>> <MERGE2 max_interval="*100000*" min_interval="*20000*" />
>> <FD_SOCK />
>> <FD timeout="*10000*" max_tries="*5*" shun="*true*" />
>> <VERIFY_SUSPECT timeout="*1500*" />
>> <BARRIER />
>> <pbcast.NAKACK use_mcast_xmit="*false*" gc_lag="*0*" retransmit_timeout
>> ="*300,600,1200,2400,4800*" discard_delivered_msgs="*true*" />
>> <UNICAST timeout="*300,600,1200*" />
>> <pbcast.STABLE stability_delay="*1000*" desired_avg_gossip="*50000*"max_bytes
>> ="*400000*" />
>> <VIEW_SYNC avg_send_interval="*60000*" />
>> <pbcast.GMS print_local_addr="*true*" join_timeout="*3000*" shun="*true
>> *" view_bundling="*true*" />
>> <FC max_credits="*2000000*" min_threshold="*0.10*" />
>> <FRAG2 frag_size="*60000*" />
>> <pbcast.STREAMING_STATE_TRANSFER />
>> - <!--
>>
>> <pbcast.STATE_TRANSFER/>
>>
>> -->
>> </config>
>>
>> d1105621@FLPREA01:/APP/sli_batch/tomcat-cp-2/bin > netstat -na |grep 7801
>> tcp 0 0 *.7801 *.* LISTEN
>> tcp 0 0 100.254.161.55.63655 100.254.161.*57*.7801
>> ESTABLISHED
>> tcp 0 0 100.254.161.*57*.7801 100.254.161.55.63655
>> ESTABLISHED
>> d1105621@FLPREA01:/APP/sli_batch/tomcat-cp-2/bin > netstat -na |grep 7800
>> tcp 0 0 100.254.161.55.56426 100.254.161.*55*.7800
>> ESTABLISHED
>> tcp 0 0 100.254.161.*55*.7800 100.254.161.55.56426
>> ESTABLISHED
>> tcp 0 0 *.7800 *.* LISTEN
>>
>> Does reincarnation problem relate with NIC environment?
>>
>> PS: I've been visited (https://sourceforge.net/mail/?group_id=6081) and
>> clicked ps-development Archives<https://sourceforge.net/mailarchive/forum.php?forum_name=javagroups-development> but
>> could not post this message? Am I wrong? Tell me how to to post this
>> message.
>>
>>
>> On Thu, Aug 19, 2010 at 5:55 PM, Bela Ban <[email protected]> wrote:
>>
>>
>>> Can you try with 2.10 ? The issue you ran into might be caused by
>>> reincarnation, I fixed this in 2.8.
>>>
>>> Please post to the JGroups mailing list...
>>>
>>>
>>> Kuwon Kang wrote:
>>>
>>>
>>>> Hello.
>>>> I'm kuwon form South Korea.
>>>>
>>>> I'm glad to send this message to you and I have a question about JGroups.
>>>>
>>>>
>>>> I'm using your JGroups(2.7.0 GA) TCP/IP framework well.
>>>>
>>>> But we are in trouble that came from few days ago.
>>>>
>>>> There are two nodes *in a server* in GMS for fail over.
>>>>
>>>> Now I'll call the node as the agent.
>>>>
>>>> I'll show you the configuration for TCPPING.
>>>>
>>>> First agent.
>>>>
>>>> <TCPPING timeout="3000"
>>>>
>>>> initial_hosts="${jgroups.tcpping.initial_hosts:100.254.161.55[7800]}"
>>>> port_range="2"
>>>> num_initial_members="2"/>
>>>>
>>>> Second agent.
>>>> <TCPPING timeout="3000"
>>>>
>>>> initial_hosts="${jgroups.tcpping.initial_hosts:100.254.161.55[7800]}"
>>>> port_range="2"
>>>> num_initial_members="2"/>
>>>>
>>>> These two same configurations in two agents were very fine but few days
>>>> ago
>>>> we have a problem.
>>>>
>>>> When the first agent shutdown and started again, it can not join the
>>>> oldest
>>>> node.
>>>>
>>>> [08/19 10:13:44] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:13:47] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:13:50] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:13:53] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:13:56] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:13:59] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:02] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:05] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:08] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:11] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:14] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:17] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:20] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:23] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:26] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:29] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:32] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:35] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:38] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:41] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:44] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:47] join(100.254.161.55:7801) sent to 100.254.161.57:7800timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 10:14:50] join(100.254.161.55:7801) sent to *100.254.161.57*:7800
>>>> timed out (after 3000 ms), retrying
>>>>
>>>> As you see, the remote address is 100.254.161.57:7800.
>>>>
>>>> After all we changed the first configuration like this:
>>>> <TCPPING timeout="3000"
>>>>
>>>> initial_hosts="${jgroups.tcpping.initial_hosts:*100.254.161.57
>>>> *[7800]}"
>>>> port_range="2"
>>>> num_initial_members="2"/>
>>>>
>>>> *Then the first agent could join the second agent.*
>>>> But still the second agent is using initial_host with ip 100.254.161.55.
>>>>
>>>> But very strange thing is that we tried to change the second agent's ip
>>>> address to 57 from 55, also it showed same error message like below
>>>> shown.
>>>>
>>>> [08/19 14:28:44] join(100.254.161.55:7800) sent to *100.254.161.55*:7801
>>>> timed out (after 3000 ms), retrying
>>>> [08/19 14:28:47] join(100.254.161.55:7800) sent to 100.254.161.55:7801timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 14:28:50] join(100.254.161.55:7800) sent to 100.254.161.55:7801timed
>>>> out (after 3000 ms), retrying
>>>> [08/19 14:28:53] join(100.254.161.55:7800) sent to 100.254.161.55:7801timed
>>>> out (after 3000 ms), retrying
>>>>
>>>>
>>>> This server configured two NIC.
>>>> The address end with 55 is service address.
>>>> The address end with 57 is physical address.
>>>>
>>>> This business system should support High Availability requirement,
>>>> therefore
>>>> they configured Virtual IP with 55.
>>>>
>>>> My question is how JGroups get the member's address among two addresses
>>>> like
>>>> out env(Service IP, Physical IP).
>>>> And how do we configure for initial_hosts like in this env.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Bela Ban
>>> Lead JGroups / Clustering Team
>>> JBoss
>>>
>>>
>>>
>> --
>> Blessings.
>> Kuwon Kang
>> ............................................................
>> IT specialist and architect.
>> Java technology engineer.
>> JavaEE architect/developer.
>> Spring Framework specialist.
>> Solution developer.
>>
>> - Model Driven Architecture.
>> - Test Driven Development.
>> - Test Driven Software Design.
>> - Refactoring Oriented Development.
>> - Practical design and modeling.
>> - Robust Software Design and Engineering.
>> ............................................................
>> Prever,Inc.
>> http://www.prever.co.kr/
>> Blog:
>> http://josh.prever.co.kr/
>> ............................................................
>>
>> God's love is eternal and leads us to live for heaven and His glory
>>
>>
>
>
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by
>
> Make an app they can't live without
> Enter the BlackBerry Developer Challenge
> http://p.sf.net/sfu/RIM-dev2dev
> ------------------------------------------------------------------------
>
> _______________________________________________
> Javagroups-development mailing list
>
>
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
------------------------------------------------------------------------------
This SF.net email is sponsored by
Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________
Javagroups-development mailing list