Re: [jgroups-users] Getting high latency!!!

On 24/03/16 17:18, Questions/problems related to using JGroups wrote:
> No, in fact you are correct. Because my cluster size is 113 not only 3.

I see. I thought there were 3 core servers and 100 clients.

> 3 for ZK servers, 10 for the 10 machines that generate threads (senders)

What's this?


> and 100 for threads (as each client has its own channel), note that only those (100
> senders) are used for workload.

Now you lost me...

> Doing this way it may be stupid. But there is a reason for choosing this
> configuration. Because I need to keep all replay messages (from Zab
> protocol) in client’s side for further check and more.
> Before, I had one channel for each 10 clients (senders), so, some replay
> messages are lost when comes to Receive call back method because the Receive
> method is not thread-safe (By the way, why Receive call back method is not
> synchronized).

This is by design, to enable faster processing by applications. As 
mentioned in the doc, regular messages from sender P are delivered in 
the sequence P sent them, but messages from other senders are delivered 
in parallel. Plus, all OOB messages are delivered in parallel.

Synchronizing receive() would hamper performance and kill the 
performance of parallel delivery. It is therefore the application's 
responsibility to deliver messages in an app-defined manner.


> When I override Receive method, I add synchronized code to
> catch all replays inside the Receive method but it does not work for me. So
> that is why I have for each sender its own channel. It could be more
> sufficient solutions to remove such overhead channel.

I must confess I didn't understand what you mean above. If you have a 
simple application, I could take a look...


> <<The problem is that your thread pools below are too small to process
> <<this load and thus requests (and acks) get queued in the regular thread
> <<pool (queue size is 50'000) and the regular pool has a max of 8 threads.
> <<This introduces latency to each request, depending on how many requests
> <<are ahead of it in the queue.
>
> I see what do you mean and it makes sense. However, I have tried your
> suggestion, unfortunately there is no improvement. However, I have noticed
> that when I have UDP default configuration (min thread pool = 2 and Max
> thread pool = 8), the number of threads pool is always 2 (equals to its
> minimum) because of min threads pool (the 2 threads) queue size is enough to
> store the load (200 messages max at any given time, though), so UDP does not
> need to create more than 2 thread pool.

I suggested to disable the queue and increase max, so all messages get 
processed. You should definitely get more than 2 processing threads!

> I really agree when you said the problem may be because of the messages are
> queued and the latency depends on how many messages ahead in queue. And as
> always we have only 2 threads pool during the workload time, this also
> supports your observation.

Yes

> Moreover, I increased the number of thread pool to 200, 400 and 600 but this
> also does not minimise the peak latencies.
> I have more tries to do regarding to your suggestion I will tell you soon.

Post a simple reproducer with config and instructions on how to run it 
and I'll take a look.

> Ibrahim
>
>
>
>
> --
> View this message in context: http://jgroups.1086181.n5.nabble.com/Getting-high-latency-tp11004p11008.html
> Sent from the JGroups - General mailing list archive at Nabble.com.

> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users