Re: [jgroups-users] Getting high latency!!!
Questions/problems related to using JGroups <[email protected]>
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <[email protected]> |
First of all, I am sorry for misunderstanding some points. Comments are inline. > No, in fact you are correct. Because my cluster size is 113 not only 3. < 3 for ZK servers, 10 for the 10 machines that generate threads (senders) <<What's this? > and 100 for threads (as each client has its own channel), note that only > those (100 > senders) are used for workload. >Now you lost me... My cluster size is 113. The 3 separate machines are dedicated for ZooKeeper servers (ZK stand for ZooKeeper, sorry for not clarify that before). Also, I have another 10 machines (let call them controllers for simplification). This machines does not involve in workloads, it just creates clients (senders or can named threads) who involves in workload, and also tell clients to start the workload and check which clients has finished the workload. So, each controller operates and manages 10 clients, as a result we have 100 clients. Therefore, I have 113 channels. The 3 channels are dedicated to ZK servers (one channel for each ZK server). 10 channels are dedicated to controllers. 100 channels are dedicated for clients. Note that all channels are in same cluster (they have the same channel name). <<This is by design, to enable faster processing by applications. As <<mentioned in the doc, regular messages from sender P are delivered in <<the sequence P sent them, but messages from other senders are delivered <<in parallel. Plus, all OOB messages are delivered in parallel. (***) Ok I see, that is why when I had 10 senders all with one channel, I lost some delivered messages (because messages are delivered in parallel). Therefore, I make each sender has its own channel and receiver. <<Synchronizing receive() would hamper performance and kill the <<performance of parallel delivery. It is therefore the application's <<responsibility to deliver messages in an app-defined manner. Ok, I see. > When I override Receive method, I add synchronized code to > catch all replays inside the Receive method but it does not work for me. > So > that is why I have for each sender its own channel. It could be more > sufficient solutions to remove such overhead channel. <<I must confess I didn't understand what you mean above. If you have a <<simple application, I could take a look... Sorry for that. See my comment above (has sign ***). And two files have been attached. One call ZabTestThreads which is for controller and the second file is named ZabClient for sender. <nabble_a href="ZabTestThreads.java">ZabTestThreads.java ZabClient.java <http://jgroups.1086181.n5.nabble.com/file/n11010/ZabClient.java> . > <<The problem is that your thread pools below are too small to process > <<this load and thus requests (and acks) get queued in the regular thread > <<pool (queue size is 50'000) and the regular pool has a max of 8 threads. > <<This introduces latency to each request, depending on how many requests > <<are ahead of it in the queue. > > I see what do you mean and it makes sense. However, I have tried your > suggestion, unfortunately there is no improvement. However, I have noticed > that when I have UDP default configuration (min thread pool = 2 and Max > thread pool = 8), the number of threads pool is always 2 (equals to its > minimum) because of min threads pool (the 2 threads) queue size is enough > to > store the load (200 messages max at any given time, though), so UDP does > not > need to create more than 2 thread pool. <<I suggested to disable the queue and increase max, so all messages get <<processed. You should definitely get more than 2 processing threads! Yes, that is absolutely what I found When I used your suggestion (got more than 2 processing threads). When I said I only had two processing threads this happened when I used UDP default config (not using your suggestion). Thank you Ibrahim -- View this message in context: http://jgroups.1086181.n5.nabble.com/Getting-high-latency-tp11004p11010.html Sent from the JGroups - General mailing list archive at Nabble.com. ------------------------------------------------------------------------------ Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140