Re: [jgroups-dev] FC and performance

Hi Richard,


note that I plan to split FC into a multicast and unicast implementation 
[1]. The reasons are that this simplifies the code, reduces locking, 
makes multicasts not dependent on unicasts (they all use the same 
credits) and allows for selective removal of either unicast or multicast 
flow control.

Comments inline


Richard Achmatowicz wrote:
> Hi Bela

> I've been looking over FC with a view to finding a configuration of it 
> which does not suffer under profiling. By the way, when I use 
> jProfiler, all processes in the cluster have the same profiling 
> config, so they should all run equally slowly.

So you're profiling all processes in JProfiler ?

> In the process, i've come across some questions which i'm not quite 
> sure about.
> Here are the questions:
>
> (i) how can a peer accumulate negative credits for a sender when flow 
> control prevents sending more bytes than you are allowed?

I've looked at the code and don't see how this can happen. Did you see 
this ? If so, can you reproduce it ?

If this happens, we'd compute the credits to send to the sender as 
credit_response=max_credits - remaining_credits: if remaining_credits 
was negative, we'd add them to max_credits and would return more credits 
than needed.

This would *not* be an issue because the recipient of the credits caps 
them at max_credits anyway, so nobody would add more credits than 
max_credits.


> (ii) when receiving a message in up(), you call adjustCredit() to 
> calculate how many remaining credits there are for a sender and 
> calculate a credit response which should be sent automatically and 
> without the need for a CREDIT_REQUEST. But you send when
>
>    max_credits - remaining_credits >= min_credits (or other in words 
> max_credits >= remaining_credits + min_credits)
>
> which doesn't seem to indicate a need for more credits and might 
> result in sending messages unnecessarily. Shouldn't we send credits 
> automatically when
>
>   remaining_credits < min_credits
>
> that is when we fall below the threshold? Wasn't that the original 
> idea behind the feature?

Yes, this was changed some time ago by our friend Brian: min_credits 
means we send new credits to P as soon as we've received min_credits (or 
more) from P. So when I have max_credits=1000 and min_threshold=0.2 (= 
min_credits=200), then a receiver sends 200 credits back to P as soon as 
it has accumulated 200 bytes from P.


So min_credits is not credits left, but bytes received. Again, this is a 
change from earlier semantics. To be honest, I never liked the change 
and will probably revert back to the old semantics when I create UFC and 
MFC in [1].


> What follows is my own understanding of FC - reading it is optional, 
> but you may find it funny! :-) I haven't had time to think about 
> JGroups in a while.
>
> Maybe we can chat tomorrow if necessary?

Sure, #jgroups or skype.

More comments below...

> Richard
>
> ------------------------------------------------------------------------------------------------------ 
>
>
> RA view of FC
> ==============
> From a high level view, this is how I see it:
> - each peer has a sent map, to keep track of peers it has sent 
> messages to and the credits they currently have left, decremented on 
> each send
> - each peer has a received  map, to keep track of peers they have 
> received messages from and the credits they should have left, 
> decremented on each receive
> - if there are not enough credits to send a message to a peer, FC 
> blocks the message until there are enough credits
> - if peers just sent and received messages only, all peers would soon 
> run out of credits and block, so there has to be a way to restore credits
> - we can restore credits by exchanging messages in a credit 
> request/credit response fashion (CREDIT_REQUEST, REPLENISH)
>
> - how are these credit replenishments triggered? In one of two ways:
> (i) on message send: when we block while trying to send a message, we 
> wait for a bit and then eventually make a request for credits 
> (CREDIT_REQUEST)
> (ii) on message receipt: when the credits of a process in our receive 
> map goes below a threshold, we automatically send them credits 
> (REPLENISH)
>
> blocking and credit replenishment
> ---------------------------------
> - we only block for a limited time, and then we check who we need 
> credits from (creditors) and we send credit requests to them, and then 
> go back into the blocking loop again
> - this is inefficient, as we have to run out of credits and then wait 
> (each message is of a different size, so one large message may cause 
> us to run out of credits)


Well, we should never have to send credit requests in the normal case 
anyway ! Sending credit requests is a second line of defense.

One large message won't cause us to run out of credits as we have 
fragmentation, which fragments all message to a size that's less than 
max_credits. Note that if you change FRAG2.frag_size > FC.max_credits, 
or remove FRAG2 altogether, JGroups will spit out a warning on startup. 
Maybe we should throw an exception and not allow this at all ? That's a 
reason for example to leave FRAG2 in there even if you use TCP as 
transport...


> while (length > credits available) {
>  block for fixed time  waiting for credits
>  compute creditors
>  send request for more credits to creditors
> }
> ...
> send message now that we have credits
>
> preventative credit replenishment
> ---------------------------------
> - we receive a message from a sender and adjust their credit setting 
> based on the message received (i.e decrement length of message)
> - if a condition holds, receiver notes that senders are running out of 
> credits, and sends them replenishments ahead of time
> - the message gets delivered, and in a finally clause, credits are 
> sent back to the sender
>
> questions
> ---------
> - how can a receive accumulate negative credits?

Good question: looking at the code, I don't see how this can happen. But 
if it does happen, we correct the outcome and send back max_credits to 
the sender.

> - how is it possible that we may get a request for replenishment and 
> have no credits to give?

This is a valve that Brian added to prevent credit request storms. When 
we get 2 credit requests from a sender P without rceeiving a message 
from P in the mean time, we log a warning, as this might point to a problem.

If you run P in a profiler, and the profiler slows down P, and we have a 
lot of threads sending, then all the threads enter the lock and go into 
a TIMED_WAIT state waiting on credits_available. To prevent credit 
request storms, we keep track of the last time credit requests were sent 
out and make sure that credit request are sent out no sooner than 
max_block_time after the last time credit requests were sent.

If you look at line 561 in FC (CVS head), then you'll see that only 1 
thread can be active in this section of code. So itgoes on to send 
credits. To send credits, it releases the lock and - when done - 
re-acquires the lock. Meanwhile, other threads get to the computation of 
wait_time, but because they' ve been suspended in the debugger, 
wait_time=System.currentTimeMillis() - last_credit_request might 
actually be > max_block_time, so they would send a credit request too, 
later.

This *might* be a reason why we're seeing many credit requests under a 
profiler.


> performance issues when run with jProfiler
> -------------------------------------------
> 1. we *are* running out of credits and triggering replenishments on 
> certain threads, and so blocking for 5000 ms before sending new request
> 2. we then have to wait RTT for credits to be requested and then 
> replenished from each peer in the creditors set
> 3. this happens for *each* thread making a call to send a message, and 
> so a lot of threads can accumulate ate the await()
> 4. in a test scenario:
> - we have a lot of client requests coming in through AJP from Apache 
> which will need to replicate state
> - there will be ~1000 concurrent calls on each host, many of those 
> making downcalls for replication requests
> - the number of credits each thread need will depend on the size of 
> the messaage it carries (so, for 1000 threads, 4000000 is not a lot of 
> credit!)
> - a lot of credits will be required
> - because the profiler is run on every host, all clustering code will 
> be slowed down somewhat, although uniformly, which may affect timing
> - LB should distribute these fairly evenly
>
> possible performance tuning with profiling
> ------------------------------------------
> - use a high max_credits to take account of the higher number of 
> threads and traffic in a testing scenario, so that we don't run out fast
> - use a high percentage so we preauthorize credit faster and senders 
> don't run out of credits
> - use low max blocking time so we don't wait long before requesting 
> credits if we need them

Why are you concerned about this when you run in a profiler ? A profiler 
will still give you numbers for the rest of the system, e.g. threads in 
BLOCKED states, monitors acquired etc... Note that a thread is *not* in 
BLOCKED state waiting on the credits_available Condition variable, but 
in TIMED_WAIT state.

The question is what it is you want to do with JProfiler. If you want to 
find the top culprits in BLOCKED states, switch the view to show only 
threads in BLOCKED states.

Having said that, to reduce waiting in FC, you could

    * Increase max_credits as mentioned by you above
    * Use a lower max_block_time, although I don't favor this
    * Use max_block_times (plural), as outlined in my prev email
    * If you don't use async replication, you might as well remove FC
      altogether...

Let's chat this afternoon, ping me when you're online !
Cheers,



[1] https://jira.jboss.org/browse/JGRP-1154

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss


------------------------------------------------------------------------------
This SF.net email is sponsored by 

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Javagroups-development mailing list