Re: SEDA batchController and stage overloaded

Quartz,
Thanks for your reply.
 
I guess I wasn't paying attention to that stage (the one that is getting overloaded).
This stage needs to execute a stored procedure (oh, hell, that's the problem).
 
So, this is a blocking call that really messes up all control over this stage.
Now, what should be the best to do:
- Implement some kind of emulation layer, in order to turn this jdbc call into some kind of a asynchronous one? (Hummm... still thinkin' how I would do that).
- Implement my own Thread Manager in order to ajust thread priorities to avoid this overload? (as you suggested)
- Implement some kind of blocking-queue just for this overloaded stage... (hummm... this must overload the first stage... shouldn't it?).
 
Overview of what is happening:
 
firstStage (1 thread) --> secondStage (1000 threads) ---> thirdStage (1 thread)
 
The second stage's queue length increases to 250 even with only on thread on the first stage!!
 
The other stage get no queue at all (sampled by 1000 ms).
 
Thanks in advance.
Jean Rodrigues
 
 
 


Quartz <[email protected]> wrote:
Sound like the first stage is overproducing relatively to what the second stage can swallow.
Even with only 1 thread that 1st stage may be just too powerfull. Start by adding a thread
priority setting to threads in the pool. Easy enhancements to config. You would of course lower
the first stage prio to 4, 3, 2 or 1, whatever you need.

You cannot indefinitely create threads for second stage hoping it will slow down the first stage
because if the input rate was 1000 times faster, you would need a 1000 threads on the second
stage!

Sandstorm, as is, is lacking a crutial element: blocking enqueue. I know it defeats the current
implementation philosophy, but that is the only way I figured to apply BACKPRESSURE correctly
without data loss.

You must review the threadpool controller to NOT alter the state of a stage thread pool whenever
this stage is "blocking", that is, you must track when a thread enter a queue in blocking mode.

Now, I suggest you allocate a good 30-50 hours*person of work to solve this blocking enqueue
(optionnally with timeout) problem accuratly. Final word: write your own thread manager and use
thread priority instead of thread pool size to gauge stages power. It scales much better, and does
exactly what you want, that is give power (high prio) to worst performing stages.

Works like a charm for us (sun's jvm on both windows and linux). Note you can have more than 1
thread on a stage if priority is not enough adjustement. A very bad stage could have 5 threads,
all priority adjusted, which give a 5x multiplier to the stage processing power, combined with the
thread prio 1-9. (I keep prio 10 for control purpose).

Good luck.


--- Jean wrote:
> Hi ppl,
> I've been developing with seda framework for a few months but now I've encountered some
> problems:
> - How can I make batchController act like a timer? (batching info at some rate)
> - I'm using ThreadPoolController, so I can resize stage's thread pool when needed. The problem
> is ... even with 100 threads, the queue length increases to almost 700 elements. This stage is
> suposed to respond within 2 seconds. 
> I tried to use ResponseTimeController, but no success (90 RT seems not to work and Threshold
> seems not be set correctly... dont know why yet).
> 
> Is there some advice in order to make that work?
> Thanks in advance.
> 
> Jean Rodrigues
> 
> 
> 
> 
> ---------------------------------
> Crie seu Yahoo! Mail, agora com 100MB de espaço, anti-spam e antivírus grátis!




__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail



---------------------------------
Crie seu Yahoo! Mail, agora com 100MB de espaço, anti-spam e antivírus grátis!