Re: SEDA batchController and stage overloaded

Hi,

Think of it this way: if you are even driven, you must be all the way. And so, your stored proc
must be asynchronous, yes.

But as soon as you cannot do async calls, then you must "emulate" async calls.

In those cases where async emulation cannot be done, you must forget event driven philosophy, and
be glad that seda offers a good messaging framework as plan B. Stages are a good way to partition
the architecture in roles. The constraint is to have a reliable transport of messages, hence the
blocking enqueues.

I found out that having stages and threads are not usefull for finite mandatory processing, in
which case you better process in one thread across plugged tasks (simple state machine pattern).
But if you have optionnal tasks and morphing "data paths" which may have dynamically changing
load, then sandstorm offer a great way to balance, and even limit processing abuses for tasks of
lesser importances.

Good luck.


--- Jean  <[email protected]> wrote:
> Quartz,
> Thanks for your reply.
>  
> I guess I wasn't paying attention to that stage (the one that is getting overloaded).
> This stage needs to execute a stored procedure (oh, hell, that's the problem).
>  
> So, this is a blocking call that really messes up all control over this stage.
> Now, what should be the best to do:
> - Implement some kind of emulation layer, in order to turn this jdbc call into some kind of a
> asynchronous one? (Hummm... still thinkin' how I would do that).
> - Implement my own Thread Manager in order to ajust thread priorities to avoid this overload?
> (as you suggested)
> - Implement some kind of blocking-queue just for this overloaded stage... (hummm... this must
> overload the first stage... shouldn't it?).
>  
> Overview of what is happening:
>  
> firstStage (1 thread) --> secondStage (1000 threads) ---> thirdStage (1 thread)
>  
> The second stage's queue length increases to 250 even with only on thread on the first stage!!
>  
> The other stage get no queue at all (sampled by 1000 ms).
>  
> Thanks in advance.
> Jean Rodrigues


		
__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com