Re: QueueProxy

Quartz wrote:
>>we could create a mecanism where Sandstorm detect dynamically the 
>>queue contention and change on the fly the queue implementation to 
>>improve performance.  (Sandstorm will do this operation only rarely)
> 
> 
> General comment on dynamic behaviors and predicitability:
> Although it looks cool, it kinda add 'zeros' (Laplace maths) in the system's transfer function.
> These zeros may get dangerously close to the right side of the plane when combined.
> That's the kind of assemly that create oscillations in memory/gc and CPU throughput once they are
> in synergy with the remaining components. Also, systems that change behavior according to load
> tend to sync all together of course, leading to a disastrous mega oscillation between good and bad
> throughput, unless careful dampening of those behavioral changes is coded (i.e. changing behavior
> at slower pace than the longest response time to any behavior change, the goal being to let the
> system stabilise before changing behaviors).Given such complexity, engineers tend to let the cool
> stuff behind and code it simple and linear.
> 
> Sandstorm has a lot of dynamic behaviors: thread pool sizes, thread priority, rate controllers,
> batch controllers (which already oscillates between 90 and 98 % of max throughput).
> 
> I'm not saying queue switching/proxying is not a good idea. Just keep the above in mind. Beside
> keeping it simple (which I keep forgetting time to time!), we should protect predictability.
> 

I am wondering if having a QueueProxy is a good idea?

I think that a QueueProxy could increase Sandstorm flexibility.  We 
could change the queue implementation or decorate/proxy the queue 
dynamically.  By exemple, we could move dynamically a stage on another 
computer by only changing its queue implementation with a remote queue. 
  We don't need to stop handlers that use this stage.  This could be 
useful for load-balancing stages on a cluster of workstations.

Another similar exemple could be to enable/disable dynamically the 
multiplexing of QueueElement on different computers to mirror a stage.

Also, these proxies could be chained if needed.

Do you think this greater flexibility is enough important to sacrifice 
predictability?






-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/