Re: spread toolkit benchmark and performance

On Mon, Aug 29, 2011 at 01:04:50PM +0900, arrrr gv wrote:
> I actually have wrote a prototipe with perl in my company to
> concentrate server log files in a single point of the network and
> analyze it based on certain parameters. Till now everything works
> right so I'm looking forward to put it in production, but the main
> problem I got now is my boss wanted to me to stress the
> application so he wants to be sure nothing will goes wrong based
> on the spread toolkit and all the scripting I have done around.
> 
> So could you let me know what would you recomend me to stress the
> spread toolkit and benchmark it? I don't know maybe there's a
> simple way, i would need to know the CPU, Memory and swap
> consumption based on a the spread toolkit performance when is
> working hard, I mean, to see which limits it has... I don't know
> maybe someone already have done so? how?

My suggestion is to try to imagine the worst-case scenario, double
(or triple it), then try to write a program (or multiple programs)
that simulate that situation.

For example, say you have X servers that all simultaneously start
logging like crazy (e.g. filesystem errors, out of memory, RAM going
bad, mail queue is full, etc).  So write a script that simulates
logging as fast as possible, and run it from 2X or 3X servers.

While you have all those scripts simulating the logging, observe
what your spread daemon(s) is(are) doing.  Assuming you are on a
Unix-like system, use tools like vmstat, sar, ps, even good old top
to get an idea of the kind of resources the spread daemon is using.
If you have a lot of servers, and they are connected by a fast
network connection, and you blast data at the spread daemon as fast
as possible, it will probably croak.  Few programs could keep up
with that kind of scenario.  But you can increase/decrease the
message rate from your test programs, and/or increase/decrease the
number of simultaneous writes to try to determine what your setup's
max capacity is.

You might want to consider increasing the size of your network
receive buffers on the server that runs the central logging daemon.

Ultimately, it's a matter of trial-and-error.  Try to write a
"writer" program that simulates your logging.  That program would
ideally be parameterized by size of message, message rate, etc.

Hope that helps!