Re: log file data source (aDisk perhaps)

--- [email protected] wrote:

> Ok I know this is a little bit off the roadmap, but I am involved in a =
project right now where I
> need to mine a tremendous amount of XML log files on a daily basis.  I =
currently have written a
> parser in Java that does the job but it is very slow.  Some of my analy=
sis steps are much slower
> than other steps, so I want to try to accelerate the process by using t=
he sandstorm engine to
> parallelize the slower steps and improve throughput overall.
>=20
> At some point in the future, I will modify the production systems to se=
nd a copy of the SOAP
> request/response to a SEDA listener in real time.  But in the short ter=
m I am restricted to the
> parsing of log files.
>=20
> Does anyone know of any examples that they can point me to of an aDisk =
first stage?  Or any
> general guidance on the best way to get records from a text file into t=
he data flow?  Is it as
> simple as creating a stage that reads the file, and enqueues the record=
s (so long as I restrict
> it to a single thread)?  Or is there something already implemented that=
 I didn't notice from the
> docs.
>=20
> Thanks for your help.
>=20
> -Mark

The 'physionomy' of your app requires a typical disk/memory/cpu partition=
ning.
1 thread in charge of {reading disk}, 2 threads in charge of {parsing, ma=
tching} records.

a) you want the app to do something when the disk i/o waits. That's your =
other parsing stage. The
kernel will run those threads.

b) focus on sequential reads from disk, avoid multi thread reading disk, =
it only cause disk seeks
of the HDD head and the throuput goes down by a magnitude.

c) you want to make sure you do almost no processing on the disk thread, =
that is cpu and memory
tasks that should be done on other threads (that show up well with multi =
cpus or even
hyperthreaded cpus)

d) I suppose your parsing and matching are both cpu and memory intensive,=
 so they belong to the
same kind of work. Yet, because you have 2 tasks types (cpu and memory) i=
n each task, give a
change to the kernel to context-switch those tasks by putting at least 2 =
threads on it. Modern
cpus will separate memory instructions from pure cpu instructions, like f=
loating point
calculations, although  it eventually need memory access.

e) scale up the number of thread in the file reader with the number of ph=
ysical disk AND
independant you have (raid doesn't count, it is still one disk, virtually=
.)

f) scale up the parser/matcher with the number of CPUs you have, includin=
g the hyperthreaded
factor.

g) if you have network i/o, add a stage with one thread per network card =
(per irq, actually)

Note: this is the theoritical definition of your app requirements. You ma=
y find better performance
with tuning one at a time the number of threads / staged tasks.

I believe you don't need aDisk (imho, I don't know who ever will...)
FYI: there is no such thing as selectable channels for disk NIO in java -=
- not yet.
So it is quite pointless to give more than one thread to file reading.

I believe that your application will be the fastest on 3 threads (assumin=
g a minimalistic
single-disk, hyperthreaded single cpu machine)

Good luck.


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around=20
http://mail.yahoo.com=20


-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practic=
es
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & Q=
A
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf