Re: JDOM2 and Performance.

Hi Noel

Thanks for that.

It comes out in the numbers, but,  for the record, I am doing something 
very similar to that.... only the structures are slightly (very) different.

I do a bunch of GC's, and I do one in a different thread with the 
current thread sleeping, then I repeat the GC's until the size becomes 
'stable' at a change of less than 128 bytes.
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R33

I do a complete once-through of the test suite to warm things up.
Each once-through runs the code through 6 times (hmmm... I thought it 
was 12, but that was something else I did yesterday). Each of the actual 
tests 'exercises' the code repeatedly because it's all sort of 
loop-based code (parsing, scanning, etc.).

Anyway, the output of the 'warmup' run is always much slower than the 
remaining 5 'real' runs, and I do the 'real' runs multiple times to 
ensure there is some stability.

What you see in the web-page is the result of what I believe to be fully 
JIT-compiled and 'clean' and 'reliable enough' for the purposes I want.

I know that the Java VM testing is 'tricky' when it comes to 
performance, and as such I understand that it's easy to get things 
wrong, and I'll spend more time looking at it to ensure I'm doing the 
reasonable thing, but, are you suggesting that the code I am running is 
not actually getting reliable results?

The code is structured differently to what you have suggested below, 
but, the entire 'main' loop is warmed up: 
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R124

Then, the main loop is run 5 times, and I visually inspect the numbers 
to ensure that they are consistent: 
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R135

Between each 'test' I do a full GC with 'bells and whistles'
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R160

It is quite obvious that the runs that come out of the 'real' loops are 
optimized, cached, etc.

What is not clear is whether the optimizer has completely compiled out 
some of the code. I have tried to ensure that it does not by doing some 
sort of test on each element so that it is not completely ignored.
now that I think about it though, maybe the 'devnull' Writer is too 
'light' and the optimizer may have completely skipped it entirely..... 
and the whole XMLOUtputter code with it.... I will check.

So, I appreciate the insight, and I will play around with things to see 
if increasing the number of warmup and actual 'real' runs changes the 
numbers.

I'll look in to making sure that some of the code is not being optimized 
out completely.

But, my code already is doing pretty much exactly what you are 
suggesting... (it does not calculate the deviation, but it does ignore 
the fastest and slowest run.....).

In fact, it does more because it then repeats the exact same loops 
multiple times to ensure the averages remain consistent over runs (as it 
happens, it essentially does 20 'runs' of the code to get the results - 
5 loops of 6 runs but the 6 only counts as 4 because the best and worst 
are eliminated).

Have you got specific concerns about the code? Did you run it? Do you 
think the results are 'wrong'?

Thanks for the insight in to the commons-math code. I'm always 
'discovering' more and more 'stuff' in commons code. I have some 'stuff' 
I've done at work I am trying to convince my boss (actually 
legal&compliance) to let me use in JDOM, but it's the sort of thing that 
belongs in a 'commons' type location, not JDOM....

Rolf

On 14/10/2011 4:08 AM, Noel Grandin wrote:
> Hi
>
> Performance testing on the Java VM is tricky.
> To avoid getting caught out by cache-hot/cache-cold and JIT vs. 
> not-JIT things, it's preferrable to do something like this in 
> PerfTest#timeRun(Runnnable)
>
> // warm up the caches and get the JIT going
> for (int i=0; i<10; i++) {
>    runnable.run();
> }
>
> // give the JIT time to run, and get GC to run - GC can be stubborn 
> sometimes
> for (int i=0; i<3; i++) {
> Thread.sleep(100);
> System.gc();
> }
>
> // need 20 runs to get a decent average and standard deviation
> ArithmeticMean mean = new ArithmeticMean(); // these two classes are 
> in jakarata-commons-math
> Variance deviation = new Variance();
> for (int i=0; i<20; i++) {
>   long time1 = System.currentTimeNanos();
>   runnable.run();
>   long time2 = System.currentTimeNanos();
>   mean.increment(time2 - time1);
>   deviation.increment(time2 - time1);
> }
>
> System.out.println("result  = " + mean.getMean() + " +- " + 
> deviation.getVariance());
>
> Regards, Noel Grandin
>
> Rolf wrote:
>> Hi all.
>>
>> I have put together a 'simple' system for measuring the relative 
>> performance of JDOM2. The idea is that I need to know whether I am 
>> improving or breaking JDOM performance as the code evolves.
>>
>> Currently the metric code is only useful of you compare apples to 
>> apples, and, in this case, it means processing a single (medium size) 
>> XML document on my laptop, yada-yada-yada. But, it should be useful 
>> as a tool to get a feel for what a code-change does.
>>
>> Already I can see that I probably have an issue in the SAXHandler 
>> (possibly an issue in JDOM-1.1.2 actually) because 1.1.2 is 5-times 
>> faster in that area than JDOM2.
>>
>> I have put together a results page here:
>>
>> http://hunterhacker.github.com/jdom/jdom2/performance.html
>>
>> It also describes what each test does. If you are interested in 
>> seeing the code and what it does have a look here (it is not well 
>> documented and it is still perhaps evolving):
>>
>> https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb 
>>
>>
>>
>> Rolf
>> _______________________________________________
>> To control your jdom-interest membership:
>> http://www.jdom.org/mailman/options/jdom-interest/[email protected]
>>
>
>
>
> ------------------------------------------------------------------------
> Disclaimer: http://www.peralex.com/disclaimer.html
>

_______________________________________________
To control your jdom-interest membership:
http://www.jdom.org/mailman/options/jdom-interest/[email protected]