Re: JDOM2 and Performance.

Hi

Ah, I see, I just wasn't following the code structure very well. It's tricky to read as a diff :-)

I have no problem with your code, it looks well written.

We use apache stuff extensively here at work because the license is business friendly.

Regards, Noel Grandin

Rolf Lear wrote:
> Hi Noel
>
> Thanks for that.
>
> It comes out in the numbers, but,  for the record, I am doing something very similar to that.... only the structures
> are slightly (very) different.
>
> I do a bunch of GC's, and I do one in a different thread with the current thread sleeping, then I repeat the GC's
> until the size becomes 'stable' at a change of less than 128 bytes.
> https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R33
>
> I do a complete once-through of the test suite to warm things up.
> Each once-through runs the code through 6 times (hmmm... I thought it was 12, but that was something else I did
> yesterday). Each of the actual tests 'exercises' the code repeatedly because it's all sort of loop-based code
> (parsing, scanning, etc.).
>
> Anyway, the output of the 'warmup' run is always much slower than the remaining 5 'real' runs, and I do the 'real'
> runs multiple times to ensure there is some stability.
>
> What you see in the web-page is the result of what I believe to be fully JIT-compiled and 'clean' and 'reliable
> enough' for the purposes I want.
>
> I know that the Java VM testing is 'tricky' when it comes to performance, and as such I understand that it's easy to
> get things wrong, and I'll spend more time looking at it to ensure I'm doing the reasonable thing, but, are you
> suggesting that the code I am running is not actually getting reliable results?
>
> The code is structured differently to what you have suggested below, but, the entire 'main' loop is warmed up:
> https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R124
>
> Then, the main loop is run 5 times, and I visually inspect the numbers to ensure that they are consistent:
> https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R135
>
> Between each 'test' I do a full GC with 'bells and whistles'
> https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R160
>
> It is quite obvious that the runs that come out of the 'real' loops are optimized, cached, etc.
>
> What is not clear is whether the optimizer has completely compiled out some of the code. I have tried to ensure that
> it does not by doing some sort of test on each element so that it is not completely ignored.
> now that I think about it though, maybe the 'devnull' Writer is too 'light' and the optimizer may have completely
> skipped it entirely..... and the whole XMLOUtputter code with it.... I will check.
>
> So, I appreciate the insight, and I will play around with things to see if increasing the number of warmup and actual
> 'real' runs changes the numbers.
>
> I'll look in to making sure that some of the code is not being optimized out completely.
>
> But, my code already is doing pretty much exactly what you are suggesting... (it does not calculate the deviation, but
> it does ignore the fastest and slowest run.....).
>
> In fact, it does more because it then repeats the exact same loops multiple times to ensure the averages remain
> consistent over runs (as it happens, it essentially does 20 'runs' of the code to get the results - 5 loops of 6 runs
> but the 6 only counts as 4 because the best and worst are eliminated).
>
> Have you got specific concerns about the code? Did you run it? Do you think the results are 'wrong'?
>
> Thanks for the insight in to the commons-math code. I'm always 'discovering' more and more 'stuff' in commons code. I
> have some 'stuff' I've done at work I am trying to convince my boss (actually legal&compliance) to let me use in JDOM,
> but it's the sort of thing that belongs in a 'commons' type location, not JDOM....
>
> Rolf
>
> On 14/10/2011 4:08 AM, Noel Grandin wrote:
>> Hi
>>
>> Performance testing on the Java VM is tricky.
>> To avoid getting caught out by cache-hot/cache-cold and JIT vs. not-JIT things, it's preferrable to do something like
>> this in PerfTest#timeRun(Runnnable)
>>
>> // warm up the caches and get the JIT going
>> for (int i=0; i<10; i++) {
>>    runnable.run();
>> }
>>
>> // give the JIT time to run, and get GC to run - GC can be stubborn sometimes
>> for (int i=0; i<3; i++) {
>>    Thread.sleep(100);
>>    System.gc();
>> }
>>
>> // need 20 runs to get a decent average and standard deviation
>> ArithmeticMean mean = new ArithmeticMean(); // these two classes are in jakarata-commons-math
>> Variance deviation = new Variance();
>> for (int i=0; i<20; i++) {
>>   long time1 = System.currentTimeNanos();
>>   runnable.run();
>>   long time2 = System.currentTimeNanos();
>>   mean.increment(time2 - time1);
>>   deviation.increment(time2 - time1);
>> }
>>
>> System.out.println("result  = " + mean.getMean() + " +- " + deviation.getVariance());
>>
>> Regards, Noel Grandin
>>
>> Rolf wrote:
>>> Hi all.
>>>
>>> I have put together a 'simple' system for measuring the relative performance of JDOM2. The idea is that I need to
>>> know whether I am improving or breaking JDOM performance as the code evolves.
>>>
>>> Currently the metric code is only useful of you compare apples to apples, and, in this case, it means processing a
>>> single (medium size) XML document on my laptop, yada-yada-yada. But, it should be useful as a tool to get a feel for
>>> what a code-change does.
>>>
>>> Already I can see that I probably have an issue in the SAXHandler (possibly an issue in JDOM-1.1.2 actually) because
>>> 1.1.2 is 5-times faster in that area than JDOM2.
>>>
>>> I have put together a results page here:
>>>
>>> http://hunterhacker.github.com/jdom/jdom2/performance.html
>>>
>>> It also describes what each test does. If you are interested in seeing the code and what it does have a look here
>>> (it is not well documented and it is still perhaps evolving):
>>>
>>> https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb
>>>
>>>
>>> Rolf
>>> _______________________________________________
>>> To control your jdom-interest membership:
>>> http://www.jdom.org/mailman/options/jdom-interest/[email protected]
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------------------------------------------------
>> Disclaimer: http://www.peralex.com/disclaimer.html
>>
>

Disclaimer: http://www.peralex.com/disclaimer.html

_______________________________________________
To control your jdom-interest membership:
http://www.jdom.org/mailman/options/jdom-interest/[email protected]