Re: FF3.1beta3 in Peacekeeper performance test

Mike Cramer wrote:
> "..Peacekeeper measures your browser's performance by testing its 
> JavaScript functionality. ..."

That's not what it tests.

> The following features are tested:
> http://service.futuremark.com/peacekeeper/faq.action

Right.

> As result they show this
> http://www.heise.de/mobil/Neuer-Benchmark-fuer-Internetbrowser--/zoom/134567/0 

Which is semi-meaningless, since they don't ever explain how they 
compute those numbers.

 > which means that FF 3.1b3 is only ~1/2 as fast as safari 4.0

No, it means their scoring function, which they never explain, assigned 
it half as high a score.  What that means in practice depends on what 
that function is.  If you can get them to tell you that, I'd love to 
know the results.

> I tried this test on my mac pro (2,4ghz, 4gb) and I got the following 
> results:
> Safari 3.2.1   791points
> FF3.1beta3     592points

Which means what?

> Has anyone an explaination why FF3.1 performs so poor in this test

It's pretty easy to write tests on which FF3.1 performs worse than 
Safari 4 (e.g. dromaeo).  If you want me to explain numbers that are 
being computed by some hidden formula, I think you're out of luck. ;)

For what it's worth, I spent a few hours on Friday trying to get this 
test suite into a shape where I could run one test at a time (and hence 
profile the tests to see where we spend time).  I gave up when I 
couldn't even get any of the tests but the array tests from their web 
server: either they send different tests for the same URI based on 
cookies, or their code is cleverly obfuscated beyond my ability to 
comprehend.  Or both.

Given that they don't publish sub-scores for the parts of the benchmark, 
don't provide their benchmark's tests in a way that can be read (much 
less run individually), and don't explain how their final score is 
computed from the sub-scores, the credibility of these folks is about 0 
for me right now.  They could just be making all the numbers up using a 
random number generator; I have no way to tell.

-Boris