Re: [benchmarks] Project plan

Hi Alexander,

On Thu, May 21, 2009 at 6:07 PM, Alexander Hjalmarsson <[email protected]> wrot=
e:
> The deltablue and raytrace benchmarks are finished. The Richards test is =
almost finished, since it has some bugs left that needs to be fixed. I'm th=
inking of using both phpWhirl and j4p5 as a part of the test since both of =
these use tests eval and some other functionality. I would also like to inc=
lude some smaller and simpler tests that I believe are good for some purpos=
es. In example, a test for recursive functions (could be Fibonacci or Acker=
mann), XML parsing and array performance (maybe by using the gauss-jordan-a=
gorithm). Any ideas here that should be added/taken away is nice to hear:)

OK, the plan sounds good, especially since most of those are ready to
go. I believe Olafur has a fix for the Richards bug; I hope so, as I
spent a few hours on it and couldn't find it.

As for the suggestion of using micro-benchmarks, I have a few
thoughts. Obviously, microbenchmarks say little about the performance
of a language. However, in the same way that each extension in PHP
provides a set of tests, it may be useful to allow each extension to
provide benchmarks. It wouldnt be up to you to create all those
benchmarks, but something simple involving the areas you suggest is a
pretty decent idea. I believe roadsend has a few extension specific
benchmarks for just that purpose (see
http://phc.googlecode.com/svn/trunk/test/subjects/benchmarks/roadsend/bench=
marks/tests/)
, we can probably use them.

Note that the important part here would be to define a simple way for
extension authors to provide benchmarks. In the long term, this might
integrate with run-tests.php and the phpt format, but for now I would
go for the simplest option that works.


> To the 2nd question. I have minimal experiences from this so I would real=
ly like some guidance here. What tools can be used and what should not be u=
sed?
There are two areas we wish to measure: run-time speed and
memory-usage. For run-time speed, we wish to say _why_ the speed has
changed. So what I would measure is the following:
- Actual run-time: using the `time` command
- runtime, minus startup time: using getmicrotime
- memory usage: using the valgrind 'massif' profiler:
http://valgrind.org/docs/manual/ms-manual.html
- detailed counts of actual execution (cache misses, branch
mispredictions, etc): using the valgrind 'cachegrind' tool:
http://valgrind.org/docs/manual/cg-manual.html

For the latter, I wrote some scripts to and compare of callgrind,
which may be useful to you. See the valbench* scripts in
http://phc.googlecode.com/svn/trunk/test/framework/bench/.



> To the web application deadline, I find it hard to put that on a schedule=
. As you say, there's lots of work to be done and it might be more realisti=
c to try to get one application fully finished. I don't have any help more =
than the PHP community. Is there any "old" application or something that I =
might take a look at for directions of how it should be done? Has any work =
been done in this part?

Yes, this is tricky. The benchmarks that are listed in the RFC, such
as Rubbos and SPECweb, have frameworks for measuring requests per
second. THe first is open source, so you can take a look. The second
is not, but if you would like to take a look, I can give you access to
a machine with it installed (obviously, you should not copy code from
it).


> I starting lightly today with some code writing and will keep going on an=
d off as my studies will finish (I have two more exams, the last one in Jun=
e 1th). I hope I will be able to get a nice start with some efficient work =
and not only problems.


It sounds like you have a good idea of what you're going to work on.
The latter part might be tough, but I expect be quite successful with
this project. Of course, the sooner code appears the better, so do be
sure to work out of the CVS repository.

Thanks,
Paul



> -----Original Message-----
> From: Paul Biggar [mailto:[email protected]]
> Sent: den 21 maj 2009 17:42
> To: Alexander Hjalmarsson
> Cc: Nuno Lopes; [email protected]
> Subject: [benchmarks] Project plan
>
> Hi Alexander,
>
> I see that you added a project plan on
> http://wiki.php.net/gsoc/2009/benchmarks. I have some questions and
> comments:
>
>> Late may and early June: Research and start coding on the CLI-version of=
 the benchmark suite. This involves porting necessarily benchmarks and plac=
e them in the benchmark suite.
>
> 1. There are a number of benchmarks ported already. Which do you intend t=
o port?
> 2. What are you planning to do for the 'suite' part of the benchmarks?
> That is, we'll need a driver to test run-time/memory usage/cachegrind
> stats/etc.
>
>
>> Mid june - late june: The CLI version of the benchmark suite should have=
 a clear shape.
>
> I think this date is quite pessimistic. Since there isnt a huge deal
> to be done for the CLI benchmarks, you should move this up.
>
>
>> Early July (Mid term): The CLI-application should be totally complete an=
d work should be focused on the web applications such as phpBB, worpress, j=
oomla and/or SugarCRM.
>> Early august: Applications should start to pile up. Hopefully there shou=
ld be plenty of them here, but it will depend on the difficulty to modify t=
hem and the size of them.
>
> By contrast, this is quite optimistic, to my mind. What exactly is
> planned here? Are you anticipating help from outside sources?
> Otherwise, I'm not sure that converting even 1 application in 3 weeks
> in doable (bear in mind that web apps need test loads).
>
>
>> August 10th - August 17th: Documentation and bug checking.
>
> Please document as you go. It will help us know what's going on.
>
>
> Other than that, a decent plan, but it could do with more detail. When
> did you intend to start?
>
> Thanks,
> Paul
>
>
> --
> Paul Biggar
> [email protected]
>
>



--=20
Paul Biggar
[email protected]