Let me suggest an alternative.
=
Within the Phoronix Test Suite ecosystem, we have a
continious
= integration/validation system called Phoromatic
([1]http://www.phoromatic.c= om/). We have a brief theory of
operation
on it captured
=
[2]https://docs.google.com/present/view?id=ds439pg_42hpg57m86
. You can see a Linux oriented tracker available at
[3]http://ph oromatic.com/kernel-tracker.php (the nice graphs are when
you sel ect 180 days and the ION 330 systems). We've re-assigned the
sy stems to other projects so it is no longer being updated, but it
serv= ed our purposes.
This system will allow you to just de= dicate a machine to be
updateable and pick up the directions for whic= h test to be run.
The
FreeBSD project (or contributors) would m= aintain the slave or
test
machines. The test suite would be sel= ected and managed by
the
FreeBSD project or contributors. The g= lue code to emit
triggers
(possibly SVN, git or other submission, or = even just
daily) and the
scripts to update the systems would also be maintained by the
FreeBSD community.
Phoronix Me= dia would be happy to host it on Phoromatic.com
(we've
played with ho= sted with project branding) and provide the
data store
and the analyt= ics. We'd also be willing to make
enhancements to
support the F= reeBSD project.
This should solve the "I don't have ti= me to maintain a
automated
test infrastructure". You don't need= to, just write the
glue
scripts, and dedicate a couple of machines.&= nbsp; I believe
FreeBSD
vendors like ixSystems may be able to support= this effort
with a
dedicated machine. You can have as many mac= hines as you
like
demonstrating AMD/Intel/32/64/large mem/low mem,etc= .
The comments around interactivi= ty can also be measured to
some
extent. We have the model of a = "monitor". This can be
configured
to determine jitter around a = number of system
variables and to
possibly inject actions to measure = impacts.
We're more than happy to work with you guys, = and are willing to
help
do a lot of the infrastructure and automation= lifting.
Regards,
Matthew
On 12/22/11 8:56 AM, Adrian Chadd wrot= e:
Guys, girls, fuzzy creatures,
This is by far the best example of a constructive email in this entire thread.
If people would like to help, Erik here is exactly the kind of person
with exactly the kind of software that needs a hand.
I think enough philosophizing has been done - now we have questions
that need answering; theories that need testing. And that requires,
you know, coding. :)
The best thing right now would be for *BSD people to pick up the
Phronix test suite, try to compile/run it, and provide feedback. Do
your own benchmarks on your own hardware and report back the results.
That's how we fix the "benchmarking problem." We don't fix it by
armchair philosophy, we fix it by getting our hands dirty. :)
2c,
Adrian
On 22 December 2011 03:21, Erik Cederstrand [4]<erik@
cederstrand.dk> wrote:
=
Den 21/12/2011 kl. 19.48 skrev Alexander Leidinger:
And related to the subject: wasn't it you who developed the automatic benchmarking stuff? If yes, why not make it available? If you don't have he resources, I offer my help to make it available somewhere.
Yes, that's me. I'm mostly out of time right now, but I'd like to offer help if someone wants to pick up the project.
For those who haven't heard about it, it's a system designed specifically to track performance of FreeBSD over time by comparing revisions of FreeBSD, everything else being equal. It consists of a tinderbox-like build script for a build server, a script to install FreeBSD and run benchmarks on at least one slave, and a database-backed website to aggregate and visualize results.
The framework does work as-is, but it really needs to be updated: convert the scripts to use the SVN repo instead of CVS, improve visualization and search on the web fronted, and improve the benchmarking script so it's easier to extend. I don't have hardware available to run the benchmarks, but I think there's hardware available in the FreeBSD cluster.
Here's a link to the source code: [5]http://dev.affect-it.dk/tracker.tgz
And to my thesis describing how it works: [6]http://dev.affect-it.dk/tracker.pdf
Just send me a mail if you're interested.
Thanks,
Erik_______________________________________________
[7]free
[email protected] mailing list
[8]http://lists.freebsd.org/mailman/
listinfo/freebsd-performance
To unsubscribe, send any mail to [9]"[email protected]"
_______________________________________________
[10]freebsd-performance@freeb
sd.org mailing list
[11]http://lists.freebsd.org/mailman/listinfo/freebsd-per
formance
To unsubscribe, send any mail to [12]"[email protected]"
=
References
1. 3D"http://www.phoromatic.com/"
2. 3D"https://docs.google.com/present/view?id=ds439pg_42hpg57m86 3. 3D"http://phoromatic.com/kernel-tracker.php"
4. 3D"mailto:[email protected]"
5. 3D"http://dev.affect-it.dk/tracker.tgz"
6. 3D"http://dev.affect-it.dk/tracker.pdf"
7. 3D"mailto:freebsd-performance@freeb 8. 3D"http://lists.freebsd.org/mailman/listinfo/freebs 9. 3D"mailto:[email protected]"
10. 3D"mailto:[email protected] 11. 3D"http://lists.freebsd.org/mailman/listinfo/freebsd-per 12. ="mailto:[email protected]"
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[email protected]"
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.