Re: time budgets and continuous integration

Hi, I do something along these lines. One of our constant integration
jobs (we use TeamCity now, after switching from CC.Net) builds our
games, brings the game data up to date then launches the game. We
implemented a testing framework in the game that will cycle through the
levels, one by one, and after each one loads it dumps out a text file
with information such as average frame rate, memory used by various
systems, time taken to load, etc. TeamCity also allows you to send
statistics to its database, which it will then graph up for you (I like
using this to monitor code size for text/data/etc. so I can see if
someone commits some bloat, accidentally or not - but it can equally be
used to plot average frame rate from build to build). We have a test
harness will launches the game tests; the harness is responsible for
relaunching the game test when it crashes, and picking up the messages
the game sends (messages such as callstacks on crash, crash dumps, error
messages and so on).

We actually have quite an extensive range of tests we run, for example
another job will build all of our engines (PC/PS3/360/Wii - PS2 and PSP
not long ago too), all of our tools (exporting from Maya, platform
specific converters, plugins for Photoshop, and so on), and then runs a
series of tests that involve taking Maya scenes and generating platform
specific data, which we then load up in our engines (per platform), and
then generate a screen, which is then diffed against some reference
images, to ensure everything keeps working. It is amazingly effective,
and I cannot imagine living without it now we have it - as it is
constantly picking up 'knock on' type errors that would have otherwise
slipped in. This particular job generates a HTML report that shows the
results of all the tests, with the image vs reference image vs diff
image, and other information about how to recreate the test and so on.

We also have TeamCity hooked up with our source control (AccuRev) in
such a way that on a lot of streams, the way code flows up the tree
(promoted) is by passing TeamCity jobs. We use this to sync code between
different teams (we have multiple projects on going here), so
essentially the code one team is working on has to pass a lot of compile
tests, render tests, tools tests, game tests, before it gets promoted
high enough for other teams to be exposed to the code. We currently do
not stop code flowing based on frame rate, or code size, we have the
functionality there to do it, but various issues make it a little
tricky, not least of all because there are a lot of legitimate reasons
why there might be an increase in code size, or drop in frame rate and
so on. Just in case you aren't aware, AccuRev is a source control system
that is based on a tree, so you can have different 'streams' of code at
different levels, with code automatically flowing down the tree - I
thought I should mention that to make the above paragraph make more
sense to those that haven't used AccuRev or something simliar.

There are loads of areas we could improve on, but I have been working on
our build system here for a few years now, and it is definitely starting
to come together. We don't always use CI as a way to stop 'bad code' or
'bad data' getting pushed around, but we do use it as a system to keep
an eye on things, it provides an excellent way to find out exactly when
the frame rate halfed, or the level memory doubled, of the scene got
darker and so on.

Steve


-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of
Richard
Sent: 02 February 2010 16:13
To: [email protected]
Subject: Re: [Sweng-Gamedev] time budgets and continuous integration


In article
<[email protected]>,
    Hernan Saez <[email protected]>  writes:

> Now, how do you warrantee a constant framerate by using CI? Is it
possible?
> What tools are there? Sorry, I don't really have much experience on
this.

Go back to the start of this thread where I outlined a way to do this.

All you're doing is treating performance as another testable
requirement.
-- 
"The Direct3D Graphics Pipeline" -- DirectX 9 draft available for
download
 
<http://legalizeadulthood.wordpress.com/the-direct3d-graphics-pipeline/>

      Legalize Adulthood! <http://legalizeadulthood.wordpress.com>
_______________________________________________
Sweng-Gamedev mailing list
[email protected]
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.
com

_______________________________________________
Sweng-Gamedev mailing list
[email protected]
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com