Re: Architecting and programming a graphics engine in C++ & OpenGL

So much good advice on this thread. I'm loathe to add to it, but I want
to highlight a "big picture" trend that might help  give you context and
inform how you proceed, especially if you decide to focus on rendering
architecture and write your own from the ground up. Here it is...

 

Over time rendering will be less and less about the care and feeding of
GPUs and more about (embarrassingly) parallel computation. For the last
decade and change, the dominant model in graphics has been "CPU makes a
big list of requests, GPU turns them into pixels." That's still sort of
true, but becoming less so every day. As CPUs gain cores and as GPUs
become increasingly programmable and CPU like, rendering is becoming all
about deciding how you divide up work across a series of (usually
heterogeneous) processors. At the start of this console generation,
people started spinning the renderer off onto a separate thread from the
main game. A few years later, console and PC engines started to use
multiple threads just for rendering (usually some sort of parallel
command buffer generation). In the past few years (earlier on the PS3),
people started adopting hybrid CPU/GPU techniques, moving "GPU work"
like post-processing, skinning, anti-aliasing, lighting, occlusion, etc.
off the GPU and onto the CPU (especially SPUs). When you look at the
really cutting edge stuff, especially in the world of DX10/11, you see
even more of complex CPU/GPU integration. Over time, GPUs and CPUs will
be even harder to tell apart and provide more and more options for how
do divide the work between them. It's an awesome, but also scarily
complex and open ended future.

 

The one thing I can say with confidence is that to be competitive,
future renderers will need to do the majority of their computation in
parallel across multiple processors. And that's where the really
important, but also really not easy problems are. As we all know,
parallel programming is really tricky, doubly so if you are starting
with a big renderer that was never designed with it in mind.
Fortunately, there's tons of great literature on the topic (much of
which like Data Oriented Design and Mike Acton's publications have
already been mentioned). It's definitely something you should be
thinking and reading about if you want to try your hand at writing your
own renderer. Of course, there's a universe of places to start and
things to explore, but if your goal is to land a job at a high end game
dev studio, there are, for my money, few more impressive things you
could demonstrate than a renderer that is thoughtfully and productively
parallel. 

 

Good luck with whatever you decide to do!

-Nathan

 

From: [email protected]
[mailto:[email protected]] On Behalf Of
Tommy Brett
Sent: Friday, March 18, 2011 11:26 AM
To: [email protected]
Subject: Re: [Sweng-Gamedev] Architecting and programming a graphics
engine in C++ & OpenGL

 

 

On Fri, Mar 18, 2011 at 1:30 PM, Massimo Del Zotto
<[email protected]> wrote:

2011/3/18 Tommy Brett <[email protected]>

	... pursue my original intended career as a games programmer,
... it will render planets with atmosphere and real time shadows, or die
trying, and in any case will serve as my main portfolio piece when I
eventually start applying for positions. 

Hello Tommy, I also started working on my engine for this same reason
("as portfolio"). I don't know how other people are set but I suppose
you will do more or less what I've been doing up to now so I hope you
will find my advices useful.

 

	What are the best practices for building a graphics engine in
C++? Exactly how OOP should I go?

A stupid answer would be "as far as you need but no more". When I
started working on my system I tried to make pretty much of everything
OOP. I eventually figured out that OOP is not always the best tool for
the job. Some systems are natively data-oriented and some will
inevitably need reinterpret_cast (oh, the horror). You will have to get
your hands dirty. In some cases, having less abstraction can be helpful.
In general, my design rarely relies on objects but is really based on
"subsystems" which (as an implementative detail) emerge from aggregating
objects. Rather than objects, I really think in terms of "protocols"
(quite similar to "network protocols"). You might say this is exactly
what objects are made for. Yes but the whole point is: don't be afraid
of steering away from OOP every once in a while.

The more you go "high level" the more OOP will result useful but at low
level, bad surprises might await you and the systems have to be coupled
somehow anyway.

 

This is why I mentioned 'how much is too much', because from my own
experimentation and from reading previous gamedev mailing list
discussions, OOP is clearly not always the best method for graphics
engines.

 

	As for how to write good C++ engines I'd suggest to remember
that while C++ has no explicit "interface" classes, they are indeed
there and they will help you a lot (the compiler also seems to like them
pretty much IMHO). Many people seem to forget this so just in case...

 

Interfaces are definitely something I've gotten comfortable with using
in my AS3 programming, it's a neat alternative to deriving from a base
class.

 

	 

	 

		How should I handle the use of multiple shaders on
chunks of geometry?

	You're not going to solve this right now. Use iterative design.
You really have no choice, this would cause you to meltdown your head. I
don't quite figure out what you mean by "multiple shaders on chunks of
geometry". Are you referring to multiple materials? Are you referring to
multipass techniques?

	Your first step is surely to figure out what you mean by
"shader" because when you have an engine, "shaders" as most APIs provide
will probably lack something here or there. For example my "engine
shaders" provide special binding points to link engine logic. Say you
want to make everything fade to black progressively as player's health
drops to 0.

	They also include additional information which are not part of
the "API shader" such as blend mode to use (this is similar to DirectX
"effects") and other additional stuff.

	Now, those requirements weren't introduced in a day but evolved
out of the needs. This is why some people advises to "write games not
engines"... you really need to understand what you need for real before
you can hope to understand a way to solve the problem in a reusable,
"engine" way.

 

Both materials and multi pass rendering techniques. I suppose it's a
broad question and I think you're right with the 'just code it'
approach. What I meant was, given a collection of meshes, each with
their own textures and effects, what would be the best way to organize
those chunks of geometry data, how would I manage engine state from
chunk to chunk? E.g. switching shaders, would switching shaders mid
render even be viable?

 

	 

	 

		Currently my idea for a graphics engine involves
something like an engine class that takes render packets of data, has
some sort of ability to sort them into an order that best facilitates
speedy rendering, and changes its state depending on the information in
the packets.

	This makes me think about 

	http://realtimecollisiondetection.net/blog/?p=86

	but you probably don't mean this kind of low-level sort... or do
you? According to what you write next, it seems you're thinking at a
much higher level which apparently seems to be scenegraph's
responsability. I don't know.

	As a final note here, the link I provided is effective but until
you don't finalize the systems involved it creates a strong dependancy
which will slow you down (where in the hierarchy should I put this
batch?). Start easy first! Leave the order emerge from your design
first. Benchmark. Refine. Repeat. Be aware of premature optimization.
When in doubt write easy to understand code, the bugs will generally
emerge some time later, often because of change in requirements, and you
want the code to NOT bite you. You probably have plenty of perf anyway
and your first objective is get the work done.

	 

	 

		So I suppose what I'm looking for are book suggestions,
or perhaps open-sourced graphics engines that I could look at and learn
from ... the trouble is there aren't any such positions available where
I live, and relocation would be difficult to justify based upon the
possibly low salary expectations of a junior position

	It is always a good idea to consider already available engines.
You really have to look around you (at least your state as very minimum)
and  figure out what to do as there's a serious social aspect to
consider here.

	From the few demos I've given out I can say that presenting
something which is 100% yours is fairly impressive, especially for small
realities, it seems that the idea of having somebody which can work out
problems top-to-bottom is very appreciated. Problem is that it takes a
lot - a lot - of effort to get to the first demo. It still takes a lot
to go to the second as you will keep working on it which in fact will
result in less demos (or reuse of the very same one).

	Second problem is, as other people have said, turning a pre-made
engine into something yours is indeed valuable skill as well, which is
likely much more important if you're going to apply in consolidated
companies. Not my case here... :-(

	 

	Pre-made engines give you a great kickstart (it's awesome for
morale)! On the cons, I have to admit some projects I've looked at
seemed to be halting after a while. Let's not hide the fact: once you
choose an engine you choose a line of thinking and this implies you will
think your design in a certain way. Which is easier. The day you get a
problem which is not directly solvable by the engine, you start having
issues.

	I say this because a friend of mine had exactly this problem.
Its initial enthusiasm of having "parallax mapping and shadows
everywhere" quickly faded when he figured out the hard truth: the engine
prevented him from understanding the problems involved.

	Of course, if you take your time to make sure you know what
you're doing, this should not be a problem and you'll get best of both
worlds. I just wanted to drop this warning.

 

Your reply and other people's has given me a perspective that I didn't
consider before - taking someone elses engine and modifying it to suit
my own needs would certainly be an exemplary show of skill, especially
for studios that use 3rd party tools themselves

 

	 

	Experimenting a bit with modding games such as Unreal or Quake
is very useful IMHO. If you haven't tried making a Quake map then you
should really try right now. Take a look at the formats. Do some black
box analysis. Take your time... if you can afford it.

	Also try to figure out what assets you have access to and make
sure you understand the data encoded in ("know your data"). I've
scratched my head more than a year in making a shader work according to
a "reference implementation" before figuring out that the "reference"
asset pipeline is way smarter than I originally believed, let's say it
works around data format's dumbness...

 

Actually it's funny you mention Quake mapping, when I started fiddling
with games and programming, I started with programming q-basic and
making Doom maps, which eventually progressed to Quake and Half-Life (I
rather enjoyed Worldcraft, even though it took forever and a day for
them to release decent texture mapping tools) and ignited my desire to
become a programmer (luckily I was young enough at the time that abusing
goto didn't stick with me for long). It's also why I have Michael
Abrash's Black Book, I'm fascinated by fps engines and how they
calculate potentially visible data from a vast map.

 

	 

	 

	I hope this helps,

	Massimo

 

Greatly so, thank you.

 

	
	_______________________________________________
	Sweng-Gamedev mailing list
	[email protected]
	
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.
com

_______________________________________________
Sweng-Gamedev mailing list
[email protected]
http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com