Re: Reflection in game engine (c++)
Mike Shaver <[email protected]> Mon, 18 Mar 2013 17:26:34 -0700
| Newsgroups | gmane.games.devel.sweng |
|---|---|
| Message-ID | <[email protected]> |
--===============1521491893== Content-Type: multipart/alternative; boundary=Apple-Mail-6D92B325-D659-444E-A371-29BF542663DF Content-Transfer-Encoding: 7bit --Apple-Mail-6D92B325-D659-444E-A371-29BF542663DF Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable When I did this for a non-game app, I defined it with self-aware, strong-AI m= acros, and compiled the resulting data into a shared library. The app (game)= linked against the library, as did one important tool: the exporter. It was= a wee little program that dumped the compiled metadata to an interchange fo= rmat. Today I would choose JSON or XML. The build process would regenerate m= etadata.json whenever metadata.so changed, and extending it to multiple libr= aries/output files was straightforward. Runtime reflection formats have many constraints that tool-interchange forma= ts don't, and tool formats often want things the runtime abhors. Put the lin= e-numbers in the output and remix with git-blame output to tell you who last= changed an entity? Sure. Web explorer of the resources, without linking the= parser library into PHP...I'll be over here. I would similarly not use Collada files at runtime. phone-typed On Mar 18, 2013, at 4:30 PM, Adrian Stephens <[email protected]> wrote:= >=20 > I'm glad we're still talking about this - I think it's important and I wan= ted to add my own observations: >=20 > The macro/template approach to reflection only applies to one side of the d= ata/code interface. If all you need to do is serialize runtime data to be r= ead back in by runtime code, then you're ok; but if you want to access that m= etadata from an external tool, you either have to recompile the tool or plug= in when anything changes, or write something that parses that code. >=20 > If you're going to have to write a parser anyway, it seems to me you migh= t as well use it for both code and data views. Which means you can use a mo= re natural syntax for your structure definitions that even non-programmers c= an use. >=20 > To map this back into your C++ code, you can pretty trivially write out he= ader files containing typedefs that correspond to the metadata, and generate= whatever you need to allow the runtime to interpret serialized data. This a= lso address the separation of code and assets: we don't want artists having t= o sync the code, and this keeps the data definitions with the assets that th= ey define. >=20 > This is what I do, anyway. I have a data language, which defines types, a= nd which can also be used to define data, and all data is stored using these= definitions. This includes bitmaps, sounds, models, etc. The data languag= e is separated from the data representation so this doesn't imply that bitma= ps are stored in a script-like array of colors - but it does mean that tools= can use the reflection system on more classes of data than just small gamep= lay entities. And when distinctions like that are removed, and all things a= re treated the same way, lots of clever things just fall out. >=20 > Some other random thoughts: > As far as I can see, all C++ needs to allow full reflection is some sort o= f fieldsof(T) operator that enumerates the fields of a structure. e.g. (typ= ed straight into email, so probably bollocks - and not representative of my a= ctual system!) >=20 > struct metadata { > enum TYPE {INT, PTR, ARRAY, STRUCT} type; > metadata(TYPE &_type) : type(_type) {} > }; > template<typename A, typename... B> struct md_fields : md<A>, md_fields<B>= {}; > template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() := metadata(STRUCT) {} }; > template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(P= TR) {} }; > template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; m= d() : metadata(ARRAY), n(N) {} }; > template<> struct md<int> : metadata { md() : metadata(INT) {} }; >=20 > template<typename T> metadata *get_md(const T &t) { static md<T> m; return= &m; } >=20 >=20 >=20 > Adrian Stephens >=20 > On Mar 18, 2013, at 10:45 AM, Douglas Cox wrote: >=20 >> That sounds pretty similar to the goals of the engine that we're working o= n -- basically faster iteration times for everyone (this includes engine pro= grammers as well). >>=20 >> All of our code (including the compiler) is written in a C#-like language= , so we have perfect reflection from that. Attributes are pretty much requi= red for providing additional information to property editors for things like= valid ranges of a field or notifications of a property change (most of whic= h can go away in Final builds). >>=20 >> Another thing to consider may be allowing for data derivation. This can s= ave quite a bit of work when the need for copies of objects come up that onl= y need one or two properties changed. If you pick a serialization format fo= r the editable data that knows about the data (fieldtype + fieldname + value= ), it makes it a lot easier to get this working. >>=20 >> In the past, we've used Macros and they worked fine. They're a bit more a= nnoying to extend with optional parameters and you have to repeat the field/= property name, but that's not terrible since you can generally get it down t= o compile-time errors if you get something wrong (other than forgetting to a= dd a new field). >>=20 >> -Doug >>=20 >>=20 >>=20 >> On Fri, Mar 15, 2013 at 3:45 AM, Tinco Andringa <[email protected]> wrote: >>> I think the point of the discussion is to not just assume C++ isn't >>> going to do it. As far as I understand the goal is to have the ability >>> to develop tools that aid programmers and artist in iterating rapidly. >>> This means for artists that they can swap artwork in at runtime or >>> quickly boot the game from a serialized state with new artwork, not >>> just on his/her own computer but also over a network to perhaps a >>> console. For programmers this may mean that they can view the game >>> state easily, launch the game in various states easily, swap in/out >>> scripts easily, maybe even change code without fully recompiling or >>> even restarting the game. >>>=20 >>> Am I missing something important? >>>=20 >>> On Fri, Mar 15, 2013 at 8:33 AM, Massimo Del Zotto <[email protected]= > wrote: >>> > I have limited experience but I have been very impressed by the separa= tion >>> > provided by scripting languages. >>> > Considering the goals: >>> > >>> > My goal is to have fast iteration times, both artists and programmers >>> > >>> > It appears to me C++ isn't going to do it. Especially on the artist si= de of >>> > the thing. >>> > >>> > By the way, I would like to have an insight on how this reflection >>> > information is to be used. If it is used to simulate duck typing I'd u= se a >>> > tool which supports duck typing natively. >>> > >>> > Massimo >>> > >>> > _______________________________________________ >>> > Sweng-Gamedev mailing list >>> > [email protected] >>> > http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryde= r.com >>> > >>> _______________________________________________ >>> Sweng-Gamedev mailing list >>> [email protected] >>> http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.= com >>=20 >> _______________________________________________ >> Sweng-Gamedev mailing list >> [email protected] >> http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.c= om >=20 > _______________________________________________ > Sweng-Gamedev mailing list > [email protected] > http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.co= m --Apple-Mail-6D92B325-D659-444E-A371-29BF542663DF Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D= utf-8"></head><body dir=3D"auto"><div>When I did this for a non-game app, I d= efined it with self-aware, strong-AI macros, and compiled the resulting data= into a shared library. The app (game) linked against the library, as did on= e important tool: the exporter. It was a wee little program that dumped the c= ompiled metadata to an interchange format. Today I would choose JSON or XML.= The build process would regenerate metadata.json whenever metadata.so chang= ed, and extending it to multiple libraries/output files was straightforward.= </div><div><br></div><div>Runtime reflection formats have many constraints t= hat tool-interchange formats don't, and tool formats often want things the r= untime abhors. Put the line-numbers in the output and remix with git-blame o= utput to tell you who last changed an entity? Sure. Web explorer of the reso= urces, without linking the parser library into PHP...I'll be over here.</div= ><div><br></div><div>I would similarly not use Collada files at runtime.</di= v><div><br>phone-typed<div><br></div></div><div><br>On Mar 18, 2013, at 4:30= PM, Adrian Stephens <<a href=3D"mailto:[email protected]">adrians@i= sopodlabs.net</a>> wrote:<br><br></div><blockquote type=3D"cite"><div><di= v><br></div><div>I'm glad we're still talking about this - I think it's impo= rtant and I wanted to add my own observations:</div><div><br></div><div>The m= acro/template approach to reflection only applies to one side of the data/co= de interface. If all you need to do is serialize runtime data to be re= ad back in by runtime code, then you're ok; but if you want to access that m= etadata from an external tool, you either have to recompile the tool or plug= in when anything changes, or write something that parses that code.</div><di= v><br></div><div>If you're going to have to write a parser anyway, it s= eems to me you might as well use it for both code and data views. Whic= h means you can use a more natural syntax for your structure definitions tha= t even non-programmers can use.</div><div><br></div><div>To map this back in= to your C++ code, you can pretty trivially write out header files containing= typedefs that correspond to the metadata, and generate whatever you need to= allow the runtime to interpret serialized data. This also address the= separation of code and assets: we don't want artists having to sync th= e code, and this keeps the data definitions with the assets that they define= .</div><div><br></div><div>This is what I do, anyway. I have a data la= nguage, which defines types, and which can also be used to define data, and a= ll data is stored using these definitions. This includes bitmaps, soun= ds, models, etc. The data language is separated from the data represen= tation so this doesn't imply that bitmaps are stored in a script-like array o= f colors - but it does mean that tools can use the reflection system on more= classes of data than just small gameplay entities. And when distincti= ons like that are removed, and all things are treated the same way, lots of c= lever things just fall out.</div><div><br></div><div>Some other random thoug= hts:</div><div>As far as I can see, all C++ needs to allow full reflection i= s some sort of fieldsof(T) operator that enumerates the fields of a structur= e. e.g. (typed straight into email, so probably bollocks - and not rep= resentative of my actual system!)</div><div><br></div><div>struct metadata {= </div><div><span class=3D"Apple-tab-span" style=3D"white-space:pre"> </s= pan>enum TYPE {INT, PTR, ARRAY, STRUCT} type;</div><div><span class=3D"Apple= -tab-span" style=3D"white-space:pre"> </span>metadata(TYPE &_type) : t= ype(_type) {}</div><div>};</div><div>template<typename A, typename... B&g= t; struct md_fields : md<A>, md_fields<B> {};</div><div>template= <typename T> struct md : metadata, md_fields<fieldsof(T)&= gt; { md() : metadata(STRUCT) {} };</div><div>template<typename T> str= uct md<T*> : metadata { md<T> t; md() : metadata(PTR) {= } };</div><div>template<typename T, int N> struct md<T[N]> : met= adata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };</div><div>t= emplate<> struct md<int> : metadata { md() : metadata(INT) {} };= </div><div><br></div><div>template<typename T> metadata *get_md(const T= &t) { static md<T> m; return &m; }</div><div><br></div><div><= br></div><div><br></div><div>Adrian Stephens</div><br><div><div>On Mar 18, 2= 013, at 10:45 AM, Douglas Cox wrote:</div><br class=3D"Apple-interchange-new= line"><blockquote type=3D"cite"><div dir=3D"ltr">That sounds pretty similar t= o the goals of the engine that we're working on -- basically faster iteratio= n times for everyone (this includes engine programmers as well).<div><br></d= iv><div>All of our code (including the compiler) is written in a C#-like lan= guage, so we have perfect reflection from that. Attributes are pretty m= uch required for providing additional information to property editors for th= ings like valid ranges of a field or notifications of a property change (mos= t of which can go away in Final builds).</div> <div><br></div><div style=3D"">Another thing to consider may be allowing for= data derivation. This can save quite a bit of work when the need for c= opies of objects come up that only need one or two properties changed.  = ;If you pick a serialization format for the editable data that knows about t= he data (fieldtype + fieldname + value), it makes it a lot easier to get thi= s working.</div> <div style=3D""><br></div><div style=3D"">In the past, we've used Macros and= they worked fine. They're a bit more annoying to extend with optional= parameters and you have to repeat the field/property name, but that's not t= errible since you can generally get it down to compile-time errors if you ge= t something wrong (other than forgetting to add a new field).</div> <div style=3D""><br></div><div style=3D"">-Doug</div><div style=3D""><br></d= iv></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Fri= , Mar 15, 2013 at 3:45 AM, Tinco Andringa <span dir=3D"ltr"><<a href=3D"m= ailto:[email protected]" target=3D"_blank">[email protected]</a>></span> wrote:<b= r> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">I think the point of the discussion is to not j= ust assume C++ isn't<br> going to do it. As far as I understand the goal is to have the ability<br> to develop tools that aid programmers and artist in iterating rapidly.<br> This means for artists that they can swap artwork in at runtime or<br> quickly boot the game from a serialized state with new artwork, not<br> just on his/her own computer but also over a network to perhaps a<br> console. For programmers this may mean that they can view the game<br> state easily, launch the game in various states easily, swap in/out<br> scripts easily, maybe even change code without fully recompiling or<br> even restarting the game.<br> <br> Am I missing something important?<br> <div class=3D"HOEnZb"><div class=3D"h5"><br> On Fri, Mar 15, 2013 at 8:33 AM, Massimo Del Zotto <<a href=3D"mailto:mas= [email protected]">[email protected]</a>> wrote:<br> > I have limited experience but I have been very impressed by the separat= ion<br> > provided by scripting languages.<br> > Considering the goals:<br> ><br> > My goal is to have fast iteration times, both artists and programmers<b= r> ><br> > It appears to me C++ isn't going to do it. Especially on the artist sid= e of<br> > the thing.<br> ><br> > By the way, I would like to have an insight on how this reflection<br> > information is to be used. If it is used to simulate duck typing I'd us= e a<br> > tool which supports duck typing natively.<br> ><br> > Massimo<br> ><br> </div></div><div class=3D"HOEnZb"><div class=3D"h5">> ___________________= ____________________________<br> > Sweng-Gamedev mailing list<br> > <a href=3D"mailto:[email protected]">Sweng-Gamedev@= lists.midnightryder.com</a><br> > <a href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-mi= dnightryder.com" target=3D"_blank">http://lists.midnightryder.com/listinfo.c= gi/sweng-gamedev-midnightryder.com</a><br> ><br> _______________________________________________<br> Sweng-Gamedev mailing list<br> <a href=3D"mailto:[email protected]">Sweng-Gamedev@lists= .midnightryder.com</a><br> <a href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnigh= tryder.com" target=3D"_blank">http://lists.midnightryder.com/listinfo.cgi/sw= eng-gamedev-midnightryder.com</a><br> </div></div></blockquote></div><br></div> _______________________________________________<br>Sweng-Gamedev mailing lis= t<br><a href=3D"mailto:[email protected]">Sweng-Gamedev@= lists.midnightryder.com</a><br><a href=3D"http://lists.midnightryder.com/lis= tinfo.cgi/sweng-gamedev-midnightryder.com">http://lists.midnightryder.com/li= stinfo.cgi/sweng-gamedev-midnightryder.com</a><br></blockquote></div><br></d= iv></blockquote><blockquote type=3D"cite"><div><span>_______________________= ________________________</span><br><span>Sweng-Gamedev mailing list</span><b= r><span><a href=3D"mailto:[email protected]">Sweng-Gamed= [email protected]</a></span><br><span><a href=3D"http://lists.midni= ghtryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com">http://lists.midn= ightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com</a></span><br></d= iv></blockquote></body></html>= --Apple-Mail-6D92B325-D659-444E-A371-29BF542663DF-- --===============1521491893== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sweng-Gamedev mailing list [email protected] http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com --===============1521491893==--