Re: Reflection in game engine (c++)
Mike Shaver <[email protected]> Mon, 18 Mar 2013 21:07:58 -0700
| Newsgroups | gmane.games.devel.sweng |
|---|---|
| Message-ID | <[email protected]> |
--===============1980037146== Content-Type: multipart/alternative; boundary=Apple-Mail-E5EC6C69-850F-4C2C-A620-88CC5243CE6E Content-Transfer-Encoding: 7bit --Apple-Mail-E5EC6C69-850F-4C2C-A620-88CC5243CE6E Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable I wanted to just have one place that things were declared, and for that to b= e with real code rather in a separate "please don't edit this no matter how m= uch VC++ tells you this is the definition" file. My point is really that it's over-constraining the problem to choose one for= mat that is both fast+compact enough for runtime use, and flexible+straightf= orward enough to bend into all the parts of the tool pipeline. It's the work of 5 lines of Python to go from JSON to XML or CSV or .ini fil= es or whatever. That someone has linked the format parser into a dozen diffe= rent software environments is indeed an advantage that I would take seriousl= y. I generally don't know what my tools are going to be until I've written t= hem, so free option value is pretty nice. I've done the "IDL to headers and packed metadata files" thing, and it was p= retty annoying (and error-prone) to wire it into even a half-dozen straight-= C tools. I may be overreacting to that experience, but mutatis mutandis I ve= ry much prefer one tiny exporter and then letting tools transcode however th= ey want. Mike phone-typed On Mar 18, 2013, at 7:58 PM, Adrian Stephens <[email protected]> wrote:= >=20 > So you're writing macros to turn your C structures into data, building the= m into a library, linking it with your exporter and running the exporter to g= enerate a script-format of the structures. Which, then has to be parsed by y= our tools. I honestly didn't mean that to sound so negative, but wouldn't i= t have been easier the other way round - start with a simple domain-specific= language to define your types and spit out C header files? >=20 > The PHP thing works because you're proposing outputting JSON, which somebo= dy presumably has already linked into it. If that sort of thing is difficul= t you could certainly output JSON at the same time you output C header files= , or I suppose you could probably even use JSON *as* your script format. >=20 >=20 > On Mar 18, 2013, at 5:26 PM, Mike Shaver wrote: >=20 >> When I did this for a non-game app, I defined it with self-aware, strong-= AI macros, and compiled the resulting data into a shared library. The app (g= ame) linked against the library, as did one important tool: the exporter. It= was a wee little program that dumped the compiled metadata to an interchang= e format. Today I would choose JSON or XML. The build process would regenera= te metadata.json whenever metadata.so changed, and extending it to multiple l= ibraries/output files was straightforward. >>=20 >> Runtime reflection formats have many constraints that tool-interchange fo= rmats don't, and tool formats often want things the runtime abhors. Put the l= ine-numbers in the output and remix with git-blame output to tell you who la= st changed an entity? Sure. Web explorer of the resources, without linking t= he parser library into PHP...I'll be over here. >>=20 >> I would similarly not use Collada files at runtime. >>=20 >> phone-typed >>=20 >>=20 >> On Mar 18, 2013, at 4:30 PM, Adrian Stephens <[email protected]> wro= te: >>=20 >>>=20 >>> I'm glad we're still talking about this - I think it's important and I w= anted to add my own observations: >>>=20 >>> The macro/template approach to reflection only applies to one side of th= e data/code interface. If all you need to do is serialize runtime data to b= e read back in by runtime code, then you're ok; but if you want to access th= at metadata from an external tool, you either have to recompile the tool or p= lugin when anything changes, or write something that parses that code. >>>=20 >>> If you're going to have to write a parser anyway, it seems to me you mi= ght as well use it for both code and data views. Which means you can use a m= ore natural syntax for your structure definitions that even non-programmers c= an use. >>>=20 >>> To map this back into your C++ code, you can pretty trivially write out h= eader files containing typedefs that correspond to the metadata, and generat= e whatever you need to allow the runtime to interpret serialized data. This= also address the separation of code and assets: we don't want artists havin= g to sync the code, and this keeps the data definitions with the assets that= they define. >>>=20 >>> This is what I do, anyway. I have a data language, which defines types,= and which can also be used to define data, and all data is stored using the= se definitions. This includes bitmaps, sounds, models, etc. The data langu= age is separated from the data representation so this doesn't imply that bit= maps are stored in a script-like array of colors - but it does mean that too= ls can use the reflection system on more classes of data than just small gam= eplay entities. And when distinctions like that are removed, and all things= are treated the same way, lots of clever things just fall out. >>>=20 >>> Some other random thoughts: >>> As far as I can see, all C++ needs to allow full reflection is some sort= of fieldsof(T) operator that enumerates the fields of a structure. e.g. (t= yped straight into email, so probably bollocks - and not representative of m= y actual system!) >>>=20 >>> struct metadata { >>> enum TYPE {INT, PTR, ARRAY, STRUCT} type; >>> metadata(TYPE &_type) : type(_type) {} >>> }; >>> template<typename A, typename... B> struct md_fields : md<A>, md_fields<= B> {}; >>> template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md()= : metadata(STRUCT) {} }; >>> template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata= (PTR) {} }; >>> template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t;= md() : metadata(ARRAY), n(N) {} }; >>> template<> struct md<int> : metadata { md() : metadata(INT) {} }; >>>=20 >>> template<typename T> metadata *get_md(const T &t) { static md<T> m; retu= rn &m; } >>>=20 >>>=20 >>>=20 >>> Adrian Stephens >>>=20 >>> On Mar 18, 2013, at 10:45 AM, Douglas Cox wrote: >>>=20 >>>> That sounds pretty similar to the goals of the engine that we're workin= g on -- basically faster iteration times for everyone (this includes engine p= rogrammers as well). >>>>=20 >>>> All of our code (including the compiler) is written in a C#-like langua= ge, so we have perfect reflection from that. Attributes are pretty much req= uired for providing additional information to property editors for things li= ke valid ranges of a field or notifications of a property change (most of wh= ich can go away in Final builds). >>>>=20 >>>> Another thing to consider may be allowing for data derivation. This ca= n save quite a bit of work when the need for copies of objects come up that o= nly need one or two properties changed. If you pick a serialization format f= or the editable data that knows about the data (fieldtype + fieldname + valu= e), it makes it a lot easier to get this working. >>>>=20 >>>> In the past, we've used Macros and they worked fine. They're a bit mor= e annoying to extend with optional parameters and you have to repeat the fie= ld/property name, but that's not terrible since you can generally get it dow= n to compile-time errors if you get something wrong (other than forgetting t= o add a new field). >>>>=20 >>>> -Doug >>>>=20 >>>>=20 >>>>=20 >>>> On Fri, Mar 15, 2013 at 3:45 AM, Tinco Andringa <[email protected]> wrote: >>>>> I think the point of the discussion is to not just assume C++ isn't >>>>> going to do it. As far as I understand the goal is to have the ability= >>>>> to develop tools that aid programmers and artist in iterating rapidly.= >>>>> This means for artists that they can swap artwork in at runtime or >>>>> quickly boot the game from a serialized state with new artwork, not >>>>> just on his/her own computer but also over a network to perhaps a >>>>> console. For programmers this may mean that they can view the game >>>>> state easily, launch the game in various states easily, swap in/out >>>>> scripts easily, maybe even change code without fully recompiling or >>>>> even restarting the game. >>>>>=20 >>>>> Am I missing something important? >>>>>=20 >>>>> On Fri, Mar 15, 2013 at 8:33 AM, Massimo Del Zotto <[email protected]= om> wrote: >>>>> > I have limited experience but I have been very impressed by the sepa= ration >>>>> > provided by scripting languages. >>>>> > Considering the goals: >>>>> > >>>>> > My goal is to have fast iteration times, both artists and programmer= s >>>>> > >>>>> > It appears to me C++ isn't going to do it. Especially on the artist s= ide of >>>>> > the thing. >>>>> > >>>>> > By the way, I would like to have an insight on how this reflection >>>>> > information is to be used. If it is used to simulate duck typing I'd= use a >>>>> > tool which supports duck typing natively. >>>>> > >>>>> > Massimo >>>>> > >>>>> > _______________________________________________ >>>>> > Sweng-Gamedev mailing list >>>>> > [email protected] >>>>> > http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightry= der.com >>>>> > >>>>> _______________________________________________ >>>>> Sweng-Gamedev mailing list >>>>> [email protected] >>>>> http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryde= r.com >>>>=20 >>>> _______________________________________________ >>>> Sweng-Gamedev mailing list >>>> [email protected] >>>> http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder= .com >>>=20 >>> _______________________________________________ >>> Sweng-Gamedev mailing list >>> [email protected] >>> http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.= com >> _______________________________________________ >> Sweng-Gamedev mailing list >> [email protected] >> http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.c= om >=20 > _______________________________________________ > Sweng-Gamedev mailing list > [email protected] > http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.co= m --Apple-Mail-E5EC6C69-850F-4C2C-A620-88CC5243CE6E Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D= utf-8"></head><body dir=3D"auto"><div>I wanted to just have one place that t= hings were declared, and for that to be with real code rather in a separate "= please don't edit this no matter how much VC++ tells you this is the definit= ion" file.</div><div><br></div><div>My point is really that it's over-constr= aining the problem to choose one format that is both fast+compact enough for= runtime use, and flexible+straightforward enough to bend into all the parts= of the tool pipeline.</div><div><br></div><div>It's the work of 5 lines of P= ython to go from JSON to XML or CSV or .ini files or whatever. That someone h= as linked the format parser into a dozen different software environments is i= ndeed an advantage that I would take seriously. I generally don't know what m= y tools are going to be until I've written them, so free option value is pre= tty nice.</div><div><br></div><div>I've done the "IDL to headers and packed m= etadata files" thing, and it was pretty annoying (and error-prone) to wire i= t into even a half-dozen straight-C tools. I may be overreacting to that exp= erience, but mutatis mutandis I very much prefer one tiny exporter and then l= etting tools transcode however they want.</div><div><br></div><div>Mike<br><= br>phone-typed<div><br></div></div><div>On Mar 18, 2013, at 7:58 PM, Adrian S= tephens <<a href=3D"mailto:[email protected]">[email protected]= </a>> wrote:<br><br></div><blockquote type=3D"cite"><div><div><br></div><= div>So you're writing macros to turn your C structures into data, building t= hem into a library, linking it with your exporter and running the exporter t= o generate a script-format of the structures. Which, then has to be pa= rsed by your tools. I honestly didn't mean that to sound so negative, b= ut wouldn't it have been easier the other way round - start with a simple do= main-specific language to define your types and spit out C header files?</di= v><div><br></div><div>The PHP thing works because you're proposing outputtin= g JSON, which somebody presumably has already linked into it. If that s= ort of thing is difficult you could certainly output JSON at the same time y= ou output C header files, or I suppose you could probably even use JSON *as*= your script format.</div><div><br></div><br><div><div>On Mar 18, 2013, at 5= :26 PM, Mike Shaver wrote:</div><br class=3D"Apple-interchange-newline"><blo= ckquote type=3D"cite"><meta http-equiv=3D"content-type" content=3D"text/html= ; charset=3Dutf-8"><div dir=3D"auto"><div>When I did this for a non-game app= , I defined it with self-aware, strong-AI macros, and compiled the resulting= data into a shared library. The app (game) linked against the library, as d= id one important tool: the exporter. It was a wee little program that dumped= the compiled metadata to an interchange format. Today I would choose JSON o= r XML. The build process would regenerate metadata.json whenever metadata.so= changed, and extending it to multiple libraries/output files was straightfo= rward.</div><div><br></div><div>Runtime reflection formats have many constra= ints that tool-interchange formats don't, and tool formats often want things= the runtime abhors. Put the line-numbers in the output and remix with git-b= lame output to tell you who last changed an entity? Sure. Web explorer of th= e resources, without linking the parser library into PHP...I'll be over here= .</div><div><br></div><div>I would similarly not use Collada files at runtim= e.</div><div><br>phone-typed<div><br></div></div><div><br>On Mar 18, 2013, a= t 4:30 PM, Adrian Stephens <<a href=3D"mailto:[email protected]">adr= [email protected]</a>> wrote:<br><br></div><blockquote type=3D"cite"><d= iv><div><br></div><div>I'm glad we're still talking about this - I think it'= s important and I wanted to add my own observations:</div><div><br></div><di= v>The macro/template approach to reflection only applies to one side of the d= ata/code interface. If all you need to do is serialize runtime data to= be read back in by runtime code, then you're ok; but if you want to access t= hat metadata from an external tool, you either have to recompile the tool or= plugin when anything changes, or write something that parses that code.</di= v><div><br></div><div>If you're going to have to write a parser anyway, &nbs= p;it seems to me you might as well use it for both code and data views. &nbs= p;Which means you can use a more natural syntax for your structure definitio= ns that even non-programmers can use.</div><div><br></div><div>To map this b= ack into your C++ code, you can pretty trivially write out header files cont= aining typedefs that correspond to the metadata, and generate whatever you n= eed to allow the runtime to interpret serialized data. This also addre= ss the separation of code and assets: we don't want artists having to s= ync the code, and this keeps the data definitions with the assets that they d= efine.</div><div><br></div><div>This is what I do, anyway. I have a da= ta language, which defines types, and which can also be used to define data,= and all data is stored using these definitions. This includes bitmaps= , sounds, models, etc. The data language is separated from the data re= presentation so this doesn't imply that bitmaps are stored in a script-like a= rray of colors - but it does mean that tools can use the reflection system o= n more classes of data than just small gameplay entities. And when dis= tinctions like that are removed, and all things are treated the same way, lo= ts of clever things just fall out.</div><div><br></div><div>Some other rando= m thoughts:</div><div>As far as I can see, all C++ needs to allow full refle= ction is some sort of fieldsof(T) operator that enumerates the fields of a s= tructure. e.g. (typed straight into email, so probably bollocks - and n= ot representative of my actual system!)</div><div><br></div><div>struct meta= data {</div><div><span class=3D"Apple-tab-span" style=3D"white-space:pre">= </span>enum TYPE {INT, PTR, ARRAY, STRUCT} type;</div><div><span class=3D"A= pple-tab-span" style=3D"white-space:pre"> </span>metadata(TYPE &_= type) : type(_type) {}</div><div>};</div><div>template<typename A, typena= me... B> struct md_fields : md<A>, md_fields<B> {};</div><div= >template<typename T> struct md : metadata, md_fields<fie= ldsof(T)> { md() : metadata(STRUCT) {} };</div><div>template<typename T= > struct md<T*> : metadata { md<T> t; md() : metada= ta(PTR) {} };</div><div>template<typename T, int N> struct md<T[N]&= gt; : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} };</d= iv><div>template<> struct md<int> : metadata { md() : metadata(I= NT) {} };</div><div><br></div><div>template<typename T> metadata *get_= md(const T &t) { static md<T> m; return &m; }</div><div><br></= div><div><br></div><div><br></div><div>Adrian Stephens</div><br><div><div>On= Mar 18, 2013, at 10:45 AM, Douglas Cox wrote:</div><br class=3D"Apple-inter= change-newline"><blockquote type=3D"cite"><div dir=3D"ltr">That sounds prett= y similar to the goals of the engine that we're working on -- basically fast= er iteration times for everyone (this includes engine programmers as well).<= div><br></div><div>All of our code (including the compiler) is written in a C= #-like language, so we have perfect reflection from that. Attributes a= re pretty much required for providing additional information to property edi= tors for things like valid ranges of a field or notifications of a property c= hange (most of which can go away in Final builds).</div> <div><br></div><div style=3D"">Another thing to consider may be allowing for= data derivation. This can save quite a bit of work when the need for c= opies of objects come up that only need one or two properties changed.  = ;If you pick a serialization format for the editable data that knows about t= he data (fieldtype + fieldname + value), it makes it a lot easier to get thi= s working.</div> <div style=3D""><br></div><div style=3D"">In the past, we've used Macros and= they worked fine. They're a bit more annoying to extend with optional= parameters and you have to repeat the field/property name, but that's not t= errible since you can generally get it down to compile-time errors if you ge= t something wrong (other than forgetting to add a new field).</div> <div style=3D""><br></div><div style=3D"">-Doug</div><div style=3D""><br></d= iv></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Fri= , Mar 15, 2013 at 3:45 AM, Tinco Andringa <span dir=3D"ltr"><<a href=3D"m= ailto:[email protected]" target=3D"_blank">[email protected]</a>></span> wrote:<b= r> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">I think the point of the discussion is to not j= ust assume C++ isn't<br> going to do it. As far as I understand the goal is to have the ability<br> to develop tools that aid programmers and artist in iterating rapidly.<br> This means for artists that they can swap artwork in at runtime or<br> quickly boot the game from a serialized state with new artwork, not<br> just on his/her own computer but also over a network to perhaps a<br> console. For programmers this may mean that they can view the game<br> state easily, launch the game in various states easily, swap in/out<br> scripts easily, maybe even change code without fully recompiling or<br> even restarting the game.<br> <br> Am I missing something important?<br> <div class=3D"HOEnZb"><div class=3D"h5"><br> On Fri, Mar 15, 2013 at 8:33 AM, Massimo Del Zotto <<a href=3D"mailto:mas= [email protected]">[email protected]</a>> wrote:<br> > I have limited experience but I have been very impressed by the separat= ion<br> > provided by scripting languages.<br> > Considering the goals:<br> ><br> > My goal is to have fast iteration times, both artists and programmers<b= r> ><br> > It appears to me C++ isn't going to do it. Especially on the artist sid= e of<br> > the thing.<br> ><br> > By the way, I would like to have an insight on how this reflection<br> > information is to be used. If it is used to simulate duck typing I'd us= e a<br> > tool which supports duck typing natively.<br> ><br> > Massimo<br> ><br> </div></div><div class=3D"HOEnZb"><div class=3D"h5">> ___________________= ____________________________<br> > Sweng-Gamedev mailing list<br> > <a href=3D"mailto:[email protected]">Sweng-Gamedev@= lists.midnightryder.com</a><br> > <a href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-mi= dnightryder.com" target=3D"_blank">http://lists.midnightryder.com/listinfo.c= gi/sweng-gamedev-midnightryder.com</a><br> ><br> _______________________________________________<br> Sweng-Gamedev mailing list<br> <a href=3D"mailto:[email protected]">Sweng-Gamedev@lists= .midnightryder.com</a><br> <a href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnigh= tryder.com" target=3D"_blank">http://lists.midnightryder.com/listinfo.cgi/sw= eng-gamedev-midnightryder.com</a><br> </div></div></blockquote></div><br></div> _______________________________________________<br>Sweng-Gamedev mailing lis= t<br><a href=3D"mailto:[email protected]">Sweng-Gamedev@= lists.midnightryder.com</a><br><a href=3D"http://lists.midnightryder.com/lis= tinfo.cgi/sweng-gamedev-midnightryder.com">http://lists.midnightryder.com/li= stinfo.cgi/sweng-gamedev-midnightryder.com</a><br></blockquote></div><br></d= iv></blockquote><blockquote type=3D"cite"><div><span>_______________________= ________________________</span><br><span>Sweng-Gamedev mailing list</span><b= r><span><a href=3D"mailto:[email protected]">Sweng-Gamed= [email protected]</a></span><br><span><a href=3D"http://lists.midni= ghtryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com">http://lists.midn= ightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com</a></span><br></d= iv></blockquote></div>_______________________________________________<br>Swe= ng-Gamedev mailing list<br><a href=3D"mailto:[email protected]= er.com">[email protected]</a><br><a href=3D"http://lists= .midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com">http://list= s.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com</a><br></bl= ockquote></div><br></div></blockquote><blockquote type=3D"cite"><div><span>_= ______________________________________________</span><br><span>Sweng-Gamedev= mailing list</span><br><span><a href=3D"mailto:[email protected]= ryder.com">[email protected]</a></span><br><span><a href= =3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.= com">http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder= .com</a></span><br></div></blockquote></body></html>= --Apple-Mail-E5EC6C69-850F-4C2C-A620-88CC5243CE6E-- --===============1980037146== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sweng-Gamedev mailing list [email protected] http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com --===============1980037146==--