Re: Reflection in game engine (c++)
Adrian Stephens <[email protected]> Mon, 18 Mar 2013 19:58:48 -0700
| Newsgroups | gmane.games.devel.sweng |
|---|---|
| Message-ID | <[email protected]> |
--===============2098321457== Content-Type: multipart/alternative; boundary="Apple-Mail=_8C3E522B-60C2-4FE2-88FB-DF6D4B410B8C" --Apple-Mail=_8C3E522B-60C2-4FE2-88FB-DF6D4B410B8C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii So you're writing macros to turn your C structures into data, building = them into a library, linking it with your exporter and running the = exporter to generate a script-format of the structures. Which, then has = to be parsed by your tools. I honestly didn't mean that to sound so = negative, but wouldn't it have been easier the other way round - start = with a simple domain-specific language to define your types and spit out = C header files? The PHP thing works because you're proposing outputting JSON, which = somebody presumably has already linked into it. If that sort of thing = is difficult you could certainly output JSON at the same time you output = C header files, or I suppose you could probably even use JSON *as* your = script format. On Mar 18, 2013, at 5:26 PM, Mike Shaver wrote: > When I did this for a non-game app, I defined it with self-aware, = strong-AI macros, and compiled the resulting data into a shared library. = The app (game) linked against the library, as did one important tool: = the exporter. It was a wee little program that dumped the compiled = metadata to an interchange format. Today I would choose JSON or XML. The = build process would regenerate metadata.json whenever metadata.so = changed, and extending it to multiple libraries/output files was = straightforward. >=20 > Runtime reflection formats have many constraints that tool-interchange = formats don't, and tool formats often want things the runtime abhors. = Put the line-numbers in the output and remix with git-blame output to = tell you who last changed an entity? Sure. Web explorer of the = resources, without linking the parser library into PHP...I'll be over = here. >=20 > I would similarly not use Collada files at runtime. >=20 > phone-typed >=20 >=20 > On Mar 18, 2013, at 4:30 PM, Adrian Stephens <[email protected]> = wrote: >=20 >>=20 >> I'm glad we're still talking about this - I think it's important and = I wanted to add my own observations: >>=20 >> The macro/template approach to reflection only applies to one side of = the data/code interface. If all you need to do is serialize runtime = data to be read back in by runtime code, then you're ok; but if you want = to access that metadata from an external tool, you either have to = recompile the tool or plugin when anything changes, or write something = that parses that code. >>=20 >> If you're going to have to write a parser anyway, it seems to me you = might as well use it for both code and data views. Which means you can = use a more natural syntax for your structure definitions that even = non-programmers can use. >>=20 >> To map this back into your C++ code, you can pretty trivially write = out header files containing typedefs that correspond to the metadata, = and generate whatever you need to allow the runtime to interpret = serialized data. This also address the separation of code and assets: = we don't want artists having to sync the code, and this keeps the data = definitions with the assets that they define. >>=20 >> This is what I do, anyway. I have a data language, which defines = types, and which can also be used to define data, and all data is stored = using these definitions. This includes bitmaps, sounds, models, etc. = The data language is separated from the data representation so this = doesn't imply that bitmaps are stored in a script-like array of colors - = but it does mean that tools can use the reflection system on more = classes of data than just small gameplay entities. And when = distinctions like that are removed, and all things are treated the same = way, lots of clever things just fall out. >>=20 >> Some other random thoughts: >> As far as I can see, all C++ needs to allow full reflection is some = sort of fieldsof(T) operator that enumerates the fields of a structure. = e.g. (typed straight into email, so probably bollocks - and not = representative of my actual system!) >>=20 >> struct metadata { >> enum TYPE {INT, PTR, ARRAY, STRUCT} type; >> metadata(TYPE &_type) : type(_type) {} >> }; >> template<typename A, typename... B> struct md_fields : md<A>, = md_fields<B> {}; >> template<typename T> struct md : metadata, md_fields<fieldsof(T)> { = md() : metadata(STRUCT) {} }; >> template<typename T> struct md<T*> : metadata { md<T> t; md() : = metadata(PTR) {} }; >> template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> = t; md() : metadata(ARRAY), n(N) {} }; >> template<> struct md<int> : metadata { md() : metadata(INT) {} }; >>=20 >> template<typename T> metadata *get_md(const T &t) { static md<T> m; = return &m; } >>=20 >>=20 >>=20 >> Adrian Stephens >>=20 >> On Mar 18, 2013, at 10:45 AM, Douglas Cox wrote: >>=20 >>> That sounds pretty similar to the goals of the engine that we're = working on -- basically faster iteration times for everyone (this = includes engine programmers as well). >>>=20 >>> All of our code (including the compiler) is written in a C#-like = language, so we have perfect reflection from that. Attributes are = pretty much required for providing additional information to property = editors for things like valid ranges of a field or notifications of a = property change (most of which can go away in Final builds). >>>=20 >>> Another thing to consider may be allowing for data derivation. This = can save quite a bit of work when the need for copies of objects come up = that only need one or two properties changed. If you pick a = serialization format for the editable data that knows about the data = (fieldtype + fieldname + value), it makes it a lot easier to get this = working. >>>=20 >>> In the past, we've used Macros and they worked fine. They're a bit = more annoying to extend with optional parameters and you have to repeat = the field/property name, but that's not terrible since you can generally = get it down to compile-time errors if you get something wrong (other = than forgetting to add a new field). >>>=20 >>> -Doug >>>=20 >>>=20 >>>=20 >>> On Fri, Mar 15, 2013 at 3:45 AM, Tinco Andringa <[email protected]> = wrote: >>> I think the point of the discussion is to not just assume C++ isn't >>> going to do it. As far as I understand the goal is to have the = ability >>> to develop tools that aid programmers and artist in iterating = rapidly. >>> This means for artists that they can swap artwork in at runtime or >>> quickly boot the game from a serialized state with new artwork, not >>> just on his/her own computer but also over a network to perhaps a >>> console. For programmers this may mean that they can view the game >>> state easily, launch the game in various states easily, swap in/out >>> scripts easily, maybe even change code without fully recompiling or >>> even restarting the game. >>>=20 >>> Am I missing something important? >>>=20 >>> On Fri, Mar 15, 2013 at 8:33 AM, Massimo Del Zotto = <[email protected]> wrote: >>> > I have limited experience but I have been very impressed by the = separation >>> > provided by scripting languages. >>> > Considering the goals: >>> > >>> > My goal is to have fast iteration times, both artists and = programmers >>> > >>> > It appears to me C++ isn't going to do it. Especially on the = artist side of >>> > the thing. >>> > >>> > By the way, I would like to have an insight on how this reflection >>> > information is to be used. If it is used to simulate duck typing = I'd use a >>> > tool which supports duck typing natively. >>> > >>> > Massimo >>> > >>> > _______________________________________________ >>> > Sweng-Gamedev mailing list >>> > [email protected] >>> > = http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.co= m >>> > >>> _______________________________________________ >>> Sweng-Gamedev mailing list >>> [email protected] >>> = http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.co= m >>>=20 >>> _______________________________________________ >>> Sweng-Gamedev mailing list >>> [email protected] >>> = http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.co= m >>=20 >> _______________________________________________ >> Sweng-Gamedev mailing list >> [email protected] >> = http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.co= m > _______________________________________________ > Sweng-Gamedev mailing list > [email protected] > = http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.co= m --Apple-Mail=_8C3E522B-60C2-4FE2-88FB-DF6D4B410B8C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; = "><div><br></div><div>So you're writing macros to turn your C structures = into data, building them into a library, linking it with your exporter = and running the exporter to generate a script-format of the structures. = Which, then has to be parsed by your tools. I honestly = didn't mean that to sound so negative, but wouldn't it have been easier = the other way round - start with a simple domain-specific language to = define your types and spit out C header = files?</div><div><br></div><div>The PHP thing works because you're = proposing outputting JSON, which somebody presumably has already linked = into it. If that sort of thing is difficult you could certainly = output JSON at the same time you output C header files, or I suppose you = could probably even use JSON *as* your script = format.</div><div><br></div><br><div><div>On Mar 18, 2013, at 5:26 PM, = Mike Shaver wrote:</div><br = class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><meta = http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8"><div = dir=3D"auto"><div>When I did this for a non-game app, I defined it with = self-aware, strong-AI macros, and compiled the resulting data into a = shared library. The app (game) linked against the library, as did one = important tool: the exporter. It was a wee little program that dumped = the compiled metadata to an interchange format. Today I would choose = JSON or XML. The build process would regenerate metadata.json whenever = metadata.so changed, and extending it to multiple libraries/output files = was straightforward.</div><div><br></div><div>Runtime reflection formats = have many constraints that tool-interchange formats don't, and tool = formats often want things the runtime abhors. Put the line-numbers in = the output and remix with git-blame output to tell you who last changed = an entity? Sure. Web explorer of the resources, without linking the = parser library into PHP...I'll be over here.</div><div><br></div><div>I = would similarly not use Collada files at = runtime.</div><div><br>phone-typed<div><br></div></div><div><br>On Mar = 18, 2013, at 4:30 PM, Adrian Stephens <<a = href=3D"mailto:[email protected]">[email protected]</a>> = wrote:<br><br></div><blockquote type=3D"cite"><div><div><br></div><div>I'm= glad we're still talking about this - I think it's important and I = wanted to add my own observations:</div><div><br></div><div>The = macro/template approach to reflection only applies to one side of the = data/code interface. If all you need to do is serialize runtime = data to be read back in by runtime code, then you're ok; but if you want = to access that metadata from an external tool, you either have to = recompile the tool or plugin when anything changes, or write something = that parses that code.</div><div><br></div><div>If you're going to have = to write a parser anyway, it seems to me you might as well use it = for both code and data views. Which means you can use a more = natural syntax for your structure definitions that even non-programmers = can use.</div><div><br></div><div>To map this back into your C++ code, = you can pretty trivially write out header files containing typedefs that = correspond to the metadata, and generate whatever you need to allow the = runtime to interpret serialized data. This also address = the separation of code and assets: we don't want artists having to = sync the code, and this keeps the data definitions with the assets that = they define.</div><div><br></div><div>This is what I do, anyway. I = have a data language, which defines types, and which can also be used to = define data, and all data is stored using these definitions. This = includes bitmaps, sounds, models, etc. The data language is = separated from the data representation so this doesn't imply that = bitmaps are stored in a script-like array of colors - but it does mean = that tools can use the reflection system on more classes of data than = just small gameplay entities. And when distinctions like that are = removed, and all things are treated the same way, lots of clever things = just fall out.</div><div><br></div><div>Some other random = thoughts:</div><div>As far as I can see, all C++ needs to allow full = reflection is some sort of fieldsof(T) operator that enumerates the = fields of a structure. e.g. (typed straight into email, so = probably bollocks - and not representative of my actual = system!)</div><div><br></div><div>struct metadata {</div><div><span = class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>enum TYPE = {INT, PTR, ARRAY, STRUCT} type;</div><div><span class=3D"Apple-tab-span" = style=3D"white-space:pre"> </span>metadata(TYPE &_type) : = type(_type) {}</div><div>};</div><div>template<typename A, = typename... B> struct md_fields : md<A>, md_fields<B> = {};</div><div>template<typename T> struct md = : metadata, md_fields<fieldsof(T)> { md() : = metadata(STRUCT) {} };</div><div>template<typename T> struct = md<T*> : metadata { md<T> t; md() : metadata(PTR) = {} };</div><div>template<typename T, int N> struct md<T[N]> = : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} = };</div><div>template<> struct md<int> : metadata { md() : = metadata(INT) {} };</div><div><br></div><div>template<typename T> = metadata *get_md(const T &t) { static md<T> m; return &m; = }</div><div><br></div><div><br></div><div><br></div><div>Adrian = Stephens</div><br><div><div>On Mar 18, 2013, at 10:45 AM, Douglas Cox = wrote:</div><br class=3D"Apple-interchange-newline"><blockquote = type=3D"cite"><div dir=3D"ltr">That sounds pretty similar to the goals = of the engine that we're working on -- basically faster iteration times = for everyone (this includes engine programmers as = well).<div><br></div><div>All of our code (including the compiler) is = written in a C#-like language, so we have perfect reflection from that. = Attributes are pretty much required for providing additional = information to property editors for things like valid ranges of a field = or notifications of a property change (most of which can go away in = Final builds).</div> <div><br></div><div style=3D"">Another thing to consider may be allowing = for data derivation. This can save quite a bit of work when the = need for copies of objects come up that only need one or two properties = changed. If you pick a serialization format for the editable data = that knows about the data (fieldtype + fieldname + value), it makes it a = lot easier to get this working.</div> <div style=3D""><br></div><div style=3D"">In the past, we've used Macros = and they worked fine. They're a bit more annoying to extend with = optional parameters and you have to repeat the field/property name, but = that's not terrible since you can generally get it down to compile-time = errors if you get something wrong (other than forgetting to add a new = field).</div> <div style=3D""><br></div><div style=3D"">-Doug</div><div = style=3D""><br></div></div><div class=3D"gmail_extra"><br><br><div = class=3D"gmail_quote">On Fri, Mar 15, 2013 at 3:45 AM, Tinco Andringa = <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" = target=3D"_blank">[email protected]</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex">I think the point of = the discussion is to not just assume C++ isn't<br> going to do it. As far as I understand the goal is to have the = ability<br> to develop tools that aid programmers and artist in iterating = rapidly.<br> This means for artists that they can swap artwork in at runtime or<br> quickly boot the game from a serialized state with new artwork, not<br> just on his/her own computer but also over a network to perhaps a<br> console. For programmers this may mean that they can view the game<br> state easily, launch the game in various states easily, swap in/out<br> scripts easily, maybe even change code without fully recompiling or<br> even restarting the game.<br> <br> Am I missing something important?<br> <div class=3D"HOEnZb"><div class=3D"h5"><br> On Fri, Mar 15, 2013 at 8:33 AM, Massimo Del Zotto <<a = href=3D"mailto:[email protected]">[email protected]</a>> = wrote:<br> > I have limited experience but I have been very impressed by the = separation<br> > provided by scripting languages.<br> > Considering the goals:<br> ><br> > My goal is to have fast iteration times, both artists and = programmers<br> ><br> > It appears to me C++ isn't going to do it. Especially on the artist = side of<br> > the thing.<br> ><br> > By the way, I would like to have an insight on how this = reflection<br> > information is to be used. If it is used to simulate duck typing = I'd use a<br> > tool which supports duck typing natively.<br> ><br> > Massimo<br> ><br> </div></div><div class=3D"HOEnZb"><div class=3D"h5">> = _______________________________________________<br> > Sweng-Gamedev mailing list<br> > <a = href=3D"mailto:[email protected]">Sweng-Gamedev@lists.= midnightryder.com</a><br> > <a = href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnight= ryder.com" = target=3D"_blank">http://lists.midnightryder.com/listinfo.cgi/sweng-gamede= v-midnightryder.com</a><br> ><br> _______________________________________________<br> Sweng-Gamedev mailing list<br> <a = href=3D"mailto:[email protected]">Sweng-Gamedev@lists.= midnightryder.com</a><br> <a = href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnight= ryder.com" = target=3D"_blank">http://lists.midnightryder.com/listinfo.cgi/sweng-gamede= v-midnightryder.com</a><br> </div></div></blockquote></div><br></div> _______________________________________________<br>Sweng-Gamedev mailing = list<br><a = href=3D"mailto:[email protected]">Sweng-Gamedev@lists.= midnightryder.com</a><br><a = href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnight= ryder.com">http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midni= ghtryder.com</a><br></blockquote></div><br></div></blockquote><blockquote = type=3D"cite"><div><span>_______________________________________________</= span><br><span>Sweng-Gamedev mailing list</span><br><span><a = href=3D"mailto:[email protected]">Sweng-Gamedev@lists.= midnightryder.com</a></span><br><span><a = href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnight= ryder.com">http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midni= ghtryder.com</a></span><br></div></blockquote></div>______________________= _________________________<br>Sweng-Gamedev mailing list<br><a = href=3D"mailto:[email protected]">Sweng-Gamedev@lists.= midnightryder.com</a><br>http://lists.midnightryder.com/listinfo.cgi/sweng= -gamedev-midnightryder.com<br></blockquote></div><br></body></html>= --Apple-Mail=_8C3E522B-60C2-4FE2-88FB-DF6D4B410B8C-- --===============2098321457== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sweng-Gamedev mailing list [email protected] http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com --===============2098321457==--