Re: Reflection in game engine (c++)
"Jarkko Lempiainen" <[email protected]> Mon, 18 Mar 2013 20:33:41 -0400
| Newsgroups | gmane.games.devel.sweng |
|---|---|
| Message-ID | <[email protected]> |
This is a multipart message in MIME format. --===============1117138264== Content-Type: multipart/alternative; boundary="----=_NextPart_000_001F_01CE2417.E18683E0" Content-Language: en-us This is a multipart message in MIME format. ------=_NextPart_000_001F_01CE2417.E18683E0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit If you define your reflection in C++, you can still have external tools for asset processing, because class definitions must be stored along with assets. This is something you have to do regardless how you implement your reflection (assuming you don't have cumbersome "convert all asset to new version whenever the class definition changes"), because different assets may use different versions of class definition and you must be able to read & convert the old data to new version when reading it. Beyond basic class reflection, the reflection system must also be able to handle special cases. For example when dealing with textures, you reflection system must support "custom reflection" definitions where textures are read straight to the video memory without intermediate data store. Or some classes may expose different set of members depending on the instance of the class (think of exposing variants for example). I'm not sure how you would implement these with reflection defined in script without special hacks in the reflection system. Cheers, Jarkko From: [email protected] [mailto:[email protected]] On Behalf Of Adrian Stephens Sent: Monday, March 18, 2013 7:31 PM To: [email protected] Subject: Re: [Sweng-Gamedev] Reflection in game engine (c++) I'm glad we're still talking about this - I think it's important and I wanted to add my own observations: The macro/template approach to reflection only applies to one side of the data/code interface. If all you need to do is serialize runtime data to be read back in by runtime code, then you're ok; but if you want to access that metadata from an external tool, you either have to recompile the tool or plugin when anything changes, or write something that parses that code. If you're going to have to write a parser anyway, it seems to me you might as well use it for both code and data views. Which means you can use a more natural syntax for your structure definitions that even non-programmers can use. To map this back into your C++ code, you can pretty trivially write out header files containing typedefs that correspond to the metadata, and generate whatever you need to allow the runtime to interpret serialized data. This also address the separation of code and assets: we don't want artists having to sync the code, and this keeps the data definitions with the assets that they define. This is what I do, anyway. I have a data language, which defines types, and which can also be used to define data, and all data is stored using these definitions. This includes bitmaps, sounds, models, etc. The data language is separated from the data representation so this doesn't imply that bitmaps are stored in a script-like array of colors - but it does mean that tools can use the reflection system on more classes of data than just small gameplay entities. And when distinctions like that are removed, and all things are treated the same way, lots of clever things just fall out. Some other random thoughts: As far as I can see, all C++ needs to allow full reflection is some sort of fieldsof(T) operator that enumerates the fields of a structure. e.g. (typed straight into email, so probably bollocks - and not representative of my actual system!) struct metadata { enum TYPE {INT, PTR, ARRAY, STRUCT} type; metadata(TYPE &_type) : type(_type) {} }; template<typename A, typename... B> struct md_fields : md<A>, md_fields<B> {}; template<typename T> struct md : metadata, md_fields<fieldsof(T)> { md() : metadata(STRUCT) {} }; template<typename T> struct md<T*> : metadata { md<T> t; md() : metadata(PTR) {} }; template<typename T, int N> struct md<T[N]> : metadata { int n; md<T> t; md() : metadata(ARRAY), n(N) {} }; template<> struct md<int> : metadata { md() : metadata(INT) {} }; template<typename T> metadata *get_md(const T &t) { static md<T> m; return &m; } Adrian Stephens On Mar 18, 2013, at 10:45 AM, Douglas Cox wrote: That sounds pretty similar to the goals of the engine that we're working on -- basically faster iteration times for everyone (this includes engine programmers as well). All of our code (including the compiler) is written in a C#-like language, so we have perfect reflection from that. Attributes are pretty much required for providing additional information to property editors for things like valid ranges of a field or notifications of a property change (most of which can go away in Final builds). Another thing to consider may be allowing for data derivation. This can save quite a bit of work when the need for copies of objects come up that only need one or two properties changed. If you pick a serialization format for the editable data that knows about the data (fieldtype + fieldname + value), it makes it a lot easier to get this working. In the past, we've used Macros and they worked fine. They're a bit more annoying to extend with optional parameters and you have to repeat the field/property name, but that's not terrible since you can generally get it down to compile-time errors if you get something wrong (other than forgetting to add a new field). -Doug On Fri, Mar 15, 2013 at 3:45 AM, Tinco Andringa <[email protected]> wrote: I think the point of the discussion is to not just assume C++ isn't going to do it. As far as I understand the goal is to have the ability to develop tools that aid programmers and artist in iterating rapidly. This means for artists that they can swap artwork in at runtime or quickly boot the game from a serialized state with new artwork, not just on his/her own computer but also over a network to perhaps a console. For programmers this may mean that they can view the game state easily, launch the game in various states easily, swap in/out scripts easily, maybe even change code without fully recompiling or even restarting the game. Am I missing something important? On Fri, Mar 15, 2013 at 8:33 AM, Massimo Del Zotto <[email protected]> wrote: > I have limited experience but I have been very impressed by the separation > provided by scripting languages. > Considering the goals: > > My goal is to have fast iteration times, both artists and programmers > > It appears to me C++ isn't going to do it. Especially on the artist side of > the thing. > > By the way, I would like to have an insight on how this reflection > information is to be used. If it is used to simulate duck typing I'd use a > tool which supports duck typing natively. > > Massimo > > _______________________________________________ > Sweng-Gamedev mailing list > [email protected] > http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com > _______________________________________________ Sweng-Gamedev mailing list [email protected] http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com _______________________________________________ Sweng-Gamedev mailing list [email protected] http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com ------=_NextPart_000_001F_01CE2417.E18683E0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><META = HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 14 = (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman","serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.apple-tab-span {mso-style-name:apple-tab-span;} span.EmailStyle18 {mso-style-type:personal-reply; font-family:"Calibri","sans-serif"; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue = vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'>If you define your reflection in C++, you can still have external = tools for asset processing, because class definitions must be stored = along with assets. This is something you have to do regardless how you = implement your reflection (assuming you don’t have cumbersome = “convert all asset to new version whenever the class definition = changes”), because different assets may use different versions of = class definition and you must be able to read & convert the old data = to new version when reading it.<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'>Beyond basic class reflection, the reflection system must also be = able to handle special cases. For example when dealing with textures, = you reflection system must support “custom reflection” = definitions where textures are read straight to the video memory without = intermediate data store. Or some classes may expose different set of = members depending on the instance of the class (think of exposing = variants for example). I’m not sure how you would implement these = with reflection defined in script without special hacks in the = reflection system.<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'>Cheers, Jarkko<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497= D'><o:p> </o:p></span></p><div><div = style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in = 0in 0in'><p class=3DMsoNormal><b><span = style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span>= </b><span style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'> = [email protected] = [mailto:[email protected]] <b>On Behalf Of = </b>Adrian Stephens<br><b>Sent:</b> Monday, March 18, 2013 7:31 = PM<br><b>To:</b> [email protected]<br><b>Subject:</b> Re: = [Sweng-Gamedev] Reflection in game engine = (c++)<o:p></o:p></span></p></div></div><p = class=3DMsoNormal><o:p> </o:p></p><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>I'm glad we're still talking about this - I think it's = important and I wanted to add my own = observations:<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>The macro/template approach to reflection only applies = to one side of the data/code interface. If all you need to do is = serialize runtime data to be read back in by runtime code, then you're = ok; but if you want to access that metadata from an external tool, you = either have to recompile the tool or plugin when anything changes, or = write something that parses that code.<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>If you're going to have to write a parser anyway, = it seems to me you might as well use it for both code and data = views. Which means you can use a more natural syntax for your = structure definitions that even non-programmers can = use.<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>To map this back into your C++ code, you can pretty = trivially write out header files containing typedefs that correspond to = the metadata, and generate whatever you need to allow the runtime to = interpret serialized data. This also address the separation = of code and assets: we don't want artists having to sync the code, and = this keeps the data definitions with the assets that they = define.<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>This is what I do, anyway. I have a data = language, which defines types, and which can also be used to define = data, and all data is stored using these definitions. This = includes bitmaps, sounds, models, etc. The data language is = separated from the data representation so this doesn't imply that = bitmaps are stored in a script-like array of colors - but it does mean = that tools can use the reflection system on more classes of data than = just small gameplay entities. And when distinctions like that are = removed, and all things are treated the same way, lots of clever things = just fall out.<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>Some other random = thoughts:<o:p></o:p></p></div><div><p class=3DMsoNormal>As far as I can = see, all C++ needs to allow full reflection is some sort of fieldsof(T) = operator that enumerates the fields of a structure. e.g. (typed = straight into email, so probably bollocks - and not representative of my = actual system!)<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>struct metadata {<o:p></o:p></p></div><div><p = class=3DMsoNormal><span = class=3Dapple-tab-span> &n= bsp; </span>enum TYPE {INT, PTR, ARRAY, STRUCT} = type;<o:p></o:p></p></div><div><p class=3DMsoNormal><span = class=3Dapple-tab-span> &n= bsp; </span>metadata(TYPE &_type) : type(_type) = {}<o:p></o:p></p></div><div><p = class=3DMsoNormal>};<o:p></o:p></p></div><div><p = class=3DMsoNormal>template<typename A, typename... B> struct = md_fields : md<A>, md_fields<B> = {};<o:p></o:p></p></div><div><p class=3DMsoNormal>template<typename = T> struct md : metadata, md_fields<fieldsof(T)> { = md() : metadata(STRUCT) {} };<o:p></o:p></p></div><div><p = class=3DMsoNormal>template<typename T> struct md<T*> : = metadata { md<T> t; md() : metadata(PTR) {} = };<o:p></o:p></p></div><div><p class=3DMsoNormal>template<typename T, = int N> struct md<T[N]> : metadata { int n; md<T> t; md() = : metadata(ARRAY), n(N) {} };<o:p></o:p></p></div><div><p = class=3DMsoNormal>template<> struct md<int> : metadata { = md() : metadata(INT) {} };<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>template<typename T> metadata *get_md(const T = &t) { static md<T> m; return &m; = }<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>Adrian Stephens<o:p></o:p></p></div><p = class=3DMsoNormal><o:p> </o:p></p><div><div><p class=3DMsoNormal>On = Mar 18, 2013, at 10:45 AM, Douglas Cox wrote:<o:p></o:p></p></div><p = class=3DMsoNormal><br><br><o:p></o:p></p><div><p class=3DMsoNormal>That = sounds pretty similar to the goals of the engine that we're working on = -- basically faster iteration times for everyone (this includes engine = programmers as well).<o:p></o:p></p><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>All of our code (including the compiler) is written in = a C#-like language, so we have perfect reflection from that. = Attributes are pretty much required for providing additional = information to property editors for things like valid ranges of a field = or notifications of a property change (most of which can go away in = Final builds).<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>Another thing to consider may be allowing for data = derivation. This can save quite a bit of work when the need for = copies of objects come up that only need one or two properties changed. = If you pick a serialization format for the editable data that = knows about the data (fieldtype + fieldname + value), it makes it a lot = easier to get this working.<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>In the past, we've used Macros and they worked fine. = They're a bit more annoying to extend with optional parameters and = you have to repeat the field/property name, but that's not terrible = since you can generally get it down to compile-time errors if you get = something wrong (other than forgetting to add a new = field).<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>-Doug<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div></div><div><p = class=3DMsoNormal = style=3D'margin-bottom:12.0pt'><o:p> </o:p></p><div><p = class=3DMsoNormal>On Fri, Mar 15, 2013 at 3:45 AM, Tinco Andringa <<a = href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>> = wrote:<o:p></o:p></p><p class=3DMsoNormal>I think the point of the = discussion is to not just assume C++ isn't<br>going to do it. As far as = I understand the goal is to have the ability<br>to develop tools that = aid programmers and artist in iterating rapidly.<br>This means for = artists that they can swap artwork in at runtime or<br>quickly boot the = game from a serialized state with new artwork, not<br>just on his/her = own computer but also over a network to perhaps a<br>console. For = programmers this may mean that they can view the game<br>state easily, = launch the game in various states easily, swap in/out<br>scripts easily, = maybe even change code without fully recompiling or<br>even restarting = the game.<br><br>Am I missing something = important?<o:p></o:p></p><div><div><p class=3DMsoNormal><br>On Fri, Mar = 15, 2013 at 8:33 AM, Massimo Del Zotto <<a = href=3D"mailto:[email protected]">[email protected]</a>> = wrote:<br>> I have limited experience but I have been very impressed = by the separation<br>> provided by scripting languages.<br>> = Considering the goals:<br>><br>> My goal is to have fast iteration = times, both artists and programmers<br>><br>> It appears to me C++ = isn't going to do it. Especially on the artist side of<br>> the = thing.<br>><br>> By the way, I would like to have an insight on = how this reflection<br>> information is to be used. If it is used to = simulate duck typing I'd use a<br>> tool which supports duck typing = natively.<br>><br>> = Massimo<br>><o:p></o:p></p></div></div><div><div><p = class=3DMsoNormal>> = _______________________________________________<br>> Sweng-Gamedev = mailing list<br>> <a = href=3D"mailto:[email protected]">Sweng-Gamedev@lists= .midnightryder.com</a><br>> <a = href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnigh= tryder.com" = target=3D"_blank">http://lists.midnightryder.com/listinfo.cgi/sweng-gamed= ev-midnightryder.com</a><br>><br>_____________________________________= __________<br>Sweng-Gamedev mailing list<br><a = href=3D"mailto:[email protected]">Sweng-Gamedev@lists= .midnightryder.com</a><br><a = href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnigh= tryder.com" = target=3D"_blank">http://lists.midnightryder.com/listinfo.cgi/sweng-gamed= ev-midnightryder.com</a><o:p></o:p></p></div></div></div><p = class=3DMsoNormal><o:p> </o:p></p></div><p = class=3DMsoNormal>_______________________________________________<br>Swen= g-Gamedev mailing list<br><a = href=3D"mailto:[email protected]">Sweng-Gamedev@lists= .midnightryder.com</a><br><a = href=3D"http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnigh= tryder.com">http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-mid= nightryder.com</a><o:p></o:p></p></div><p = class=3DMsoNormal><o:p> </o:p></p></div></body></html> ------=_NextPart_000_001F_01CE2417.E18683E0-- --===============1117138264== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Sweng-Gamedev mailing list [email protected] http://lists.midnightryder.com/listinfo.cgi/sweng-gamedev-midnightryder.com --===============1117138264==--