Re: symbol lookup and overload resolution

Richard Kelly wrote:
> Stefan Seefeld wrote:
> 
>>
>> That sounds great ! Can you describe what you are doing and how ? Once 
>> I understand what you want to do I may be able to suggest alternative 
>> approaches with the new APIs.
> 
> 
> I welcome your thoughts and suggestions.
> 
> I call it the Concept Modeling Tool (CMT).  I've been wanting to build 
> it for at least two years now, but lack of a C++ parser always stymied 
> my efforts.  By building CMT on top of Synopsis, I've made pretty good 
> progress in less than a month!  It's still in the prototype stage right 
> now, but I hope to have it ready for use around here in the next few 
> months.

Great.

> You may already be familiar with this terminology: (1) A "concept" is a 
> set of requirements that a software component, e.g. class C, may choose 
> to meet.  (2) A component is a "model" of a concept if and only if it 
> properly adheres to all of a concept's requirements.
> 
> CMT will let a developer do many things, but these are the big ones:
> 
> 1. CMT can mark a component as claiming to model a particular concept.
> 2. CMT can query components to find out what claims they make.
> 3. CMT can automatically generate code to satisfy a component's claims.
> 4. CMT can audit code to be sure components really satisfy their claims.

Interesting.

> Right now, my prototype works on components that are non-template 
> classes, structs, and unions (a.k.a. CSUs), and it understands two 
> concepts: Assignable and Comparable.  I've attached some descriptions 
> from CMT's documentation, but they're basically like similarly named 
> concepts from the STL.  Assignable handles the copy constructor and copy 
> assignment operator; Comparable handles all the equality and inequality 
> operators.  After examining the Synopsis-generated AST, CMT knows enough 
> to automatically write the code for all those functions.  Also, it can 
> use the AST's file/line markers to find any relevant, preexisting code 
> wherever it may be and replace it with updated code.

Generating new code is one thing, modifying existing code another.
While it may be relatively simple to generate code from a high level
model (such as the AST currently in use by synopsis), there also is
a low level API to manipulate the parse tree itself.
Part of this is the creation of new PTree::Node objects via factories,
the other part is to insert these nodes into the existing parse tree,
or replace existing nodes by new ones. That will do the necessary
bookkeeping, and when you then write out the underlaying buffer (Buffer
instance), it will take care to write out the correct nodes at the correct
positions.
The big advantage of this approach is that code generation is not lossy,
i.e. the changes are local, and everything else (inclusively formatting)
is preserved.

> By default, the code it generates is correct for most of the CSUs we 
> write, but for special cases, CMT will also let you put more marks in 
> the code.  For example, if class C uses data member m_cache to maintain 
> a private cache, you could mark m_cache in the source code, and CMT will 
> omit m_cache from the functions that it generates.  Or if the class has 
> really special needs, you could mark some of its claims as "manual". 
> That means, "Yes, this class really is a model of the XYZ concept, but 
> don't muck around with those functions."
> 
> By the way, the source code marks are all in the comments (similar to 
> Java tags), so it's still just regular C++ code to our compiler.

FWIW, the parser itself (which is based on a fork of the OpenC++ project
from about 1999 or 2000) is extensible in that you can - to a certain
extend - add new keywords and grammatical rules.
The goal was precisely to do the kind of thing you want to do, i.e.
support some form of metaprogramming.

I'm currently not sure whether or not to support that direction, as
with current scripting technologies it might be easier to place the
rules for the translation elsewhere, either inlined into comments or
completely outside the code (such as synopsis custom modules), and
do the processing from scriptable processors.

> Once CMT is ready (and we have a corral full of supported concepts), our 
> developers will be freed from writing many of the rote, mundane 
> functions they currently do by hand.  And maintenance will be easier. If 
> something changes -- let's say class C adds another data member -- the 
> developer can just tell CMT to update the generated code.

Yeah, there is a lot of boilerplate code that can be generated.
Coincidentally, when I started synopsis the first language I supported
was (not surprisingly) IDL, and the parser was omniidl, which does exactly
that: produce lots of boilerplate code such as CORBA stubs to be used
by client code...

> That's what it does.  Here's how it does it.
> 
> First, CMT tells Synopsis to parse the source code files.  Then CMT 
> examines the AST.  Suppose class C claims to be Assignable.  CMT uses 
> the AST to determine class C's base classes and data members.  Then it 
> generates class C's new copy constructor and copy assignment operator, 
> and it inserts them into (its temporary copy of) the source code in the 
> appropriate places.  Next it uses the AST to find out if class C already 
> had those functions declared and/or defined anywhere; if so, CMT removes 
> the obsolete stuff from the source code.  Finally, CMT overwrites the 
> original source files with its updated copies.  If necessary, CMT will 
> interact with the project's version control software to check out files 
> that it needs to modify, etc.

Hmm, I'd be curious to see how well the code generation works. While
the AST should be good enough for high level introspection such as manual
generation, I have never thought that it was complete enough to be usable
to actually regenerate the code it was produced from. As I said, generate
new code is one thing, but regenerating the original one quite another...

> As an aside, I'm building in several safety features to make sure that 
> CMT doesn't irretrievably mess up the source code.  For example, it'll 
> save copies of the original code in safe places, it'll support several 
> levels of "undo", and if it's not sure about some preexisting code, 
> it'll move that code to the bottom of the file instead of deleting it.

I'm generally very nervous about tools messing with hand-written code,
because they always require some (often undocumented) special formatting
to be able to work. Depending on whether the process of code generation
can be intrusive, I'd much prefer to use some form of 'Generation Gap',
i.e. put the 'interface definition' and the 'implementation' into separate
files, and then let the tool only mess with the interface. Much less harm
can be done that way.

> The beauty of CMT is, by since it relies on a full-featured parser to 
> guide its analysis, developers won't have to format their code in any 
> special ways, and they won't have to keep generated and handwritten code 
> in separate files, etc.

:-)

> One last thought: eventually CMT will handle things besides C++ code. As 
> one example, we have certain requirements for the directories containing 
> our source code, e.g. must follow certain naming conventions, must 
> contain a Makefile, must contain a README file, must contain a test case 
> subdirectory, and so on.  We could define a concept which embodies all 
> of that and then let CMT help us manage our directory trees.

Yeah. I was asked before whether synopsis could be used to audit code
according to some coding standards. I think that's definitely a good
application for it.

> I welcome your thoughts and suggestions.  I know that, right now, I'm 
> not using Synopsis to its full advantage or in the most effective ways. 
>  But even now, Synopsis has been a great help to me, and I'm learning 
> more about Synopsis as I build more and more of CMT.

That is great to hear. In particular, it is exciting to see people
actually using it for more than just documentation.
I believe that synopsis' design is very powerful. I'm now looking
for ways to expose more of the internal APIs publicly, but that requires
great care.

One thing that you may be interested in is the code generation on the
parse tree level. Lots of little pieces of code exist to support that
but as I'v never used that feature myself until now, I haven't worked
on it. May be, if we define clear goals, we can get it working quickly,
with unit tests and all the other jazz.

Regards,
		Stefan