Synopsis roadmap

Hello !

I have been eagerly awaiting the release of boost 1.33, as I'm planning to use
a number of boost libraries in synopsis (boost.filesystem, boost.wave, boost.python,
and may be others).

Now that the release is out (yay !) I'm going to slowly add dependencies to
synopsis' build system.
I'm going to describe my current plans for the short/mid-term synopsis work
I anticipate to be working on as I hope to find some contributions from other
interested parties. Some of the work I'm describing here is already underway.
Notably, some experimental code that is presently in the sandbox/ subdirectory
will be moved into mainline.


While I have been working on the C++ parser over the last year, the changes and
additions only marginally affected the code that is at present used to generate
the AST (which is still synopsis' backbone).
In particular, there is an entirely new symbol table module, and a little bit of
type analysis, but none of this is used to populate the AST.


I have now started to write a new AST generator that should be able to replace
the old one over the coming months. This will have three main benefits:

* A lot of old code will be removed. Cleaning up the code will simplify the
   dependencies between the different modules, in particular the parse tree
   API and the higher layers. Parsing C++ and generating an AST will also
   become faster, since redundant intermediate representations can be avoided
   (notably, a C++ version of the AST).

* The new modules (symbol table and type analysis) will be tested and debugged
   in the process, and in the end, generate better (i.e. 'more correct') results.

* This is an important step towards scriptable source-to-source translation.


The type analysis is still far from complete. Synopsis is not able to
match template specializations, for example, and thus do two-phase lookup, nor
does it understand standard conversions, thus limitting its ability to do
overload resolution.
Nonetheless I believe the new code will match and outweigh the functionality
of the old, but with a much cleaner design, thus opening the door for synopsis'
further evolution.

In the following I provide a detailed description of the tasks I'm
going to work on over the coming weeks:

Synopsis.AST backbone factorization
===================================

When synopsis was started, the AST was the only representation of interest.
In order to be able to let processors operate on other representations,
provide a new data structure that will optionally hold representations such
as the parse tree, symbol table, type repository, and of course the original
AST.
While the AST will continue to be a pure python data structure, the parse
tree and symbol table will be exposed through a C++ extension module, most
likely the ones in sandbox/bpl/, i.e. using boost.python.

While it may be useful to make all of these storable (i.e. persistent), this
is not part of the initial work. Processors wanting to operate on the parse
tree (say) simply will have to be called from the same process as the parse
tree generator (i.e. parser).

Cpp parser
==========

Work on the new Cpp parser based on boost.wave (in sandbox/wave). Switching
to wave has a number of advantages over the old preprocessor, the most visible
one being that the boost source code becomes again parsable by synopsis.
This task will most likely require collaboration with Hartmut Kaiser, the
wave author, in particular to overcome some current limitations of wave to
make it produce all the information required by synopsis.

C++ parser / AST generator
==========================

Work on the new AST generator in sandbox/Cxx/ to replace Synopsis/Parsers/Cxx.

Cross referencer
================

Work on the new cross referencer in sandbox/cross to replace the
Synopsis.Parsers.Cxx.link module. The design of the new cross referencer is quite
a bit different from the old one:
The old cross referencer relied on the parser to dump symbol information to a
separate file (typically in 'xref/' and 'link/' subdirs) to mark up the source file
in a second scan.
The new referencer marks up the parse tree itself by injecting xhtml
tags at appropriate places into the buffer. This makes the generated output
much more correct, but it also requires the input to be 'unpreprocessed',
i.e. it assumes that the preprocessing data can be recovered from the
Cpp parser (i.e. boost.wave) to work correctly.

This processor is also the first to use synopsis' ability to dump the modified
parse tree, i.e. *re*generate the code in a more or less modified form.





I hope that at least part of this work finds some interest from other developers /
observers. If you want to contribute, either by working on one of these tasks, or
by discussing the current and proposed design, please get in touch
(http://synopsis.fresco.org/contact.html) !

Regards,
		Stefan