Re: source-to-source translators

Roman Yakovenko wrote:
> On 6/16/05, Stefan Seefeld <[email protected]> wrote:

>>In synopsis, pipelines are a compositional pattern by which processors
>>are chained into composite processors
>>(http://synopsis.fresco.org/docs/Tutorial/pipeline.html). I have found
>>this idiom to be very productive, and while it is currently only directly
>>applied to ASTs, I'm hoping that it can be applied to other representations
>>as well. Most importantly to the user, pipelines are very flexible and
>>extensible. I came up with the current design specifically to solve the
>>problem I had when trying to express *lots* of parameters that should
>>apply to the data processing, some related to the parsing itself, others
>>to the formatting.
> 
> 
> I know this design pattern. What about performance? Did you see change
> for the worse?

I haven't any data to compare against. Also, I'm not terribly worried
about the pipeline as a pattern, as the protocol is very coarse-grained.

What makes a difference for performance is the implementation of the
data representation and individual processors traversing them.

Right now the only representation being exposed to processors is
the AST, and it is entirely implemented in python. I'm not sure yet
how applying this idiom to the parse tree will work out. That's
definitely something to experiment with.

>>Synopsis at present does some form of type analysis when translating the parse
>>tree to an AST, so it *might* already work.
> 
> 
> Do you have list of cases when synopsis type analysis will not work? 
> I'd like to see it before integrating pygccxml with synopsis.

No, and I'm not sure what requirements you have for type analysis.
I should probably explain a little bit the current design to make it clear
what synopsis does at present, and where I think it will go:

The OpenC++ parser (which synopsis internally uses) originally created the parse tree
without any notion of 'symbol' and 'type', as these were only introduced in a second
pass of 'parse tree translation'.
Synopsis at present wraps this 'OpenC++ parser' within a python extension module, and
translates this parse tree (C++) into the AST (python), and does symbol lookup as well
as *some* type analysis during this translation.

This needs some redesign, for various reasons:

* The parser itself needs to do *some* symbol lookup in order to disambiguate the
   input. However, I don't think any type analysis is necessary for this.

* I'd like to make the (now internal) C++ API publicly available, in order to
   be able to write pure C++ applications using it.

* The type analysis is incomplete, as there is at present no way to associate template
   declarations / definitions with (partial) specializations, and thus a whole class
   of lookups will fail (i.e. there is no 'two phase lookup' possible right now).

As I said in my first mail, I'm aware of these limitations, and I'm trying to figure
out a roadmap to implement the missing pieces. Luckily, not all tasks synopsis can
be used for require all the above, so there are things for which synopsis can already
be considered 'correct', while there are others for which synopsis is still unsuitable.

I'd love to hear what others think about this, and in particular, if someone would
like to help in one area or another.

My impression was that pyste's current capabilities are such that synopsis would
be a good fit already. That's why I proposed to Nicodemus to adapt pyste to use
synopsis as an alternative backend. Once this works (and any emerging issues are
resolved) we could think about how to open up pyste to make it work as an extensible
pipeline much like synopsis. Then we could incrementally add new features that require
e.g. more complete type analysis (for example setting call policies automatically,
based on types, etc.).

Does this make sense ?

>>But that nitpicking aside, I believe I understand what you are saying:
>>To solve the problem of 'source translation' in general, we need a
>>language that is expressive enough for this purpose, i.e. that lets
>>us express constraints, conditions, as well as actual transformations.
> 
> 
> Yes, this is exactly what I mean. This solution is very powerful, but it has 
> it's own problems. One of them is complexity. For example - boost.spirit
> library introduces domain specific language, using C++ syntax. We could
> build something similar ( in idea ) using python. But I am not sure that
> users will learn an other language to complete the task ( exposing C++
> declarations
> to python - that is the main case for me ). I am not talking about the
> challenge
> to build such language. On other problem is "time to market". It will
> take a huge amount
> of time to build such thing. 

I think the challenge here is to find a roadmap that makes this an incremental
process, i.e. that provides 'check points' for which *useful tools* can
be written that only require the features *implemented so far*.
This will hopefully provide the necessary motivation, and attract more people.

> Right now I am writing documentation and examples for my project ( py++ ).
> When it will be ready, if you don't mind, I will contact you again, in
> order we can
> transform, :), our discussion to something useful. 

I'm very much looking forward to it !

Regards,
		Stefan