Re: Moving along...

On 13 Aug 2002, Martin v. Loewis wrote:
...[Some notes about G++]

*Mumble mumble*...stupid typedefs...*mumble mumble*
*smirk*

> In any case, I very much would like to see support for a bison-style
> input grammar, i.e. with interleaved actions.
...
> With YAPPS, I found that outputting Python code is very useful,
> atleast if the Python code is readable - you then can easily trace the
> parsing with standard Python debugging techniques.

I think we can all win here.  The requirements I am advocating deal with a
grammar specification intermediate language (henceforth referred to as the
IL) that multiple front ends and back ends can support.  We simply have to
take into account that interleaved actions are highly coupled to a back
end, and this would break part of the generalized nature of an IL.  If we
can't come to a consensus on something that is back end neutral, it will
add a measure of complexity that may simply make my proposal unworkable.
Having multiple back ends means we can emit both debuggable Python code
and optimized state machines.

One of the initial problems I see with having actions embedded in a
grammar IL is that they may be hard to preserve across transformations of
that grammar.  As I mentioned before, we could still allow tree vistor
code to be placed inline with a grammar input file, but the visitor code
would be generated by a separate back end.  Even a generic AST vistor
framework may have problems with non-terminals that are created or removed
by various transformations of a grammar.

I think you'd like a pure Python generating back end to be required, and I
agree this is a must.  The initial direction my proposal would take is to
only require the library to include one front end and one back end, and
for both of these to be highly Pythonic.

I think that some of the weird things I want to do (like take pgen
grammars, transform them to Bison input files, and then translate the
resulting state machine back into a Python parser) would be a better fit
in the realm of 3rd party modules.  I just want the IL to be built into
Python rather than having to define my own.  My goal in doing this is to
not only provide a parser generator but a framework for integrating other
Python parser generators that are not part of the core library.  Such back
end integrations would allow us to transform disparate grammar definitions
and generate LL(1), LL(n), LALR(1), Early or whatever kinds of parsers
that are supported by external parser generators (most importantly Python
based parser generators).

> I still fail to see the point of generalizing pgen - I made "pure
> Python" a requirement precisely because it ports to Jython easily, and
> also backports to older Python versions better.

Sorry to mix all this stuff up.  PEP 269 is a separate issue, and clearly
does not begin to meet the requirements I've outlined.  I just thought I
might get some mileage about discussing it in this SIG.  Based on Guido's
reply, I think I should stop toying with the idea (it certainly won't take
an entire brain trust) and do it, least I face the ire of our BDFL.

This is all just a stepping stone in my agenda for parsing all software in
the world into a giant AST (such as Microsoft would like to have us do
with their proprietary Intentional Programming system).

Agenda:
1. Collect parsers for all languages.
2. Parse all available code into AST's.
3. Store AST's in a gigantic hierarchical database.
4. ...
5. Profit!

-Jon