expandfull macro

Hi,

The expandfull macro has a peculiar property, which renders it 
useless(?) in languages whose nesting is not based on parenthesis. 
Consider the following Fortran code:

module my_mod
! comment: specification part
   real :: varname(1:5)

...

contains
! comment: contains part
   subroutine ...
...

end module my_mod

Even so there are no parenthesis, the "module-contains-end-module" 
introduces a nesting similarly to parenthesis. So my lexer generates the 
token stream

MODULE symbol semantic-list CONTAINS semantic-list newline

(where the end module my_mod part is simply represented as a newline 
token). Now using expandfull on the two semantic-lists to extract tags 
from the specification- and the contains part does not work. It took me 
a while to understand what goes wrong. The expandfull macro defined in 
grammar-macro recursively calls the lexer/parser with depth=1, which in 
the example above peels away one level too many, resulting in the 
unexpected token sequence:

REAL punctuation punctuation symbol open_paren number punctuation 
close_paren ...

The lexer wrongly dives into the (1:5) expression instead of generating 
a semantic-list (as is done by the semantic-lex-test, which adds to the 
confusion).
Well, I guess EXPANDFULL uses depth=1 instead of depth=0/nil because the 
semantic-list in a language like C starts and ends with a parenthesis, 
and so the lexer would not lex the enclosed part with depth=0/nil. Is 
this assessment correct? Somehow it feels wrong to hardcode depth=1 into 
a general purpose macro, more so as this is not documented.

My solution to this problem is to avoid EXPANDFULL and use my own 
expandfull function. A grammar action line would then look like
           :members (semantic-f90-expandfull $5 'one-specification)

Would that be a viable solution?

I was also considering generating context specific tokens instead of 
semantic-list, because the lexer already does some simple parsing and 
using context specific tokens makes the grammar rules more readable. In 
the example above I would like to generate something like this:

   MODULE symbol tok_mod_specpart CONTAINS tok_mod_body newline

However, I could not figure out how to add my own tok_something token, 
which is not bound to some string or regular expression. Is there a way 
to do that? Would that be reasonable?

Martin


------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net