Re: expandfull macro

Hi Martin,

Sorry for being slow to reply.  My kids robot building team has taken up 
all my free time this past month. :(

On 01/10/2015 05:37 PM, Martin Stein wrote:
> Hi,
>
> The expandfull macro has a peculiar property, which renders it
> useless(?) in languages whose nesting is not based on parenthesis.
> Consider the following Fortran code:
>
> module my_mod
> ! comment: specification part
>     real :: varname(1:5)
>
> ...
>
> contains
> ! comment: contains part
>     subroutine ...
> ...
>
> end module my_mod
>
> Even so there are no parenthesis, the "module-contains-end-module"
> introduces a nesting similarly to parenthesis. So my lexer generates the
> token stream
>
> MODULE symbol semantic-list CONTAINS semantic-list newline
>
> (where the end module my_mod part is simply represented as a newline
> token). Now using expandfull on the two semantic-lists to extract tags
> from the specification- and the contains part does not work. It took me
> a while to understand what goes wrong. The expandfull macro defined in
> grammar-macro recursively calls the lexer/parser with depth=1, which in
> the example above peels away one level too many, resulting in the
> unexpected token sequence:
>
> REAL punctuation punctuation symbol open_paren number punctuation
> close_paren ...
>
> The lexer wrongly dives into the (1:5) expression instead of generating
> a semantic-list (as is done by the semantic-lex-test, which adds to the
> confusion).
> Well, I guess EXPANDFULL uses depth=1 instead of depth=0/nil because the
> semantic-list in a language like C starts and ends with a parenthesis,
> and so the lexer would not lex the enclosed part with depth=0/nil. Is
> this assessment correct? Somehow it feels wrong to hardcode depth=1 into
> a general purpose macro, more so as this is not documented.
>
> My solution to this problem is to avoid EXPANDFULL and use my own
> expandfull function. A grammar action line would then look like
>             :members (semantic-f90-expandfull $5 'one-specification)
>
> Would that be a viable solution?

If you had some lexical token that could act as the parens  ( ) that 
would be best, but I don't see those in your example.  I think in 
python, there is an INDENT and DETEND lexical token that is used to 
create a INDENT_BLOCK which shows up as the semantic-list.   I'm not 
sure how your lexer is set up, but you will need the lexer to generate a 
START and END type token for your custom lists (even if it is just 
whitespace) so that when you recurse it will peel them apart at the 
right level.


> I was also considering generating context specific tokens instead of
> semantic-list, because the lexer already does some simple parsing and
> using context specific tokens makes the grammar rules more readable. In
> the example above I would like to generate something like this:
>
>     MODULE symbol tok_mod_specpart CONTAINS tok_mod_body newline
>
> However, I could not figure out how to add my own tok_something token,
> which is not bound to some string or regular expression. Is there a way
> to do that? Would that be reasonable?

You can create any kind of custom lexer token you like.  Here is a 
custom token for C ## for macros:

(define-lex-simple-regex-analyzer semantic-lex-cpp-hashhash
   "Match ## inside a CPP macro as special."
   "##" 'spp-concat)

There are a bunch of ways to make analyzers with various levels of 
convenience built in.

I hope this helps.
Eric

------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/