Trailing comments in bison

Maury Markowitz <[email protected]> Sun, 6 Dec 2020 11:24:40 -0500
Newsgroups gmane.comp.parsers.bison.general
Message-ID <[email protected]>
In (8-bit) MS BASIC, you can do this:

10 PRINT10:REM this is a remark

Or even:

20 PRINT20:' this is a remark too

But I came across some code, obvious in retrospect, that does this:

30 PRINT30' this is also a remark

In my flex I have these rules:

REM.*       { yylval.s = strndup(yytext + 3, yyleng - 3); return REM; }
'.*         { yylval.s = strndup(yytext + 1, yyleng - 1); return QUOTEREM; }

And in my bison I formerly had this:

statements:
    statement
    {
        $$ = g_list_prepend(NULL, $1);
    }
    |
    statement ':' statements
    {
	  $$ = g_list_prepend($3, $1);
    }

So it seemed obvious that I just needed:

    |
    statement "'" statements
    {
      $$ = g_list_prepend($3, $1);
    }

But that doesn't work, the scanner sees the "QUOTEREM" token and parses out the statement properly, but that eats the quote so it no longer matches this pattern. So that leaves it with the REM being part of the original statement, which then fails as a syntax error (rightfully so). I could rewind and uneat the quote, but that seems like the wrong solution. Or I could not parse the quote in flex and only look for them in the bison, but then I'd have to have a bunch of rules depending on whether it's the start of the line or inside it or has a colon, etc.

I assume this is not an uncommon issue, what's the canonical solution?