Accessing token type with .token()

Felix Nieuwenhuizen <[email protected]>
Newsgroups gmane.comp.parsers.bison.general
Message-ID <[email protected]>
(If you received this twice: I'm sorry, I'm not sure if my message was
accepted by mailman before I subscribed to the list)

Hi all,

I'm using bison with "Complete Symbols" with %define api.token.constructor,
and yylex is defined to return symbol_type with YY_DECL.

I have some testcases that test basic lexing.
Basically, I parse/lex some synthetic example files, and make sure the
right tokens and values are returned.
Here's a snippet:
for (auto id : ids) {
  auto symbol = lexer->yylex(*driver);
  ASSERT_EQ(symbol.token(), token::T_BASIC_IDENTIFIER);
  ASSERT_EQ(symbol.value.as<std::string>(), id);
}
(Where BASIC_IDENTIFIER is one of the tokens defined in my .yy file and
T_BASIC_IDENTIFIER is in the generated token_type / yytokentype enum)

This was working with Bison 3.4.x, but stopped working with Bison 3.5:
error: no member named 'token' in '...::Parser::symbol_type'
    ASSERT_EQ(symbol.token(), token::T_BASIC_IDENTIFIER);

token() was removed in commit 8c87a623, so the error is understandable.
Unfortunatly, I don't understand the distinction between internal and
external token numbers, so I'm not sure how to proceed.
I did try using .type / .type_get() instead of .token(), but as I
understand it these return the internal token numbers instead of the
external token numbers.
And since the yytokentype enum (e.g. T_BASIC_IDENTIFIER) uses the
external numbers, they don't match.

Maybe I'm doing something unorthodox, but what's the best way of
accessing/checking which token type is returned by yylex()?

Cheers, Felix
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.