Re: Wisent vs. Bison

On 05/18/2017 05:24 AM, hector wrote:
> I am trying to know if Wisent is suited for my purposes.
> 
> I'd like to know in what aspects it differs from Bison. Logically its
> internals are different.
> 
> I think one important difference (and perhaps this makes it not
> suitable) is the parser-lexer interface. This is defined by
> the definition of a token.
> 
> The tokens that Wisent takes as input are of the form:
> 
>   type start end
> 
> while in Bison they are (correct me if I'm wrong):
> 
>   type val
> 
> One consequence of this is that the bison lexer (flex)
> can do more work and give the parser something of higher
> level.

Hi Hector,

The tokens in Semantic have the start/end information so you can just 
ask for semantic-lex-token-text to get the value.  It may not convert an 
int to an int, but that isn't too hard if you need it.


> With Bison/Flex I can pass the string with backslashes removed.
> 
> The key here is that while bison takes a value (the type
> depends on toksym) Wisent takes a string.
> 
> So my general impression is that in Wisent the lexer is
> less powerful. Am I right?

The tagging parsers haven't needed that level of information, so it is 
missing.

That said, there is also a 4th string input to the semantic-lex-token 
constructor, so there is a variant where tokens can have a 4th element 
which enables the string value to be placed in the 2nd slot of the token 
list.

The C parser (which is using the bovine parser, not wisent) users this 
4th input to semantic-lex-token to compress complex macros stuff down to 
simpler values.  You can do that with the existing lex machinery so that 
you get the right string values in your match data.

In your second email you mention something about start states.  The 
default lexer doesn't really track state other than parenthetical depth. 
  You can control the depth at any level, where (parenthetical lists) 
are just returned as 'semantic-list' unless you ask to parse into them.

Does this help?
Eric

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot