Re: {Spam?} Re: state in parsers

Hi Kristof,

Sorry for a long delay in replying.

You can, if you need to, write your own lexer from scratch, and feed the 
results into the parser you choose.  Hopefully that isn't necessary.  As 
David notes, in Emacs Lisp, you can hack in most of what you need, and 
many parts are explicitly overridable.

You can use 'define-wisent-lexer' to create a new high-level lexer using 
whichever existing lex features you need, then in your mode setup hook, 
set the buffer local variable wisent-lexer-function, and wisent will 
call that.  Usually lexers analyze the whole buffer, then feed tokens 
into the parser.  You could make it wait, do it incrementally, or some 
other trick to do what you need.   You could use the existing lexers and 
fill out all the indent type tokens, and use the define-wisent-lexer 
function you create to toss out the extra symbols based on the parsers 
state which you will need to track.

There are no examples of this that I know of, so you will breaking new 
ground where we don't have much additional help for you. :(

Eric

On 12/16/2015 11:14 AM, Kristof Bastiaensen wrote:
> So wisent calls wisent-lex.  Could I implement semantic-lex to
> generate all tokens except implicit layout, then rewrite wisent-lex to take
> these tokens, handle context (based on indentations), and insert
> implicit-layout
> tokens when necessary.  Then the parser could pop the context where
> necessary.  This should work unless wisent-lex caches the tokens.
>
> Kristof
>
> On 15-12-15 20:20, David Engster wrote:
>> Kristof Bastiaensen writes:
>>> Can the lexer be controlled from the parser, or are they
>>> totally separated?  Can I modify lexer state from the parser?
>>> Is that such a hard question?
>> It is.
>>
>> The lexer and the parser are separated. But since this is Emacs Lisp,
>> nothing really is 'totally' separate. Since you can call pretty much any
>> Emacs Lisp code in your actions, you are free to set some variable there
>> which affects how your lexers work.
>>
>> The LL parser (Bovine) does something similar with the EXPAND/EXPANDFULL
>> actions, which incrementing a 'depth' variable and call the parser
>> recursively. This variable defines the depth with which the lexers
>> should look into '{}' and '()' blocks (otherwise they simply return a
>> 'semantic-list' token).
>>
>> AFAIK, the Wisent parser does not support this out of the box, but you
>> should be able to cook up something similar, but maybe you have to dig
>> into the Wisent code a bit to get this right. Sorry I cannot give you a
>> better answer.
>>
>> -David
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> cedet-semantic mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/cedet-semantic
>

------------------------------------------------------------------------------