Re: senator-next-tag: Buffer was not parsed by Semantic.

On 08/02/2014 05:28 PM, Left Right wrote:
> One more question, I'm trying to follow the inline code documentation,
> and here's something I came up with, but I have lots of questions
> about it:
>
> (define-lex-regex-analyzer fmt-lex-filler
>    "Matches the filler in the format string."
>    "[^~]+"
>    (semantic-lex-push-token
>       (semantic-lex-token
>        'filler (match-beginning 0) (match-end 0))))
>
> (define-lex wisent-fmt-lexer
>    "Lexical analyzer that handles Common Lisp format."
>    fmt-lex-filler)
>
> 1. Using regular expression in this analyzer is a really, really bad
> idea (the proper regexp is more than 300 characters long, this one is
> here just for illustration), but this complexity can be easily avoided
> if instead of regular expression I could use a function that takes,
> say, position in the buffer or something like that: is that even
> possible?
>
> 2. 'filler isn't a default kind of token, is my guess correct that I
> can somehow refer to this kind in the grammar, similar to how %type
> <symbol>  is defined, maybe? What would I need to do to make this
> possible?

There is a default whitespace token you can create from your lexers. 
For exmaple, the dot lexer starts with these:

   semantic-lex-ignore-whitespace
   semantic-lex-ignore-newline
   semantic-lex-ignore-comments

which is implemented like this:

(define-lex-regex-analyzer semantic-lex-ignore-whitespace
   "Detect and skip over whitespace tokens."
   ;; catch whitespace when needed
   "\\s-+"
   ;; Skip over the detected whitespace, do not create a token for it.
   (setq semantic-lex-end-point (match-end 0)))


which means "go to the end of the match, and don't return a token.  As 
you have in your lexer, you have to push the 'filler token to get it on 
the stack.

The reason you have to set the end point is because when you push a 
token, it looks at the end of your token, and moves there automatically, 
but if you don't push a token, you have to move it by hand.

Lexical analyzers are interesting, in that while a function is made for 
them, those functions aren't used.  Instead they also have a value, and 
those values are concatenated together to create the master lexer 
function.   Like a big cond statement.  The main lexer has logic it 
applies after each match is found, and that is where a bunch of the 
magic happens.

If you aren't trying to ignore your 'filler tokens, you will instead 
need a %token declaration for it, such as:

%token filler

If you instead had

%type<filler> syntax "[^~]+"

you wouldn't need to write your lexical analyzer at all and one would be 
provided for you.  (I think, I'm a little fuzzy on that one.)

Your filler lexer is OK if it is something you really need, but because 
it can match so much, you MUST put it at the END of your defined lexer. 
  That way you will be able to match all your other expressions, and if 
nothing works, you call it filler.

use:

M-x semantic-lex-test RET

to see how it works, or

M-x semantic-lex-debug RET

to watch your lexer run.

Good Luck
Eric

------------------------------------------------------------------------------