Re: senator-next-tag: Buffer was not parsed by Semantic.

Hi Oleg,

Replacing the lexer is pretty easy.   In most languages (I'll use dot 
again) there is a line that looks like this:

   (setq
    ;; Lexical Analysis
    semantic-lex-analyzer 'wisent-dot-lexer
    ...

which was created like this:

(define-lex wisent-dot-lexer
   "Lexical analyzer that handles DOT buffers.
It ignores whitespace, newlines and comments."
   semantic-lex-ignore-whitespace
   ...

If you follow the doc trail, you end up with this:

-----------
semantic-lex is an autoloaded compiled Lisp function in `lex.el'.

(semantic-lex START END &optional DEPTH LENGTH)

Lexically analyze text in the current buffer between START and END.
Optional argument DEPTH indicates at what level to scan over entire
lists.  The last argument, LENGTH specifies that `semantic-lex'
should only return LENGTH tokens.  The return value is a token stream.
Each element is a list, such of the form
   (symbol start-expression .  end-expression)
where SYMBOL denotes the token type.
See `semantic-lex-tokens' variable for details on token types.  END
does not mark the end of the text scanned, only the end of the
beginning of text scanned.  Thus, if a string extends past END, the
end of the return token will be larger than END.  To truly restrict
scanning, use `narrow-to-region'.
----------

So this function will parse an entire buffer and return all the lexical 
tokens for it.

You can put anything you want in there, and return tokens with any old 
SYMBOL you want too.

Semantic's first lexer (see semantic-flex) is an example of a different 
standalone lexer.  It was only after struggling with that for a while 
that the mechanism for making customer lexers came up.  It is partly 
modeled after lex/flex where regexp are associated with actions.

If regexp don't make sense for your language, then rolling your own is 
no problem.   You still need to add %token expressions in your grammar 
to let the grammar know what is going on though.  You just don't need to 
specify all the regexp along the way, or use the automatically generated 
lexers.

Your big scary regexp below is not really necessary for writing lexers 
using the lexing technique I described last time though.  Each 
expression only needs to be as long as the small piece you are looking 
at (ie - a number or symbol).   If you find that your language has a 
lexical token whose type is based on previous lexical tokens, then you 
are right that the built in lexer is probably insufficient.

The C parser has examples that parse out things like:

#include <foo.h>

and

#if SOMESYMBOL
#endif

that way, but it gets pretty hairy.

Eric

On 08/16/2014 06:52 PM, Left Right wrote:
> Just to give you a sense of what I /don't/ want to have in my code
> (below is my own code, so I'm allowed to say that it's an
> unmaintainable cuneiform)
>
> (defvar fmt-font-lock-keywords
>    ;; no-args
>    `(("~\\(@:?\\|:@?\\)?[]>()}aswvcp;_]"
>       (0 font-lock-keyword-face))
>      ;; numeric-arg
>      ("~\\([0-9]*\\|#,?\\)\\(@:?\\|:@?\\)?[i*%&|~{[]"
>       (0 font-lock-keyword-face))
>      ;; decimal
>      ("~\\([0-9]*\\|#\\(,[0-9]*\\|#\\)\\{0,3\\}\\)?\\(@:?\\|:@?\\)?[rdbox]"
>       (0 font-lock-keyword-face))
>      ;; floating-point f
>      (,(concat
>         "~\\(\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{3\\}\\(,'\\w\\)\\{1,2\\}\\)\\|"
>         "\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,3\\}\\)\\)"
>         "?\\(@:?\\|:@?\\)?f")
>       (0 font-lock-keyword-face))
>      ;; floating-point e, g
>      (,(concat
>         "~\\(\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{3\\}\\(,'\\w\\)\\{1,3\\}\\)\\|"
>         "\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,3\\}\\)\\)"
>         "?\\(@:?\\|:@?\\)?[eg]")
>       (0 font-lock-keyword-face))
>      ;; currency
>      (,(concat
>         "~\\(\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{2\\}\\(,'\\w\\)\\)\\|"
>         "\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,2\\}\\)\\)"
>         "?\\(@:?\\|:@?\\)?[$]")
>       (0 font-lock-keyword-face))
>      ;; tabulation
>      ("~\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)?\\)?\\(@:?\\|:@?\\)?t"
>       (0 font-lock-keyword-face))
>      ;; escape
>      ("~\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,2\\}\\)?\\(@:?\\|:@?\\)?^"
>       (0 font-lock-keyword-face))
>      ;; logical block
>      ("~\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,3\\}\\)?\\(@:?\\|:@?\\)?<"
>       (0 font-lock-keyword-face))
>      ;; custom function
>      (,(concat
>         "~\\(\\([0-9]+\\|'\\w\\|#\\)\\(,\\([0-9]+\\|'\\w\\|#\\)+\\)*\\)?"
>         "\\(@:?\\|:@?\\)?\\/[^\\s\\n,#@]+\\/")
>       (0 font-lock-keyword-face))))
>
> This is my previous version of font-lock coloring. I don't expect you
> to read through it, but just to make the point even more obvious: this
> is actually a single regular expression, which I chopped into pieces
> for "ease" of use. The lexer based on regexp would need to have this
> mess concatenated into a single expression. Maybe it can be
> simplified, but not by much. The corresponding parsing function, which
> doesn't use regular expressions would be somewhere between 1/3 and 1/2
> of the above code, and it would be perfectly understandable. This is
> the case similar to email parsing: you can do it with regular grammar,
> in principle, but there is no good way to do it in practice.
>
> I later found that I can provide a function to font-lock to replace
> this mess, and I would be happy if there was a way to do the same in
> place of the Semantic lexer.
>
> Best,
>
> Oleg
>
> On Sun, Aug 17, 2014 at 1:12 AM, Left Right<[email protected]>  wrote:
>> Hi and thanks for thorough replies. I think I will need to go through
>> them again, but so far I have one questions, which may possibly spare
>> me trying to intern all of this material.
>>
>> My last question wrt define-lex-regex wasn't about how could I make a
>> regular expression based lexer. The truth is: I don't want any regular
>> expressions there, they are just not cut for the task. So, let me
>> rephrase it: can I ditch the whole mechanism of the lexer and replace
>> it with a brand new code, which will handle the tokenization? What
>> will I need to do to achieve this? I don't have much experience with
>> writing lexers, in fact, I only ever used cl-yacc, which simply leaves
>> it to the programmer to implement a lexer and only requires a very
>> simple interface: a function that accepts an input stream and returns
>> a token.
>>
>> The reason I'm asking: the tokenizer mechanism looks very complex to
>> me, way more complex than what I need, besides, it works in a very
>> inconvenient way: if I could keep the state between the calls to the
>> lexer it would make my life so much easier. I also don't want to
>> depend on syntax table and whatever bizarre rules Emacs uses to
>> understand the syntax: this is not your fault, but I have an
>> impression that many of these rules are there purely by accident, they
>> are hard to discover and even harder to understand, because they don't
>> match anything you might come to expect from a lexer / parser. It
>> would be just a whole lot easier to start fresh, than to try to glue
>> together a bunch of jigsaw puzzle pieces, clearly taken from different
>> puzzles.
>>
>> I can understand the motivation for someone who gets the syntax table
>> and mode coloring for free from an existing mode and wants to reuse it
>> in order to build the lexer. I don't have these preconditions, and
>> even the little that I do have, I've written myself, and I'd rather
>> give it up to make the overall process more consistent. I.e. I don't
>> want to have to design font-lock rules, syntax table and the lexer
>> separately: to me this would be like doing the same work twice, but
>> both times using inappropriate tools.
>>
>> Best,
>>
>> Oleg
>>
>> On Sun, Aug 10, 2014 at 5:41 PM, Eric M. Ludlam<[email protected]>  wrote:
>>> On 08/02/2014 05:28 PM, Left Right wrote:
>>>>
>>>> One more question, I'm trying to follow the inline code documentation,
>>>> and here's something I came up with, but I have lots of questions
>>>> about it:
>>>>
>>>> (define-lex-regex-analyzer fmt-lex-filler
>>>>     "Matches the filler in the format string."
>>>>     "[^~]+"
>>>>     (semantic-lex-push-token
>>>>        (semantic-lex-token
>>>>         'filler (match-beginning 0) (match-end 0))))
>>>>
>>>> (define-lex wisent-fmt-lexer
>>>>     "Lexical analyzer that handles Common Lisp format."
>>>>     fmt-lex-filler)
>>>>
>>>> 1. Using regular expression in this analyzer is a really, really bad
>>>> idea (the proper regexp is more than 300 characters long, this one is
>>>> here just for illustration), but this complexity can be easily avoided
>>>> if instead of regular expression I could use a function that takes,
>>>> say, position in the buffer or something like that: is that even
>>>> possible?
>>>>
>>>> 2. 'filler isn't a default kind of token, is my guess correct that I
>>>> can somehow refer to this kind in the grammar, similar to how %type
>>>> <symbol>   is defined, maybe? What would I need to do to make this
>>>> possible?
>>>
>>>
>>> There is a default whitespace token you can create from your lexers. For
>>> exmaple, the dot lexer starts with these:
>>>
>>>    semantic-lex-ignore-whitespace
>>>    semantic-lex-ignore-newline
>>>    semantic-lex-ignore-comments
>>>
>>> which is implemented like this:
>>>
>>> (define-lex-regex-analyzer semantic-lex-ignore-whitespace
>>>    "Detect and skip over whitespace tokens."
>>>    ;; catch whitespace when needed
>>>    "\\s-+"
>>>    ;; Skip over the detected whitespace, do not create a token for it.
>>>    (setq semantic-lex-end-point (match-end 0)))
>>>
>>>
>>> which means "go to the end of the match, and don't return a token.  As you
>>> have in your lexer, you have to push the 'filler token to get it on the
>>> stack.
>>>
>>> The reason you have to set the end point is because when you push a token,
>>> it looks at the end of your token, and moves there automatically, but if you
>>> don't push a token, you have to move it by hand.
>>>
>>> Lexical analyzers are interesting, in that while a function is made for
>>> them, those functions aren't used.  Instead they also have a value, and
>>> those values are concatenated together to create the master lexer function.
>>> Like a big cond statement.  The main lexer has logic it applies after each
>>> match is found, and that is where a bunch of the magic happens.
>>>
>>> If you aren't trying to ignore your 'filler tokens, you will instead need a
>>> %token declaration for it, such as:
>>>
>>> %token filler
>>>
>>> If you instead had
>>>
>>> %type<filler>  syntax "[^~]+"
>>>
>>> you wouldn't need to write your lexical analyzer at all and one would be
>>> provided for you.  (I think, I'm a little fuzzy on that one.)
>>>
>>> Your filler lexer is OK if it is something you really need, but because it
>>> can match so much, you MUST put it at the END of your defined lexer.  That
>>> way you will be able to match all your other expressions, and if nothing
>>> works, you call it filler.
>>>
>>> use:
>>>
>>> M-x semantic-lex-test RET
>>>
>>> to see how it works, or
>>>
>>> M-x semantic-lex-debug RET
>>>
>>> to watch your lexer run.
>>>
>>> Good Luck
>>> Eric
>

------------------------------------------------------------------------------