Re: senator-next-tag: Buffer was not parsed by Semantic.

Just to give you a sense of what I /don't/ want to have in my code
(below is my own code, so I'm allowed to say that it's an
unmaintainable cuneiform)

(defvar fmt-font-lock-keywords
  ;; no-args
  `(("~\\(@:?\\|:@?\\)?[]>()}aswvcp;_]"
     (0 font-lock-keyword-face))
    ;; numeric-arg
    ("~\\([0-9]*\\|#,?\\)\\(@:?\\|:@?\\)?[i*%&|~{[]"
     (0 font-lock-keyword-face))
    ;; decimal
    ("~\\([0-9]*\\|#\\(,[0-9]*\\|#\\)\\{0,3\\}\\)?\\(@:?\\|:@?\\)?[rdbox]"
     (0 font-lock-keyword-face))
    ;; floating-point f
    (,(concat
       "~\\(\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{3\\}\\(,'\\w\\)\\{1,2\\}\\)\\|"
       "\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,3\\}\\)\\)"
       "?\\(@:?\\|:@?\\)?f")
     (0 font-lock-keyword-face))
    ;; floating-point e, g
    (,(concat
       "~\\(\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{3\\}\\(,'\\w\\)\\{1,3\\}\\)\\|"
       "\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,3\\}\\)\\)"
       "?\\(@:?\\|:@?\\)?[eg]")
     (0 font-lock-keyword-face))
    ;; currency
    (,(concat
       "~\\(\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{2\\}\\(,'\\w\\)\\)\\|"
       "\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,2\\}\\)\\)"
       "?\\(@:?\\|:@?\\)?[$]")
     (0 font-lock-keyword-face))
    ;; tabulation
    ("~\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)?\\)?\\(@:?\\|:@?\\)?t"
     (0 font-lock-keyword-face))
    ;; escape
    ("~\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,2\\}\\)?\\(@:?\\|:@?\\)?^"
     (0 font-lock-keyword-face))
    ;; logical block
    ("~\\(\\([0-9]*\\|#\\)\\(,\\([0-9]*\\|#\\)\\)\\{0,3\\}\\)?\\(@:?\\|:@?\\)?<"
     (0 font-lock-keyword-face))
    ;; custom function
    (,(concat
       "~\\(\\([0-9]+\\|'\\w\\|#\\)\\(,\\([0-9]+\\|'\\w\\|#\\)+\\)*\\)?"
       "\\(@:?\\|:@?\\)?\\/[^\\s\\n,#@]+\\/")
     (0 font-lock-keyword-face))))

This is my previous version of font-lock coloring. I don't expect you
to read through it, but just to make the point even more obvious: this
is actually a single regular expression, which I chopped into pieces
for "ease" of use. The lexer based on regexp would need to have this
mess concatenated into a single expression. Maybe it can be
simplified, but not by much. The corresponding parsing function, which
doesn't use regular expressions would be somewhere between 1/3 and 1/2
of the above code, and it would be perfectly understandable. This is
the case similar to email parsing: you can do it with regular grammar,
in principle, but there is no good way to do it in practice.

I later found that I can provide a function to font-lock to replace
this mess, and I would be happy if there was a way to do the same in
place of the Semantic lexer.

Best,

Oleg

On Sun, Aug 17, 2014 at 1:12 AM, Left Right <[email protected]> wrote:
> Hi and thanks for thorough replies. I think I will need to go through
> them again, but so far I have one questions, which may possibly spare
> me trying to intern all of this material.
>
> My last question wrt define-lex-regex wasn't about how could I make a
> regular expression based lexer. The truth is: I don't want any regular
> expressions there, they are just not cut for the task. So, let me
> rephrase it: can I ditch the whole mechanism of the lexer and replace
> it with a brand new code, which will handle the tokenization? What
> will I need to do to achieve this? I don't have much experience with
> writing lexers, in fact, I only ever used cl-yacc, which simply leaves
> it to the programmer to implement a lexer and only requires a very
> simple interface: a function that accepts an input stream and returns
> a token.
>
> The reason I'm asking: the tokenizer mechanism looks very complex to
> me, way more complex than what I need, besides, it works in a very
> inconvenient way: if I could keep the state between the calls to the
> lexer it would make my life so much easier. I also don't want to
> depend on syntax table and whatever bizarre rules Emacs uses to
> understand the syntax: this is not your fault, but I have an
> impression that many of these rules are there purely by accident, they
> are hard to discover and even harder to understand, because they don't
> match anything you might come to expect from a lexer / parser. It
> would be just a whole lot easier to start fresh, than to try to glue
> together a bunch of jigsaw puzzle pieces, clearly taken from different
> puzzles.
>
> I can understand the motivation for someone who gets the syntax table
> and mode coloring for free from an existing mode and wants to reuse it
> in order to build the lexer. I don't have these preconditions, and
> even the little that I do have, I've written myself, and I'd rather
> give it up to make the overall process more consistent. I.e. I don't
> want to have to design font-lock rules, syntax table and the lexer
> separately: to me this would be like doing the same work twice, but
> both times using inappropriate tools.
>
> Best,
>
> Oleg
>
> On Sun, Aug 10, 2014 at 5:41 PM, Eric M. Ludlam <[email protected]> wrote:
>> On 08/02/2014 05:28 PM, Left Right wrote:
>>>
>>> One more question, I'm trying to follow the inline code documentation,
>>> and here's something I came up with, but I have lots of questions
>>> about it:
>>>
>>> (define-lex-regex-analyzer fmt-lex-filler
>>>    "Matches the filler in the format string."
>>>    "[^~]+"
>>>    (semantic-lex-push-token
>>>       (semantic-lex-token
>>>        'filler (match-beginning 0) (match-end 0))))
>>>
>>> (define-lex wisent-fmt-lexer
>>>    "Lexical analyzer that handles Common Lisp format."
>>>    fmt-lex-filler)
>>>
>>> 1. Using regular expression in this analyzer is a really, really bad
>>> idea (the proper regexp is more than 300 characters long, this one is
>>> here just for illustration), but this complexity can be easily avoided
>>> if instead of regular expression I could use a function that takes,
>>> say, position in the buffer or something like that: is that even
>>> possible?
>>>
>>> 2. 'filler isn't a default kind of token, is my guess correct that I
>>> can somehow refer to this kind in the grammar, similar to how %type
>>> <symbol>  is defined, maybe? What would I need to do to make this
>>> possible?
>>
>>
>> There is a default whitespace token you can create from your lexers. For
>> exmaple, the dot lexer starts with these:
>>
>>   semantic-lex-ignore-whitespace
>>   semantic-lex-ignore-newline
>>   semantic-lex-ignore-comments
>>
>> which is implemented like this:
>>
>> (define-lex-regex-analyzer semantic-lex-ignore-whitespace
>>   "Detect and skip over whitespace tokens."
>>   ;; catch whitespace when needed
>>   "\\s-+"
>>   ;; Skip over the detected whitespace, do not create a token for it.
>>   (setq semantic-lex-end-point (match-end 0)))
>>
>>
>> which means "go to the end of the match, and don't return a token.  As you
>> have in your lexer, you have to push the 'filler token to get it on the
>> stack.
>>
>> The reason you have to set the end point is because when you push a token,
>> it looks at the end of your token, and moves there automatically, but if you
>> don't push a token, you have to move it by hand.
>>
>> Lexical analyzers are interesting, in that while a function is made for
>> them, those functions aren't used.  Instead they also have a value, and
>> those values are concatenated together to create the master lexer function.
>> Like a big cond statement.  The main lexer has logic it applies after each
>> match is found, and that is where a bunch of the magic happens.
>>
>> If you aren't trying to ignore your 'filler tokens, you will instead need a
>> %token declaration for it, such as:
>>
>> %token filler
>>
>> If you instead had
>>
>> %type<filler> syntax "[^~]+"
>>
>> you wouldn't need to write your lexical analyzer at all and one would be
>> provided for you.  (I think, I'm a little fuzzy on that one.)
>>
>> Your filler lexer is OK if it is something you really need, but because it
>> can match so much, you MUST put it at the END of your defined lexer.  That
>> way you will be able to match all your other expressions, and if nothing
>> works, you call it filler.
>>
>> use:
>>
>> M-x semantic-lex-test RET
>>
>> to see how it works, or
>>
>> M-x semantic-lex-debug RET
>>
>> to watch your lexer run.
>>
>> Good Luck
>> Eric

------------------------------------------------------------------------------