Re: senator-next-tag: Buffer was not parsed by Semantic.

Hi and thanks for thorough replies. I think I will need to go through
them again, but so far I have one questions, which may possibly spare
me trying to intern all of this material.

My last question wrt define-lex-regex wasn't about how could I make a
regular expression based lexer. The truth is: I don't want any regular
expressions there, they are just not cut for the task. So, let me
rephrase it: can I ditch the whole mechanism of the lexer and replace
it with a brand new code, which will handle the tokenization? What
will I need to do to achieve this? I don't have much experience with
writing lexers, in fact, I only ever used cl-yacc, which simply leaves
it to the programmer to implement a lexer and only requires a very
simple interface: a function that accepts an input stream and returns
a token.

The reason I'm asking: the tokenizer mechanism looks very complex to
me, way more complex than what I need, besides, it works in a very
inconvenient way: if I could keep the state between the calls to the
lexer it would make my life so much easier. I also don't want to
depend on syntax table and whatever bizarre rules Emacs uses to
understand the syntax: this is not your fault, but I have an
impression that many of these rules are there purely by accident, they
are hard to discover and even harder to understand, because they don't
match anything you might come to expect from a lexer / parser. It
would be just a whole lot easier to start fresh, than to try to glue
together a bunch of jigsaw puzzle pieces, clearly taken from different
puzzles.

I can understand the motivation for someone who gets the syntax table
and mode coloring for free from an existing mode and wants to reuse it
in order to build the lexer. I don't have these preconditions, and
even the little that I do have, I've written myself, and I'd rather
give it up to make the overall process more consistent. I.e. I don't
want to have to design font-lock rules, syntax table and the lexer
separately: to me this would be like doing the same work twice, but
both times using inappropriate tools.

Best,

Oleg

On Sun, Aug 10, 2014 at 5:41 PM, Eric M. Ludlam <[email protected]> wrote:
> On 08/02/2014 05:28 PM, Left Right wrote:
>>
>> One more question, I'm trying to follow the inline code documentation,
>> and here's something I came up with, but I have lots of questions
>> about it:
>>
>> (define-lex-regex-analyzer fmt-lex-filler
>>    "Matches the filler in the format string."
>>    "[^~]+"
>>    (semantic-lex-push-token
>>       (semantic-lex-token
>>        'filler (match-beginning 0) (match-end 0))))
>>
>> (define-lex wisent-fmt-lexer
>>    "Lexical analyzer that handles Common Lisp format."
>>    fmt-lex-filler)
>>
>> 1. Using regular expression in this analyzer is a really, really bad
>> idea (the proper regexp is more than 300 characters long, this one is
>> here just for illustration), but this complexity can be easily avoided
>> if instead of regular expression I could use a function that takes,
>> say, position in the buffer or something like that: is that even
>> possible?
>>
>> 2. 'filler isn't a default kind of token, is my guess correct that I
>> can somehow refer to this kind in the grammar, similar to how %type
>> <symbol>  is defined, maybe? What would I need to do to make this
>> possible?
>
>
> There is a default whitespace token you can create from your lexers. For
> exmaple, the dot lexer starts with these:
>
>   semantic-lex-ignore-whitespace
>   semantic-lex-ignore-newline
>   semantic-lex-ignore-comments
>
> which is implemented like this:
>
> (define-lex-regex-analyzer semantic-lex-ignore-whitespace
>   "Detect and skip over whitespace tokens."
>   ;; catch whitespace when needed
>   "\\s-+"
>   ;; Skip over the detected whitespace, do not create a token for it.
>   (setq semantic-lex-end-point (match-end 0)))
>
>
> which means "go to the end of the match, and don't return a token.  As you
> have in your lexer, you have to push the 'filler token to get it on the
> stack.
>
> The reason you have to set the end point is because when you push a token,
> it looks at the end of your token, and moves there automatically, but if you
> don't push a token, you have to move it by hand.
>
> Lexical analyzers are interesting, in that while a function is made for
> them, those functions aren't used.  Instead they also have a value, and
> those values are concatenated together to create the master lexer function.
> Like a big cond statement.  The main lexer has logic it applies after each
> match is found, and that is where a bunch of the magic happens.
>
> If you aren't trying to ignore your 'filler tokens, you will instead need a
> %token declaration for it, such as:
>
> %token filler
>
> If you instead had
>
> %type<filler> syntax "[^~]+"
>
> you wouldn't need to write your lexical analyzer at all and one would be
> provided for you.  (I think, I'm a little fuzzy on that one.)
>
> Your filler lexer is OK if it is something you really need, but because it
> can match so much, you MUST put it at the END of your defined lexer.  That
> way you will be able to match all your other expressions, and if nothing
> works, you call it filler.
>
> use:
>
> M-x semantic-lex-test RET
>
> to see how it works, or
>
> M-x semantic-lex-debug RET
>
> to watch your lexer run.
>
> Good Luck
> Eric

------------------------------------------------------------------------------