Re: senator-next-tag: Buffer was not parsed by Semantic.

One more question, I'm trying to follow the inline code documentation,
and here's something I came up with, but I have lots of questions
about it:

(define-lex-regex-analyzer fmt-lex-filler
  "Matches the filler in the format string."
  "[^~]+"
  (semantic-lex-push-token
     (semantic-lex-token
      'filler (match-beginning 0) (match-end 0))))

(define-lex wisent-fmt-lexer
  "Lexical analyzer that handles Common Lisp format."
  fmt-lex-filler)

1. Using regular expression in this analyzer is a really, really bad
idea (the proper regexp is more than 300 characters long, this one is
here just for illustration), but this complexity can be easily avoided
if instead of regular expression I could use a function that takes,
say, position in the buffer or something like that: is that even
possible?

2. 'filler isn't a default kind of token, is my guess correct that I
can somehow refer to this kind in the grammar, similar to how %type
<symbol> is defined, maybe? What would I need to do to make this
possible?

Thanks!

Oleg

On Sat, Aug 2, 2014 at 2:59 PM, Left Right <[email protected]> wrote:
> Hi Eric,
>
> Sorry it took me so long to reply. I was finally able to at least get
> the dot-mode to work. The way I managed was by requiring:
>
> (require 'cogre/dot-mode)
> (require 'cogre/wisent-dot)
> (require 'cogre/wisent-dot-wy)
>
> I also needed to update from Semantic bundled with Emacs 24.3.50 to
> the one I pulled from VCS today, otherwise, as I discovered post
> factum, it was trying to use a different parser (LR(1) instead of LL),
> I'm not sure how does this change come about, since the dot mode files
> didn't change across the versions. Yet when it was reading the grammar
> using LR parser, it would run into shift/reduce conflicts.
>
> I'm still struggling with my mode though, and, if you will be so kind,
> could you, please, explain few things about dot grammar?
>
> %type  <punctuation> syntax "\\s.+"
>
> I searched high and low, but I can't find an exhaustive reference to
> Emacs-style regexp, therefore I can't tell for sure what does this
> regexp mean: but I came to believe that it means a single "whitespace"
> character followed by whatever. I can't understand the meaning of this
> line, despite reading the documentation:
>
> ---- begin quote ----
>
> — %-Decl: %type <type-name> [property1 value1 ...]
>
> Explicitly declare a lexical type, and optionally give it properties.
>
> type-nameIs a symbol that identifies the type.
> propertyIs a property name, a valid Emacs Lisp symbol.
> valueIs a property value, a valid Emacs Lisp constant expression.
>
> Even if %token, %keyword, and precedence declarations can implicitly
> declare types, an explicit declaration is required for every type:
>
> To assign it properties.
> To auto-generate a lexical rule that detects tokens of this type. For
> more information, Auto-generation of lexical rules.
>
> ---- end quote ----
>
> What does this declaration do? This looks suspiciously similar to the
> entries in syntax table, but then it doesn't make much sense, since
> Emacs has a different way to mark punctuation...
>
> Second:
>
> %token <block>       BRACKET_BLOCK "(LBRACKET RBRACKET)"
>
> ---- begin quote ----
>
> The %token statement declares a terminal symbol (a token) which is not
> a keyword.
>
> — %-Decl: %token [<type-name>] token-name match-value
> — %-Decl: %token [<type-name>] token-name1 ...
>
> Respectively declare one token with an optional type, and a match
> value, or several tokens with the optional same type, and no match
> value.
>
> type-nameIs an optional symbol, enclosed between < and >, that
> specifies (and implicitly declares) a type for this token (see type
> Decl). If omitted the token has no type.
> token-nameIs the terminal symbol used in grammar rules to represent this token.
> match-valueIs an optional string. Depending on type-name properties,
> it will be interpreted as an ordinary string, a regular expression, or
> have a more elaborate meaning. If omitted the match value will be nil,
> which means that this token will be considered as the default token of
> its type (see type Decl for more information).
>
> ---- end quote ----
>
> The documentation speaks about some "more elaborate meaning". Can you
> tell, please, what is this meaning? The two things inside the
> parenthesis are another tokens which match literal brackets, but does
> this one match "[]" or "\\[[^\\]]+\\]"?
>
> Third:
>
> ;;; Bland default types
> %type  <symbol>
> %token <symbol> symbol
>
> %type  <string>
> %token <string> string
>
> %type  <number>
> %token <number> number
>
> I understand what is this supposed to do, but I can't understand how
> it achieves that. Can you, please, interpret that in words? To me this
> looks like magic: how does token `number' know how to match numbers?
>
> PS. The links to Bison documentation in the online version are broken
> (they point to www.randomsample.de instead of www.gnu.org)
>
> Thanks,
>
> Oleg
>
> On Sat, Jul 12, 2014 at 4:40 AM, Eric M. Ludlam <[email protected]> wrote:
>> Hi Oleg,
>>
>> I'm not sure how to debug the fcns you posted below.  I think they are ok.
>> Since you appear to be defining your own mode, let me instead annotate
>> through how the the parsing for "dot" works which is found in these files:
>>
>> lisp/cedet/cogre/dot-mode.el
>> lisp/cedet/cogre/wisent-dot.wy
>> lisp/cedet/cogre/wisent-dot.el
>>
>> and generated file
>>
>> lisp/cedet/cogre/wisent-dot-wy.el
>>
>> I picked this mode because it is pretty simple, just enough to get the
>> layout code of COGRE working.  It is also not installed by default in
>> semantic-new-buffer-setup-functions.
>>
>>
>> Lets start in dot-mode.el:
>>
>> Note the syntax table.  This part is critical for the lexer to work.  If you
>> duplicated some other mode, you probably have one of these.
>>
>> In cogre-dot-mode which is named such to avoid conflict with other dot
>> modes.  Note it sets up comment-start and comment-start-skip - these are
>> important for the lexer also.
>>
>> Also note the hook running at the end.
>>
>> Note the auto-mode-alist modification.
>>
>> Lastly, note the mode-local-parent stuff.  That is setup to make sure that
>> cogre-dot-mode is in agree with graphviz-dot-mode.  You don't need anything
>> like this if your mode is standalone.
>>
>>
>> Next is wisent-dot.wy.
>>
>> At the beginning is the langauage-mode setting that matches, in this case,
>> the core graphviz mode which I had to make optional.  I think you did this
>> correctly already.
>>
>> At the end after the %% is a lexer definition.  This uses a bunch of default
>> stuff, plus lexers defined in the language for keywords, etc.
>>
>> You can then compile this grammar into wisent-dot-wy.el.  If you are in a
>> compile debug cycle, you need to then enter wisent-dot-wy.el, and force eval
>> with C-M-x several tables because the defvars carefully save old values if
>> you just eval the buffer causing a no-op. :(
>>
>> Last is the key piece: wisent-dot.el
>>
>> Note that this pulls in wisent-dot-wy, plus wisent itself and any sources to
>> functions you need to override.
>>
>> The override for semantic-tag-components is important to implement if you
>> have ANY tags that are compound, such as a class with fields, etc.
>>
>> Note wisent-dot-setup-parser.  It installs the parser using a function from
>> wisent-dot-wy.el.  That is how the parser gets pulled in.
>>
>> It also sets up the lexer, extra syntax mods needed, and a few other rndom
>> things such as command separators and how to convert your tag classes into
>> text strings.  On the whole, the first statement and the first 2 variables
>> are the most important.  The rest is optional.
>>
>> Lastly are hooks to run the parser setup.
>>
>> These hooks can be replaced by adding the setup function to
>> semantic-new-buffer-setup-functions.
>>
>> Feel free to start with the hook, and use the setup function when you want
>> to make semantic support optional with your mode.
>>
>> If you already did all this, it could be that your parser is broken, or
>> parser recompiles are not getting loaded in correctly.  Fire up a new emacs
>> and load your code and test it to avoid the recompile issue.  If that helps,
>> you need to hand load variable changes from generated files.
>>
>> Another good trick is to use semantic-show-parser-state-mode.  this shows
>> symbols in the mode line to tell you how the parser is doing.  It will
>> either refuse to start if the parser is not installed, or show % if the
>> parser is broken, or if the buffer you are parsing is just not complete.
>>
>> Another fun one is semantic-highlight-edits-mode which shows how the buffer
>> is edited and reparsed which is helpful if the incremental parser is broken
>> with your language parser.
>>
>> Lastly use semantic-show-unmatched-syntax-mode to see if the parser is just
>> tagging your whole buffer as unparsable.  If this happens, you need to work
>> on your parser some more.
>>
>> I hope this helps.
>> Eric

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Cedet-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/cedet-devel