Re: senator-next-tag: Buffer was not parsed by Semantic.

On 08/02/2014 07:59 AM, Left Right wrote:
> Hi Eric,
>
> Sorry it took me so long to reply. I was finally able to at least get
> the dot-mode to work. The way I managed was by requiring:
>
> (require 'cogre/dot-mode)
> (require 'cogre/wisent-dot)
> (require 'cogre/wisent-dot-wy)
>
> I also needed to update from Semantic bundled with Emacs 24.3.50 to
> the one I pulled from VCS today, otherwise, as I discovered post
> factum, it was trying to use a different parser (LR(1) instead of LL),
> I'm not sure how does this change come about, since the dot mode files
> didn't change across the versions. Yet when it was reading the grammar
> using LR parser, it would run into shift/reduce conflicts.

Hi Oleg,

My setup for CEDET in my .emacs is basically the same as in the INSTALL 
file with the version of CEDET from BZR, and that will load up .dot 
files just fine.   It is surprising to me you need all the extra loads. 
  Perhaps the build didn't create the autoload files for you?

> I'm still struggling with my mode though, and, if you will be so kind,
> could you, please, explain few things about dot grammar?
>
> %type<punctuation>  syntax "\\s.+"

In this case \s means "match a syntax type", and the "." means the 
syntax code for punctuation.   The \\ is quoting in one slash.

Here's a doc snippet:

`\sCODE'
      matches any character whose syntax is CODE.  Here CODE is a
      character that represents a syntax code: thus, `w' for word
      constituent, `-' for whitespace, `(' for open parenthesis, etc.
      To represent whitespace syntax, use either `-' or a space
      character.  *Note Syntax Class Table::, for a list of syntax codes
      and the characters that stand for them.

so the whole statement is "Create lexical tokens of type punctuation 
that matches the regular expression of punctuation from the Emacs syntax 
table.  In otherwords, it is a statement translating from Emacs speak to 
lexer speak.


> I searched high and low, but I can't find an exhaustive reference to
> Emacs-style regexp, therefore I can't tell for sure what does this
> regexp mean: but I came to believe that it means a single "whitespace"
> character followed by whatever. I can't understand the meaning of this
> line, despite reading the documentation:

There is a doc node in the "Elisp" manual called "Syntax of Regular 
Expressions" that I use.

> ---- begin quote ----
>
> — %-Decl: %type<type-name>  [property1 value1 ...]
>
> Explicitly declare a lexical type, and optionally give it properties.
>
> type-nameIs a symbol that identifies the type.

This would be a type for the lexer.

> propertyIs a property name, a valid Emacs Lisp symbol.
> valueIs a property value, a valid Emacs Lisp constant expression.

This lets you specify that the syntax (the property) matches some 
regexp.  If you leave it blank there are some handy defaults.

> Even if %token, %keyword, and precedence declarations can implicitly
> declare types, an explicit declaration is required for every type:
>
> To assign it properties.
> To auto-generate a lexical rule that detects tokens of this type. For
> more information, Auto-generation of lexical rules.
>
> ---- end quote ----
>
> What does this declaration do? This looks suspiciously similar to the
> entries in syntax table, but then it doesn't make much sense, since
> Emacs has a different way to mark punctuation...
>
> Second:
>
> %token<block>        BRACKET_BLOCK "(LBRACKET RBRACKET)"
>
> ---- begin quote ----

Once you have a lexical %type you can create %tokens that are more 
specific.   For example you might say"

%type <punctuation>  syntax "\\s."

to match a single punctuation, and then say

%token <punctuation> PLUS "+"

to create a token you can use in your grammar called plus.

This two step process lets the lexer quickly find your punctuation, and 
then convert generic punctuation into handy named tokens for use in your 
grammar.

<block> tokens are special in that the Emacs syntax table supports block 
concepts, and we use blocks to speed up grammar parsing.  While unusual 
in grammars, it lets us parse buffers more quickly by skipping over 
large chunks of text.

Thus the combination of:

%type  <block>
%token <block>       BRACKET_BLOCK "(LBRACKET RBRACKET)"
%token <open-paren>  LBRACKET      "["
%token <close-paren> RBRACKET      "]"

Says "I have a %type in my lexer called block".
I can create a <block> token that is composed of the LBRACKET and RBRACKET.
I have an <open-parent> lexical type called LBRACKET which matches [.

Then the lexer has a special 'depth' parameter, and if set to 0, will 
return BRACKET_BLOCK.  IN the match of BRACKET_BLOCK you can expand, and 
then get the LBRACKET token, like this:

graphgeneric
   : GRAPH BRACKET_BLOCK SEMI
     (TAG "GRAPH" 'generic-graph :attributes (EXPANDFULL $2 
attribute-block))
   ;

where EXPANDFULL on $2 says "run this grammar again on the buffer 
contents inside $2 (the BRACKET_BLOCK) starting with the grammar symbol 
"attribute-block".  The lexer will be run on that block with a depth of 
1, forcing it to look inside the parens (or brackets).

attribute-block
   : LBRACKET
     ()
   | RBRACKET
     ()
   | COMMA
     ()
  ;; This is a catch-all in case we miss some keyword.
   | symbol EQUAL name
     (TAG $1 'attribute :value $3)
   ;

so now in bracket block, we match the brackets, commas, etc, and just 
start creating tags for each attribute name found.

This set of nested tags needs to be matched outside of the dot grammar 
with a function for expanding tags.  In wisent-dot.el you will find 
semantic-tag-components which matches 'generic-graph from the first 
rule, and returns the :attributes which is the list of tags created with 
attribute-block.

> The %token statement declares a terminal symbol (a token) which is not
> a keyword.
>
> — %-Decl: %token [<type-name>] token-name match-value
> — %-Decl: %token [<type-name>] token-name1 ...
>
> Respectively declare one token with an optional type, and a match
> value, or several tokens with the optional same type, and no match
> value.
>
> type-nameIs an optional symbol, enclosed between<  and>, that
> specifies (and implicitly declares) a type for this token (see type
> Decl). If omitted the token has no type.
> token-nameIs the terminal symbol used in grammar rules to represent this token.
> match-valueIs an optional string. Depending on type-name properties,
> it will be interpreted as an ordinary string, a regular expression, or
> have a more elaborate meaning. If omitted the match value will be nil,
> which means that this token will be considered as the default token of
> its type (see type Decl for more information).
>
> ---- end quote ----
>
> The documentation speaks about some "more elaborate meaning". Can you
> tell, please, what is this meaning? The two things inside the
> parenthesis are another tokens which match literal brackets, but does
> this one match "[]" or "\\[[^\\]]+\\]"?

You are now pushing the boundary of what I am familiar with as I didn't 
develop most of this system.  Perhaps my earlier examples helped?

> Third:
>
> ;;; Bland default types
> %type<symbol>
> %token<symbol>  symbol
>
> %type<string>
> %token<string>  string
>
> %type<number>
> %token<number>  number
>
> I understand what is this supposed to do, but I can't understand how
> it achieves that. Can you, please, interpret that in words? To me this
> looks like magic: how does token `number' know how to match numbers?

In the wisent-dot example, the grammar code:

;;; Bland default types
%type  <symbol>
%token <symbol> symbol

is matched with:

(define-lex wisent-dot-lexer
   "Lexical analyzer that handles DOT buffers.
It ignores whitespace, newlines and comments."

...

   wisent-dot-wy--<symbol>-regexp-analyzer


and there is code generated like this in wisent-dot-wy.el

(define-lex-regex-type-analyzer wisent-dot-wy--<symbol>-regexp-analyzer
   "regexp analyzer for <symbol> tokens."
   "\\(\\sw\\|\\s_\\)+"
   nil
   'symbol)


so basically, there are default regular expressions for many types like 
symbol that will autogenerate lexer pieces.  You still need to assemble 
your lexer by ordering the pieces from most specific to most generic.


This is mostly derived from the fact that Emacs has a built-in 
lexer-like thing created using syntax tables.  A good major-mode defines 
a good syntax table, and then the lexer can be very simple, basically 
matching up syntax types via \\s to lexical types needed by the parser. 
  You can then overlay more specific token types on top of those.   By 
using the syntax table, the semantic lexer takes advantage of the Emacs 
scanners built in C, and can go very fast.

I hope that helps.
Eric

------------------------------------------------------------------------------
_______________________________________________
Cedet-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/cedet-devel