Re: Problem with lexical analyser in scm.el

On 07/29/2015 12:38 PM, Rupert Swarbrick wrote:
> In the current version of cedet/semantic/bovine/scm.el, we have the
> following:
>
>    ;; Note: Analyzer from Henry S. Thompson
>    (define-lex-regex-analyzer semantic-lex-scheme-symbol
>      "Detect and create symbol and keyword tokens."
>      "\\(\\sw\\([:]\\|\\sw\\|\\s_\\)*\\)"
>      ;; (message (format "symbol: %s" (match-string 0)))
>      (semantic-lex-push-token
>       (semantic-lex-token
>        (or (semantic-lex-keyword-p (match-string 0)) 'symbol)
>        (match-beginning 0) (match-end 0))))
>
> If you have something like the following:
>
>    (define (/foo arguments)
>      (write "Hello"))
>
> then semantic dies with an error when trying to parse the file. The
> regular expression doesn't match because the forward slash symbol has
> syntax class _ ("symbol") in Scheme mode (and would need to have syntax
> class "w"). Another problem with the regex is that it matches a name
> like 0foo, which I don't think it should do.
>
> I see there's a formal syntax in "The Scheme Programming Language, 4th
> edition", which you can find at:
>
>    http://www.scheme.com/tspl4d/grammar.html
>
> Matching that with a regex seems a bit difficult but I came up with the
> following, which appears to work:
>
>    (defun semantic-lex-scheme-identifier-at-point (&optional bound)
>      "Returns a pair (START . END) if there is a valid Scheme
>    identifier at point. Returns nil otherwise."
>      ;; This tries to follow the formal semantics given in The Scheme Programming
>      ;; Language, 4th ed.
>      (let ((initial-re "[a-zA-Z!$%&*/:<=>?~_^]")
>            (subsequent-re "[a-zA-Z!$%&*/:<=>?~_^0-9.+-@]"))
>        (when (or (looking-at initial-re)
>                  (member (get-char-code-property (char-after)
>                                                  'general-category)
>                          '("Lu" "Ll" "Lt" "Lm" "Lo" "Mn" "Nl" "No"
>                            "Pd" "Pc" "Po" "Sc" "Sm" "Sk" "So" "Co")))
>          (cons (point)
>                (save-excursion
>                  (while (or (looking-at subsequent-re)
>                             (member (get-char-code-property (char-after)
>                                                             'general-category)
>                                     '("Lu" "Ll" "Lt" "Lm" "Lo" "Mn" "Nl" "No"
>                                       "Pd" "Pc" "Po" "Sc" "Sm" "Sk" "So" "Co"
>                                       "Nd" "Mc" "Me")))
>                    (forward-char))
>                  (point))))))

Thanks for identifying this Rupert.

For cases where the syntax table is setup for editing convenience and 
does not match how the mode actually uses it, there is a way to update 
the syntax table for the lexer.   This patch shows how:

-----
diff --git a/lisp/cedet/semantic/bovine/scm.el 
b/lisp/cedet/semantic/bovine/scm.el
index 4e01c33..bba1f68 100644
--- a/lisp/cedet/semantic/bovine/scm.el
+++ b/lisp/cedet/semantic/bovine/scm.el
@@ -106,6 +106,10 @@ syntax as specified by the syntax table."
          imenu-create-index-function 'semantic-create-imenu-index
          )
    (setq semantic-lex-analyzer #'semantic-scheme-lexer)
+  (setq semantic-lex-syntax-modifications '((?/ "w")
+                                            (?@ "_")
+                                            )
+        )
    )

  (provide 'semantic/bovine/scm)

-------------

There are, however, a bunch of other things in your code snippet that I 
don't recognize.  Would an expanded version of my patch above solve the 
problem, or is there more to it?

Thanks
Eric

------------------------------------------------------------------------------