Problem with lexical analyser in scm.el

In the current version of cedet/semantic/bovine/scm.el, we have the
following:

  ;; Note: Analyzer from Henry S. Thompson
  (define-lex-regex-analyzer semantic-lex-scheme-symbol
    "Detect and create symbol and keyword tokens."
    "\\(\\sw\\([:]\\|\\sw\\|\\s_\\)*\\)"
    ;; (message (format "symbol: %s" (match-string 0)))
    (semantic-lex-push-token
     (semantic-lex-token
      (or (semantic-lex-keyword-p (match-string 0)) 'symbol)
      (match-beginning 0) (match-end 0))))

If you have something like the following:

  (define (/foo arguments)
    (write "Hello"))

then semantic dies with an error when trying to parse the file. The
regular expression doesn't match because the forward slash symbol has
syntax class _ ("symbol") in Scheme mode (and would need to have syntax
class "w"). Another problem with the regex is that it matches a name
like 0foo, which I don't think it should do.

I see there's a formal syntax in "The Scheme Programming Language, 4th
edition", which you can find at:

  http://www.scheme.com/tspl4d/grammar.html

Matching that with a regex seems a bit difficult but I came up with the
following, which appears to work:

  (defun semantic-lex-scheme-identifier-at-point (&optional bound)
    "Returns a pair (START . END) if there is a valid Scheme
  identifier at point. Returns nil otherwise."
    ;; This tries to follow the formal semantics given in The Scheme Programming
    ;; Language, 4th ed.
    (let ((initial-re "[a-zA-Z!$%&*/:<=>?~_^]")
          (subsequent-re "[a-zA-Z!$%&*/:<=>?~_^0-9.+-@]"))
      (when (or (looking-at initial-re)
                (member (get-char-code-property (char-after)
                                                'general-category)
                        '("Lu" "Ll" "Lt" "Lm" "Lo" "Mn" "Nl" "No"
                          "Pd" "Pc" "Po" "Sc" "Sm" "Sk" "So" "Co")))
        (cons (point)
              (save-excursion
                (while (or (looking-at subsequent-re)
                           (member (get-char-code-property (char-after)
                                                           'general-category)
                                   '("Lu" "Ll" "Lt" "Lm" "Lo" "Mn" "Nl" "No"
                                     "Pd" "Pc" "Po" "Sc" "Sm" "Sk" "So" "Co"
                                     "Nd" "Mc" "Me")))
                  (forward-char))
                (point))))))

  (define-lex-analyzer semantic-lex-scheme-symbol
    "Detect and create Scheme symbol and keyword tokens."
    ;; Since semantic-lex-scheme-identifier-at-point doesn't set up match-data,
    ;; it's a bit of a faff to use the framework in define-lex-analyzer if we
    ;; don't want to have to call the function twice. So we cheat and set an
    ;; "always true" condition, then conditionally push tokens ourselves.
    (let ((bounds (semantic-lex-scheme-identifier-at-point)))
      (when bounds
        (semantic-lex-push-token
         (semantic-lex-token
          (or (semantic-lex-keyword-p
               (buffer-substring (car bounds) (cdr bounds)))
              'symbol)
          (car bounds) (cdr bounds))))))


Could we replace the existing lexical analyzer with this or something
similar? I have a copyright assignment on file for Emacs if that's
relevant.


Thanks,

Rupert

------------------------------------------------------------------------------