SMIE: wrote a mode for Roto, stuck on indentation
"Zack Weinberg" <[email protected]>
| Newsgroups | gmane.emacs.help |
|---|---|
| Message-ID | <[email protected]> |
For the past while I’ve been trying to write a major mode for the Roto
programming language, which is an embeddable scripting language in the
same niche as Lua, but with Rust-ish syntax and static typing. (See
<https://roto.docs.nlnetlabs.nl/en/stable/>.) It’s the first time
I’ve ever tried to write an entire major mode and I’m struggling with
getting SMIE to do what I want.
I’ve listed the SMIE-related problems I’ve run into below, and I’d
particularly like any advice you can give me on how to fix them, but
any other feedback on things that aren’t quite right about the mode
and/or the Lisp would also be quite welcome. The .el file is
attached.
(All tests with Emacs 30.2 (Debian package), roto-basic-offset nil,
smie-indent-basic 4.)
Problem 1: This fragment should be indented like this:
test foo {
print("abcd");
}
but C-M-\ reindents it like this:
test foo {
print("abcd");
}
(There is an example in the manual that is supposed to be about
exactly this, but it does not work for me. If you look at
roto--smie-rules-function, you’ll see that the
(and (member token '("(" "[" "{")) ...) case is *almost* like that
example, but looking for :after instead of :before; with :before it did
absolutely nothing in my tests, with :after it corrects the
indentation of the “print” lines but *not* the indentation of the
close braces.)
Problem 1a: If you’re not using “cuddled” curly braces, braces should
be vertically aligned with the parent keyword, “Allman” style,
test foo
{
print("abcd");
}
C–M-\ on the whole thing reindents it like this:
test foo
{
print("abcd");
}
Similar things happen with all the constructs that end in a block.
Problem 2: If you have a bunch of things one after another they keep
getting indented more and more, as if they were all one big expression.
test foo {
print("abcd");
}
test bar {
print("efgh");
}
test blurf {
print("ijkl");
}
becomes
test foo {
print("abcd");
}
test bar {
print("efgh");
}
test blurf {
print("ijkl");
}
Problem 3 (probably closely related to problem 2 but not exactly the
same): This
test plugh {
let a = 123;
let bc = 456;
let def = 789;
let ghij = 0;
let klmno = 3.14159;
}
gets reindented like this:
test plugh {
let a = 123;
let bc = 456;
let def = 789;
let ghij = 0;
let klmno = 3.14159;
}
Notice here how each ‘let’ is precisely aligned with the number on
the previous line. It’s treating ‘123; let bc = ...’ as _all_ being
right-hand side of the ‘let a =’. Stefan Monnier told me on Mastodon
that I ought to be able to fix this by adding some entries to the
part of the grammar that’s written as a raw precedence table, so that
SMIE knows ‘=’ binds tighter than ‘;’ (and ‘,’). That makes sense,
but I couldn’t make it work; if you uncomment either of the commented-
out lines marked as “disabled due to precedence conflict,” you’ll get
(some of) these load-time warnings:
⛔ Warning (smie): Conflict: ; </= ;
⛔ Warning (smie): Conflict: , </> ;
⛔ Warning (smie): Conflict: , </= ,
Note the absence of a “total number of warnings” line; these are
coming from smie-merge-prec2s, not smie-bnf->prec2 or
smie-precs->prec2; so that must mean there’s something in the BNF
grammar that is in conflict with having (assoc ";") (assoc ",")
as the first two raw OPG rules, but I don’t see where it is.
Problem 4 isn’t an _indentation_ problem, but it’s a SMIE problem.
Write
test foo {}
in a roto-mode buffer, enable blink-matching-paren, and put the cursor
on any character of the word “test”. That word will be highlighted as
an unmatched opener. This also happens for all the other keywords
that introduce a construct that ends with a brace block.
zw
p.s. Please cc: me on all replies, I'm not subscribed.
roto-mode.el
(text/x-emacs-lisp, 20.2 KB)
;; roto-mode.el --- A major-mode for editing Roto source code -*-lexical-binding: t-*-
;; Copyright (c) 2026 Zack Weinberg et al.
;; This mode is free software. It may be modified and redistributed
;; under the same terms as Roto itself OR under the same terms as
;; Emacs itself, at your option.
;; Last updated 2026-03-02, supports Roto 0.10.0.
;;;###autoload
(autoload 'roto-mode "roto-mode" "Major mode for Roto code." t)
;;;###autoload
(add-to-list 'auto-mode-alist '("\\.roto\\'" . roto-mode))
(require 'rx)
(require 'smie)
(provide 'roto-mode)
(defgroup roto-mode nil
"Support for editing Roto code."
:link '(url-link "https://roto.docs.nlnetlabs.nl/en/stable/")
:group 'languages)
;; TODO:
;; (1) Figure out whether we actually need to use autoload cookies
;; to set the safe-local-variable property for this variable
;; (the elisp manual is ambiguous)
;; (2) Figure out how express "this can be set to a nonnegative integer
;; OR to nil" in both the custom type and the safety predicate.
(defcustom roto-basic-offset nil
"Basic amount of indentation per nesting level of Roto code.
If this is nil, the value of `smie-indent-basic' is used."
:type '(natnum)
:safe 'integerp
:group 'roto-mode
:local t)
;; Currently empty, but present for concreteness.
(defvar-keymap roto-mode-map
:doc "Keymap for roto-mode")
;;
;; Lexical syntax rules
;;
(defvar roto-mode-syntax-table
(let ((st (make-syntax-table)))
;; The default syntax table already has all XID_Start and
;; XID_Continue characters as either "word" or "symbol", and it
;; has () [] {} as matched delimiter characters, " as string
;; open/close, \ as string escape.
;;
;; These are all "symbol" in the default syntax table; we want
;; them to be "punctuation", like in C.
(dolist (ch '(?& ?* ?+ ?- ?/ ?< ?= ?> ?|))
(modify-syntax-entry ch "." st))
;; Roto uses C-like 'x' notation for character constants
(modify-syntax-entry ?' "\"" st)
;; Roto uses #...\n comments
(modify-syntax-entry ?# "<" st)
(modify-syntax-entry ?\n ">" st)
st)
"Syntax table for Roto mode")
;; This regex matches an _intentional generalization_ of IP address
;; literals, possibly with /nn suffix. The generalization is for
;; two reasons: so that incomplete and erroneous address literals are
;; still treated _syntactically_ as address literals, and because the
;; generalized regex is easier to make efficient. Caution: ordering
;; matters in a bunch of places.
(defconst roto--tk-pseudo-ipaddr
(rx-let
((v4c (+ (any "0-9")))
(v6c (+ (any "0-9" "a-f" "A-F")))
(cidr (seq "/" (+ (any "0-9"))))
;; v4c . v4c is indistinguishable from a float literal,
;; so the minimum number of v4c's in a pseudo-ip-literal is three.
(v4addr (seq v4c (>= 2 (+ ".") v4c)))
(v6addr (or (seq v6c (+ (+ ":") v6c))
(seq (+ ":") v6c (* (+ ":") v6c))))
(v6suffix (or (+ (+ ".") v4c)
(+ ":"))))
(rx (seq (or v4addr
(seq v6addr (? v6suffix))
(>= 2 ":"))
(? cidr)))))
;; Not currently used but preserved as documentation: this much messier
;; and slower regex matches only _correct_ IP address literals. It's
;; derived from the BNF grammar in
;; https://datatracker.ietf.org/doc/html/draft-main-ipaddr-text-rep-02
;; and does not recognize legacy inet_aton notation. It could potentially
;; be used to highlight erroneous IP address literals differently from
;; correct ones.
(defconst roto--tk-exact-ipaddr
(rx-let
((d8 (or (any "0-9") ; 0-9
(seq (any "1-9") (any "0-9")) ; 10-99
(seq "1" (any "0-9") (any "0-9")) ; 100-199
(seq "2" (any "0-4") (any "0-9")) ; 200-249
(seq "2" "5" (any "0-5")))) ; 240-255
(h16 (** 1 4 (any "0-9" "a-f" "A-F")))
(v4addr (seq (= 3 (seq d8 ".")) d8))
(ls32 (or (seq h16 ":" h16) v4addr))
(v6addr
(or
(seq (= 6 (seq h16 ":")) ls32)
(seq "::" (= 5 (seq h16 ":")) ls32)
(seq (? h16) "::" (= 4 (seq h16 ":")) ls32)
(seq (? (** 0 1 (seq h16 ":")) h16) "::" (= 3 (seq h16 ":")) ls32)
(seq (? (** 0 2 (seq h16 ":")) h16) "::" (= 2 (seq h16 ":")) ls32)
(seq (? (** 0 3 (seq h16 ":")) h16) "::" h16 ":" ls32)
(seq (? (** 0 4 (seq h16 ":")) h16) "::" ls32)
(seq (? (** 0 5 (seq h16 ":")) h16) "::" h16)
(seq (? (** 0 6 (seq h16 ":")) h16) "::"))))
(rx (seq (or v4addr v6addr) ; the actual address
(? "/" (1+ digit)))))) ; subnet annotation
(defconst roto--tk-constant
(rx symbol-start
(or "true" "false" (seq "AS" (1+ digit)))
symbol-end))
(defconst roto--tk-number
(rx symbol-start
(or
;; FIXME: The Sublime grammar permits '_' as a visual
;; separator in numbers but it looks like the actual
;; parser *doesn't*. Following Sublime for now.
(seq "0x" (1+ (in hex ?_)))
;; This handles both decimal integers and floats.
(seq digit
(0+ (in digit ?_))
;; FIXME: because of ambiguity issues, we
;; recognize a decimal point as such only if it
;; has digits on both sides; this is not
;; consistent with the actual parser
(? "." (1+ (in digit ?_)))
(? (any ?e ?E) (? (any ?+ ?-)) (1+ (in digit ?_)))))
symbol-end))
(defconst roto--tk-type
(rx symbol-start
(or (seq (any ?u ?i) (or "8" "16" "32" "64"))
(seq (any ?f) (or "32" "64"))
"bool" "()" "!"
;; any identifier whose first character is an uppercase letter
;; and whose second character is *not* an uppercase letter
(seq upper
(or lower (syntax symbol))
(0+ (or (syntax word) (syntax symbol)))))
symbol-end))
;; control flow keywords; 'not' is really an operator, but it fits better
;; here than anyone else
(defconst roto--tk-control
(rx symbol-start
(or "accept" "else" "for" "if" "in" "match" "reject" "return" "while" "not")
symbol-end))
;; keywords that introduce a definition; 'type' included for Roto <=0.9
;; compatibility; matches both the keyword and the name (see below)
(defconst roto--tk-defun
(rx symbol-start
(group (or "filter" "filtermap" "fn" "record" "test" "type"))
(+ blank)
;; this should be XID_Start (* XID_Continue) but there doesn't seem
;; to be a good way to do that
(group (or alpha (syntax symbol))
(0+ (or (syntax word) (syntax symbol))))
symbol-end))
(defconst roto--tk-defvar
(rx symbol-start
(group "let")
(+ blank)
;; this should be XID_Start (* XID_Continue) but there doesn't seem
;; to be a good way to do that
(group (or alpha (syntax symbol))
(0+ (or (syntax word) (syntax symbol))))
symbol-end))
;;
;; Syntax highlighting
;;
;; TODO: Highlight function argument names as variable-name-face.
;; This requires fancier parsing than I think is feasible via
;; font-lock-keywords.
(defun roto--syntax-propertize (start end)
"The `syntax-propertize-function' hook for Roto code."
(goto-char start)
(while (re-search-forward roto--tk-pseudo-ipaddr end t)
(put-text-property (match-beginning 0) (+ (match-beginning 0) 1)
'syntax-table (string-to-syntax "|"))
(put-text-property (- (match-end 0) 1) (match-end 0)
'syntax-table (string-to-syntax "|"))))
(defconst roto--font-lock-keywords-lv1
`((,roto--tk-defvar (1 font-lock-keyword-face)
(2 font-lock-variable-name-face))
(,roto--tk-defun (1 font-lock-keyword-face)
(2 font-lock-function-name-face))
;; 'import' is highlighted like roto--defuns but we don't bother
;; highlighting the thing that's imported
("\\_<import\\_>" . font-lock-keyword-face))
"Font Lock keywords for Roto mode, level 1 highlighting")
(defconst roto--font-lock-keywords-lv2
`((,roto--tk-type . font-lock-type-face)
(,roto--tk-control . font-lock-builtin-face)
(,roto--tk-constant . font-lock-constant-face)
,@roto--font-lock-keywords-lv1)
"Font Lock keywords for Roto mode, level 2 highlighting")
(defconst roto--font-lock-keywords-lv3
(if (boundp 'font-lock-number-face)
`((, roto--tk-number 'font-lock-number-face)
,@roto--font-lock-keywords-lv2)
roto--font-lock-keywords-lv2)
"Font Lock keywords for Roto mode, level 3 highlighting.
Identical to level 2 if `font-lock-number-face' isn't defined
(this face is documented in the Elisp manual but does not seem
to exist in my copy of Emacs 30.2).")
(defconst roto-font-lock-defaults
'((roto--font-lock-keywords-lv1
roto--font-lock-keywords-lv1
roto--font-lock-keywords-lv2
roto--font-lock-keywords-lv3))
"Font Lock parameters for Roto mode")
;;
;; Indentation
;;
(defsubst roto--maybe-forward-merged-token (next merged otherwise)
"Subroutine of `roto--forward-token': If the first non-whitespace,
non-comment characters after point match the regex NEXT, advance point
to the end of NEXT and return MERGED. Otherwise, leave point where it
was and return OTHERWISE. NEXT must begin with '\\=', i.e. it must be
anchored to point on the left."
(if-let*
((end-of-next
(save-excursion
(forward-comment (point-max))
(when (re-search-forward next nil t)
(point)))))
(progn
(goto-char end-of-next)
merged)
otherwise))
(defsubst roto--default-forward-token ()
"Subroutine of `roto--forward-token': The Roto token immediately after
point is known not to need special handling. Move point forward over
it, and return it as a string. May return the empty string, in which
case point is immediately before either a string literal or a balanced
delimiter and it should be handled according to the syntax table.
This is almost the same as `smie-default-forward-token'; the differences
are that it does not skip comments (`roto--forward-token' did that
already) and it consumes at most one punctuation character."
(buffer-substring-no-properties
(point)
(progn (if (zerop (skip-syntax-forward "." (+ (point) 1)))
(skip-syntax-forward "w_'"))
(point))))
(defsubst roto--special-forward-token (token)
"Subroutine of `roto--forward-token': The Roto token immediately
*before* point, TOKEN, has been determined to need special handling.
Do whatever is necessary, move point to the appropriate starting
position for the *next* call to `roto--forward-token', then return
the finalized token as a string (this is not necessarily the same as
TOKEN).
Like `roto--default-forward-token', may return the empty string, in
which case point is immediately before either a string literal or a
balanced delimiter and it should be handled according to the syntax
table."
(pcase token
;; When the next token is ")", we need to look farther ahead and
;; find out if the token after it is "->" and, if so, treat
;; them as a single merged token.
(")" (roto--maybe-forward-merged-token "\\=->" ") ->" ")"))
;; Similarly for 'else' and 'if'.
("else" (roto--maybe-forward-merged-token "\\=if\\_>" "else if" "else"))
;; ':' can either be an operator or the beginning of an IP
;; address literal. If it's the beginning of a literal, point
;; (which is immediately after the ':') will have syntax-ppss
;; context 'string, and we need to back up one character and
;; tell SMIE to punt to the syntax tables.
((and ":"
(guard (eq (syntax-ppss-context (syntax-ppss)) 'string)))
(forward-char -1)
"")
;; Otherwise, the only "special handling" TOKEN needs is to be
;; identified as a two-character punctuator.
(other other)))
(defun roto--forward-token ()
"SMIE lexical analysis hook: Move point forward over the next Roto token,
skipping comments and whitespace, and return the text of that token.
May return the empty string, in which case point is immediately before
either a string literal or a balanced delimiter and it should be handled
according to the syntax table."
(forward-comment (point-max))
;; For speed, we do just one forward regex search for any of the
;; tokens that need special handling, advance point past it, and
;; then figure out what to do with the token.
(save-match-data
(if (re-search-forward
(rx point (or ":" ")" "==" ">=" "<=" "!=" "&&" "||" "->"
(seq "else" symbol-end)))
(+ (point) 4) t)
(roto--special-forward-token (match-string-no-properties 0))
(roto--default-forward-token))))
(defsubst roto--maybe-backward-merged-token (prev merged otherwise)
"Subroutine of `roto--forward-token': If the last non-whitespace,
non-comment characters before point match the regex PREV, advance point
to the beginning of PREV and return MERGED. Otherwise, leave point
where it was and return OTHERWISE. NEXT must end with '\\=', i.e. it
must be anchored to point on the right."
(if-let*
((beg-of-prev
(save-excursion
(forward-comment (- (point)))
(when (re-search-backward prev nil t)
(point)))))
(progn
(goto-char beg-of-prev)
merged)
otherwise))
(defsubst roto--default-backward-token ()
"Subroutine of `roto--backward-token': The Roto token immediately after
point is known not to need special handling. Move point backward over
it, and return it as a string. May return the empty string, in which
case point is immediately before either a string literal or a balanced
delimiter and it should be handled according to the syntax table.
This is almost the same as `smie-default-backward-token'; the differences
are that it does not skip comments (`roto--backward-token' did that
already) and it consumes at most one punctuation character."
(buffer-substring-no-properties
(point)
(progn (if (zerop (skip-syntax-backward "." (- (point) 1)))
(skip-syntax-backward "w_'"))
(point))))
(defsubst roto--special-backward-token (token)
"Subroutine of `roto--backward-token': The Roto token immediately
*before* point, TOKEN, has been determined to need special handling.
Do whatever is necessary, move point to the appropriate starting
position for the *next* call to `roto--backward-token', then return
the finalized token as a string (this is not necessarily the same as
TOKEN).
Like `roto--default-backward-token', may return the empty string, in
which case point is immediately before either a string literal or a
balanced delimiter and it should be handled according to the syntax
table."
(pcase token
;; When the next token is "->", we need to look farther backward
;; and find out if the token before it is ")" and, if so, treat
;; them as a single merged token.
("->" (roto--maybe-backward-merged-token ")\\=" ") ->" "->"))
;; Similarly for "if" and "else".
("if" (roto--maybe-backward-merged-token "\\_<else\\=" "else if" "if"))
;; ':' can either be an operator or the end of an IP address
;; literal. If it's the end of a literal, point (which is
;; immediately before the ':') will have syntax-ppss context
;; 'string, and we need to reverse one character and tell SMIE to
;; punt to the syntax tables.
((and ":"
(guard (eq (syntax-ppss-context (syntax-ppss)) 'string)))
(forward-char 1)
"")
;; Otherwise, the only "special handling" TOKEN needs is to be
;; identified as a two-character punctuator.
(other other)))
(defun roto--backward-token ()
"SMIE lexical analysis hook: Move point backward over the previous Roto
token, skipping comments and whitespace, and return the text of that
token. May return the empty string, in which case point is immediately
after either a string literal or a balanced delimiter and it should be
handled according to the syntax table."
(forward-comment (- (point)))
;; For speed, we do just one backward regex search for any of
;; the tokens that need special handling, advance point past it,
;; and then figure out what to do with the token.
(save-match-data
(if (re-search-backward
(rx (or ":" "==" ">=" "<=" "!=" "&&" "||" "->"
(seq symbol-start "if"))
point)
(- (point) 2) t)
(roto--special-backward-token (match-string-no-properties 0))
(roto--default-backward-token))))
(defconst roto--smie-grammar
(smie-prec2->grammar
(smie-merge-prec2s
(smie-bnf->prec2
;; We don't bother making a distinction between file-scope "item"
;; and block-scope "expression" constructs; we lump them both
;; into the "sexp" production. SMIE's default indentation rules
;; should actually work better this way.
'((id)
(path (id) (path "." path))
(type (type "?")
(type "[" sexps "]")
(id))
(typed-id (id ":" type))
(sexps (sexp) (sexp ";" sexps) (sexp "," sexps))
(sexp (id)
(path)
(typed-id)
("fn" id "(" sexps ")" "{" sexps "}")
("fn" id "(" sexps ") ->" type "{" sexps "}")
("filter" id "(" sexps ")" "{" sexps "}")
("filtermap" id "(" sexps ")" "{" sexps "}")
("test" id "{" sexps "}")
("record" id "{" sexps "}")
("type" id "{" sexps "}")
("import" path)
("let" sexp)
("accept" sexp)
("reject" sexp)
("return" sexp)
;; let's see how far we get with no connection between clauses
;; of an if-else chain because I'm genuinely stumped how to
;; connect them
("if" sexp "{" sexps "}")
("else" "{" sexps "}")
("else if" sexp "{" sexps "}")
("while" sexp "{" sexps "}")
("for" id "in" sexp "{" sexps "}")
("match" sexp "{" sexps "}")))
'((nonassoc "{" "}" "(" ")" "->"))
'((assoc ";")
(assoc ",")
(assoc "."))
)
(smie-precs->prec2
'(
;; disabled due to precedence conflicts
;(assoc ";")
;(assoc ",")
(right "=" "->")
(left "&&" "||")
(right "not")
(nonassoc ">" ">=" "<" "<=" "==" "!=")
(left "+" "-")
(left "*" "/"))))))
(defun roto--smie-rules-function (kind token)
"SMIE indentation rule adjuster for Roto mode"
(cond
((and (eq token 'basic)
(eq kind :elem))
roto-basic-offset)
((and (member token '("=" "->" ") ->"))
(eq kind :after))
roto-basic-offset)
((and (member token '("(" "[" "{"))
(eq kind :after)
(smie-rule-hanging-p))
(smie-rule-parent))
((member token '("," ";" "."))
(smie-rule-separator kind))))
;;
;; Imenu
;;
(defconst roto-imenu-patterns
(rx-let
;; this should be XID_Start (* XID_Continue) but there doesn't seem
;; to be a good way to do that
((id (seq (group (or alpha (syntax symbol))
(0+ (or (syntax word) (syntax symbol))))
symbol-end)))
`(("*Functions*" ,(rx symbol-start "fn" (+ blank) id) 1)
("*Filters*"
,(rx symbol-start (or "filter" "filtermap") (+ blank) id) 1)
("*Records*"
,(rx symbol-start (or "record" "type") (+ blank) id) 1)
("*Tests*" ,(rx symbol-start "tests" (+ blank) id) 1)))
"Imenu patterns for Roto mode. All global items are categorized and
added to the menu.")
;;
;; Putting it all together.
;;
(define-derived-mode roto-mode prog-mode "Roto"
"Major mode for editing Roto files
(see <https://roto.docs.nlnetlabs.nl/en/stable/>).
\\{roto-mode-map}"
:group 'roto-mode
(setq-local
comment-style 'indent
comment-start "#"
comment-end ""
comment-end-can-be-escaped nil
comment-use-syntax t
font-lock-defaults roto-font-lock-defaults
imenu-generic-expression roto-imenu-patterns
syntax-propertize-function 'roto--syntax-propertize)
(smie-setup roto--smie-grammar 'roto--smie-rules-function
:forward-token 'roto--forward-token
:backward-token 'roto--backward-token))