Re: SMIE: general feedback and a half-baked idea

> First, let me say that if SMIE didn’t exist I probably wouldn’t have
> attempted to write a major mode for Roto at all.

🙂

> The documentation issue, first, is that it’s really unclear *what to
> leave out* of a SMIE BNF grammar.  I’m a compiler guy, I can read over
> https://roto.docs.nlnetlabs.nl/en/stable/ and distill it down to an
> approximate *complete* BNF for the language, but the impression I have
> so far is you shouldn’t actually give SMIE a complete BNF, you should
> only tell it about things that are directly relevant to indentation
> and sexp movement.  The trouble is figuring out where to draw the
> line.  What I could have used -- and would be willing to help write --
> is a series of examples, starting from the complete BNF for *all of* a
> simple but non-toy language (Roto itself might serve this purpose, in
> fact), and walking the reader through what to remove, what to fudge,
> and what you actually have to put clever hacks into the lexer to deal
> with.

Side note: have you considered using tree-sitter instead?

I have never tried to do what you describe: I always went the other way,
starting from something pretty minimal and adding things as needed.
But your approach would probably lead to more useful docs, indeed.

Any chance you could keep a blog of your progress in roto-mode, where
you record how you progressively change the grammar in response to
specific problems.

> [Tangent: Speaking of putting clever hacks into the lexer, the feature
> where you can return ""/nil from a forward/backward token function and
> it means “do something sensible based on the syntax table” is actually
> rather annoying to work with, both because it means forward/backward
> token functions cannot be used as motion commands for testing, and
> because it means sometimes you might discover, after having moved
> point some distance, that you need to put it back where it
> was. save-excursion helps, but there’s no way to exit a save-excursion
> *without* putting point back; so you wind up with nasty back-and-forth
> dances like
>
> (defun maybe-forward-merged-token (next merged otherwise)
>   (if-let*
>       ((end-of-next
>         (save-excursion
>           (forward-comment (point-max))
>           (when (re-search-forward next nil t)
>             (point)))))
>       (progn
>         (goto-char end-of-next)
>         merged)
>     otherwise))

[ Not sure how that relates to the "do something sensible" behavior,
  but yes, I have code like that in many of my SMIE modes, tho often
  I don't bother with `save-excursion` and just record the position of
  point at start and use `goto-char` to go back.  ]

> I think it would be better if the “do something sensible” behavior
> were in smie-default-(forward/backward)-token, which would *always*
> move over what it thinks the next token is and return it.

It's worth a try.
Note that `smie-default-(forward/backward)-token` is not something you
need to use, it's provided just as a kind of "example" and I've found it
useful to get started: it makes it possible to get useful behavior (for
further development, not for actual use) with just a basic grammar.
You can definitely write your tokenizer functions so that they “do
something sensible” themselves and just return a dummy token after
doing so.

> Now, the idea.  Rust-style control flow blocks are a bad fit for OPGs
> because in
>
> while i < limit {
>     operation();
>     i = i + 1;
> }
>
> the opening curly brace after `i < limit` is doing double duty as the
> *end* marker for the controlling expression and the *beginning* marker
> for the loop body.

Very much so.

> One would *like* to write a BNF something like
>
>     (stmts (stmt) (stmts ";" stmt))
>     (block "{" stmts "}")
>     (stmt
>       (expr)
>       ("while" expr block)
>       ; etc
>     )
>
> but that’s not allowed because you can’t have two consecutive
> nonterminals in a BNF that’s to be lowered to an OPG table.  I am
> fairly sure that the contorted grammar I had to write instead is a
> significant chunk of why the indentation engine is doing the silly
> things I described in my previous message.  (The situation with
> Roto is actually even worse than it might appear from the BNF in
> the roto-mode.el I sent with the previous message, because { ... }
> in Roto can be notation for an “anonymous record” as well as acting
> like a progn form in Lisp.  I had to omit that part of the real
> grammar entirely.)
>
> It occurred to me that we might be able to lift this limitation of
> OPGs, at least in the cases where it matters for this style of
> language, by leaning harder on parse-partial-sexp.  What if SMIE
> partitioned the buffer into chunks by parenthesis depth, parsed
> each chunk independently, and treated chunks at deeper nesting
> levels than the current one as *terminal* symbols?  I’m not sure
> what that would do to the existing SMIE BNF minilanguage, but it
> would mean that ("while" expr BLOCK) would no longer be invalid.

Worth a try: since it's a terminal symbol, use a grammar like

    ("while" expr "<BLOCK>")

and then make your tokenizer return "<BLOCK>" when it skips over
a {...}.  It should work fine w.r.t parsing.

Once you get to the indentation, you might bump into problems that force
you to single-step the `smie.el` code to understand what's going on.
This is because during indentation SMIE sometimes moves in different
directions and assumes that the tokenizer returns "consistent" streams,
but now the streams may not be consistent any more once you move from
"inside" to "outside" of such blocks.
Note: it's also possible that it'll work fine.

> We’d need a way of marking Algolesque block opener/closer keywords
> as parentheses to be matched by parse-partial-sexp (probably text
> properties applied by the syntax-propertize-function), but I think
> that would probably be a useful thing to have *anyway*.

`syntax-propertize-function` can kind-of do it, but with warts (like
the `b` of `begin` isn't consider as part of a word any more, so `M-f
and `M-b` behave funny).