Re: Lua syntax ambiguity

As others have said, this isn't really a bug. The grammar is 
token-based; tokens are properly defined in prose. There is no 
ambiguity here, the language is described by the formal grammar plus a 
precise description of the tokenization. The only minor issue is that 
the "lexical conventions" section in the reference manual neglects to 
mention that the tokenizer is greedy. I would consider this no big 
problem; virtually every tokenizer is, not every last detail needs to 
be written down.

This does not call for an ugly solution however. Quite the opposite, 
just mirror the parsing. Parsing is text -> tokens -> AST. "Unparsing" 
should be AST -> tokens -> text. First convert an AST to a stream of 
tokens, then convert that stream of tokens to text, inserting 
separating suppressed tokens as necessary. If you're lazy, an 
emit(ttype, content) function which decides based on the last written 
token.

By the way, a hypothetical naive approach of directly feeding the 
grammar into some tool is also doomed to fail on many more token 
concatenations than "[[[" (this just happens to be an ambiguous-looking 
one). "if1thenend" is a single token, made up of the concatenation of 
four tokens. A fuzzer which simply applies the grammar rules may 
produce this along with many more abominations like it.
Conversely, a parser directly based on the grammar will be way too 
permissive (this kind of thing happened to me some years ago when I 
tried to translate the grammar to a PEG for LPEG); a naive grammar will 
be ambiguous. Should "localx = 42" be "local x = 42" or "_G.localx = 
42"?

Aside: I am not even sure the situation is salvageable in a CFG at all. 
In greedy tokenization, long strings are closed by the *first* matching 
delimiter. Is there a way to enforce this with a CFG? I don't think so. 
You can of course express valid long strings as

inner ::= '[' {.} ']'
padded ::= inner | '=' padded '='
long ::= '[' padded ']'

where . stands for any character. But I see no way to ensure that {.} 
must not contain a string of equal signs of length matching the 
delimiters, for all infinitely many possible numbers of delimiters. The 
following "program" would be valid per the naive CFG, but is obviously 
invalid when tokenized and subsequently parsed:

--[[ blah blah blah ]]
syntax error!
blah blah blah ]]

- Lars

On Sun, Aug 16 2026 at 21:09:21 +02:00:00, 'Martin Eden' via lua-l 
<[email protected]> wrote:
> On 2026-08-16 19:58, Andrey Dobrovolsky wrote:
> > Hi Martin,
> 
> Hi Andrew,
> 
> > Simple modification of the rule
> >
> >  field ::= "[ " exp " ]" ‘=’ exp
> >
> > probably is the minimal change, however I don't know whether string
> > literals of the length greater than 1 are allowed in EBNF.
> 
> Yeah string literals are allowed in BNF but this modification will
> make code strings like "t={[a]=true}" invalid. And using space
> in literal considered bad manners there.
> 
> > I think it wouldn't be complicated for code generator to insert 
> extra
> > space before and after an index value:
> >
> > t={[ [[a]] ]=true}
> >
> > and it would be enough, for correct parsing.
> 
> Actually lexer needs any separator (one or more of any of (space, 
> newline, comment))
> between "[" from index opening and "[" from string opening.
> 
> But it can't be inserted nicely in code generation. Because proper 
> code just
> writes to output stream and can't (and shouldn't) read existing 
> output stream.
> 
> 
> So we have something like
> 
>   // serialize key and value
>   // ...
>   emit_opening_index()
>   serialize(KeyNode)
>   // ...
> 
> And that flag ("bro if you gonna open with "[" then plz insert any
> whitespace you like before it") is ugly thing to embed in code.
> 
> I've handled it by temporarily monkey-patching output stream method.
> Just another similar-ugliness solution
> 
> <https://github.com/martin-eden/lua_table_serializer/blob/06e5bedb6a248a950158b7fb4062a8d6f61957d6/src/workshop/concepts/codec_lua_graph/compile/Initializer.lua#L138-L143>
> 
> 
> In theory if we aim to comply with "our language is described by 
> formal grammar"
> we can state that any long quote MUST have at least one "=".
> 
> So "[[abc]]" becomes invalid, minimal valid will be "[=[abc]=]".
> This will resolve this syntax clash.
> 
> > Thanks for an interesting corner case,
> > -- Andrew
> 
> I'm glad you liked it.
> 
> -- Martin
> 
> --
> You received this message because you are subscribed to the Google 
> Groups "lua-l" group.
> To unsubscribe from this group and stop receiving emails from it, 
> send an email to [email protected] 
> <mailto:[email protected]>.
> To view this discussion visit 
> <https://groups.google.com/d/msgid/lua-l/eea51f32-acbe-49e2-b072-496c3cfb42e4%40disroot.org>.

-- 
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion visit https://groups.google.com/d/msgid/lua-l/082WJT.4XG7BC99I0Y4%40gmx.de.