Re: [GSoC] Proposal: Modular LilyPond Parser (C++)

Thanks for rhe replies. My English is too limited for me to respond quickly.

@Dan's question was: > It's not clear to me what "this way" is or in 
what sense the syntax is currently unstable;

By "this way" I meant the idea of recognizing more of the established 
musik funktions as fixed literals alreadyin the lexer, instead of 
identifying them later through dynamic Scheme lookup. In this sense I 
also meant "stabilizing the syntax". I did not mean that the Lilypond 
syntax itself is unstable, but rather that the vocabulary of the of the 
language is currently created in a largely dynamic way: many music 
functions are defined only during Scheme initialization.

@Han-Wen Nienhuys > recognizing more literals means you are effectively 
changing the syntax. This is the exact opposite of stability.

Thanks for the clarification. I originally assumed that commands like 
\clef would be overridable because they are defined in Scheme. After 
trying it out, I realized that this assumption was wron - the 
registration seems zo happen only once during startup, and later 
definitions are ignored. So from the musician's perspective, \clef is 
not dynamic or user-modifiable.

Because of that, I would assume that recognizing fixed comands like 
\clef earlier in the lexer would not change anything for musicians or 
for th .ly syntax. The input language wouldstay exactly the same; the 
only difference  would be that the parser no longer needs to perform a 
runtime lookup for a literal whose meaning is already fixed at startup.

I might still misunderstand parts of LilyPond's pipeline, so please take 
this with some caution - I am still trying to understand how all the 
pieces fit together. My intention is deinitely not to interfere with 
LilyPond's architecture, especially since I only understand a fraction 
of it so far.


My interest in the source code mainly comes from a practical need: for 
my editor I need reliable syntax highlightin and live rendering. As far 
as I can tell, LilyPond is currently not modular enough for that, so I 
am more or less forced to write my own lexer, parser, and renderer. To 
keep things in sync, I have to study LilyPond's internals, even if I 
don't fully grasp everything yet.

It would certainly be easier for me to stay synchronized in the long run 
if my pipeline resembled LilyPond's more close . I habe considered 
adapting my architecture further, but there are reasons why this is 
difficult,wich I would like to explain.

Lexer: Cuts an classifies tokens.
- A. Fixed tokens via a compact trie (fits into L1 cache).
      ->provides token type, position in the original string, leading 
whitespace, and expected arguments.
- B. All other tokens handled by a seperate mechanism (numbers, strings, 
identifiers, comments, etc.).
- C. A lexer mode determines which token types are allowed in the 
current context.
- D. An argument stack is used for keywords that expect a fixed number 
of arguments.
- Goal: the lexer can be used independently for syntax highlighting, 
without requiring the parser.

Parser:
- Builds a C++ AST from the tokens.
- Builds the rendering cache in parallel for fast editor updates.
- Deterministic, since the lexer already resolves ambiguities.

Additional notes:
- Scheme blocks are stored as raw text inside a dedicated AST node.
- Direct assighments to Scheme variables are treated like fixed tokens 
in the trie.
- Embedding a Scheme interpreter would be desirable, but I do not know 
how to integrate one, and I suspect it would make the parser too slow 
for live rendering.

Best regards,
Christian