Re: [GSoC] Proposal: Modular LilyPond Parser (C++)

"C. Stroppel" <[email protected]> writes:

> Please excuse my delayed response; however, the stronger the arguments
> I am addressing, the more time I require to reflect on them. First of
> all, I would like to express my gratitude for the development of
> LilyPond, which I have enjoyed using for many years.
>
>   > parser.yy and lexer.ll are machine readable descriptions of the
>     syntax, so IMO, they are superior to EBNF or a YAML based grammar.
>
> I must concede that lexer.ll and parser.yy are machine-readable to the
> extent that I have since been able to write a experimental program to
> extract some relevant data. Perhaps my concern was more about
> human-readability—one of the primary reasons I originally chose
> LilyPond. I feel this is partly being lost the more Scheme is
> utilized.

You are correct in some regard here: with the post-2.14 changes to the
language, in particular with regard to what music functions can and
cannot do, we reached a state where including a version of the parser
syntax in the notation manual appendices became pointless because it
stopped conveying significant information.

Part of the reason is that a lot of functionality was moved to music
functions which are not represented individually in the parser (commands
like \time, \times, \key, \octave, \once, \partial, \relative, \skip,
\transpose were all part of the syntax in 2.14 and no longer are).

That seriously empowers the user, but of course is somewhat of a problem
for automated parsing like used in syntax highlighters.

It would make sense to pull out all the music function definitions
dynamically and convert them into hardwired syntax for such purposes.
That would not reflect the dynamic nature of what LilyPond _can_ do but
be a useful starting point: few users redefine built-ins.

This is similar to how syntax-highlighting of LaTeX tends to work in
editors: the dynamic nature of TeX renders LaTeX syntax open-ended, but
in practice this is mostly ignored (with a few exceptions like pstricks
and tikz).

> While extracting tokens from lexer.ll and parser.yy, I noticed for the
> first time that many music functions, such as \clef, exist in Scheme
> but are not recognized by the lexer. This raised the question for me
> whether it would be sensible to recognize established music functions
> earlier in the lexer.

In LilyPond's lexer?  No.  In your lexer: maybe, but I would recommend
you don't hardcode such rules but let them be generated from the current
music function signatures of the particular LilyPond executable in use.

> Three arguments support doing it this way in my lexer:
>
>     1. Relieving the parser so that it no longer has to resolve
>     ambiguities.

What are you talking about here?

>     2. Stabilizing the syntax by recognizing more literals in the
>     lexer. - Guaranteed stability of literals and syntax was the
>     second reason LilyPond became attractive to me.
>          Too often, I was unable to open old compositions with new
>          software.

That is not "stabilizing".  Indeed, LilyPond syntax has become a lot
more stable since moving things into music functions since this makes
people think about how to express functionality in terms of what music
functions can parse, making ad-hoc syntax extensions a rare thing.

>     3. I want to be able to use the fast lexer for syntax highlighting
>     without the parser, reserving the parser solely for rendering.

That is just not going to happen.  LilyPond's input syntax is pretty
much the opposite of the very rigidly structured input of Scheme.  Its
close relation to verbal music descriptions is what makes the "like a
programmer" approach of non-WYSIWYG input representations borderline
tolerable to people more interested in music than computers.


-- 
David Kastrup