Re: Encoding Issue for .ly Files on CPDL

> On 7 Jun 2026, at 23:09, David Wright <[email protected]> wrote:
> 
> On Sun 07 Jun 2026 at 18:50:27 (+0200), Hans Aikema wrote:
>>> Op 7 jun 2026 om 16:50 heeft David Wright het volgende geschreven:
>>> On Sat 06 Jun 2026 at 21:40:49 (+0200), Silvain Dupertuis wrote:
>>>> It looks like this site : https://victoria.uma.es, display .ly files
>>>> correctly (that is, Unicode encoding).
>>>> 
>>>> It probably means that vi[c]toria.uma.es is correctly set
>>> 
>>> Yes, set—and also set correctly.
>>> .
>>> […]
>>>> Actually, these settings are at the level of the Apache server and/or
>>>> on an .htaccess file at the root of the website, so that when CPDL
>>>> displays files stored on victoria.uma.es website, they will be
>>>> displayed according to the settings of victoria.uma.es, not those of
>>>> cdpl...
>>> 
>>> As CPDL serves files with different character sets, it wouldn't be
>>> appropriate to set charset=utf-8 "at the root of the website", ie
>>> for all files on CPDL. (BTW, CPDL doesn't serve/display files stored
>>> on the victoria.uma.es website: the victoria.uma.es server does.)
>>> 
>> 
>> It is perfectly possible to configure charset=utf-8 for lilypond files only, no need to set it for all (text) files, it can easily be restricted to .ly files only.
> 
> Why would you do that when CPDL can download
>  https://www.cpdl.org/wiki/images/e/eb/Alma-2026-02.ly   UTF-8
>  https://www.cpdl.org/wiki/images/9/9f/Ledoulxbaisir.ly  Windows-1252
> files?
> 

It should’ve been perfectly safe… up to Lilypond 2.4 special characters were supposed to be added in Latex-mode, as of 2.6 they were supposed to be encoded in UTF-8 as per
https://lilypond.org/doc/v2.8/Documentation/user/lilypond.html

So the win1252-encoded file is an anomoly I would say.

> As you already wrote, "if they can be in any encoding there is
> essentially nothing that CPDL can do as they also wouldn't know
> whether the author encoded the text with utf-8, iso-8859-1,
> iso-8859-15 or even other encodings common in non-latin-alphabet
> regions".

I wrote that before spotting in the documentation that input with special characters is supposed to be done in UTF-8

> Well, CPDL can leave it to you and your browser to guess. I think
> that's better than announcing charset=utf-8 with a file that isn't.

The browser will not do anything, it will typically render USASCII as per the RFC still the characterset it should assume when MIME-type and/or charset does not state otherwise


>> Lilypond will not render properly when using special characters unless you code your file in utf-8. So lilypond files are either 'usascii characters only' in which case a utf-8 encoding is safe to interpret it or they contain special characters in which case they will be encoded as utf-8 as that is the only encoding supported by lilypond.
> 
> Making the assumption "they will be encoded as utf-8" is unsafe.

Should’ve been safe, as the requirement for Lilypond since ages has been to encode in UTF-8 and before that in plain USASCII, so having wrong display on wrong files (not encoded in line with lilypond requirements II would consider perfectly fine…

Convert-ly-ing the latin-1encoded file (to update the lilypond code itself to modern syntax) and then re-converting it back to latin-1 yields

Processing `/Users/aikebah/Ledoulxbaisir.ly'
Parsing...
/Users/aikebah/Ledoulxbaisir.ly:8:30: warning: non-UTF-8 input
	composer = "Thomas Cr
                             �quillon " 
ERROR: In procedure ly:parse-file:
Throw to key `decoding-error' with args `("scm_from_utf8_stringn" "input locale conversion error" 2 #vu8(84 104 111 109 97 115 32 67 114 237 113 117 105 108 108 111 110 32))'.
Result: 1