Re: Encoding Issue for .ly Files on CPDL

On Sat 06 Jun 2026 at 21:40:49 (+0200), Silvain Dupertuis wrote:
> It looks like this site : https://victoria.uma.es, display .ly files
> correctly (that is, Unicode encoding).
> 
> It probably means that vi[c]toria.uma.es is correctly set

Yes, set—and also set correctly.

> Le 06.06.26 à 17:41, David Wright a écrit :
> > On Tue 02 Jun 2026 at 09:10:00 (-0400), Gabriel Ellsworth wrote:
> > > [ … ] As I mentioned in the forum, there is no
> > > problem with diacritics when I view in my browser the excellent work of
> > > Nancho Álvarez:
> > > https://victoria.uma.es/guerrero/ly/Guerrero-Adios_Verde_Ribera.ly.
> > Perhaps they were able to enforce only utf-8 files from the start.

I suppose I should spell this out. My guess is that the website
https://victoria.uma.es//partituras.html, run by a mathematician
and Debian enthusiast, is likely to have all its text files served
in the one character set, Unicode, and be configured correctly.
It obviously helps when there's a single point of control.

> for but not the
> site https://www.cpdl.org/wiki/[in]

OTOH a heterogeneous website like CPDL, where files are uploaded
by a large number of individuals running different OSes in various
languages, might be expected to contain text files that use different
character sets. And indeed, that appears to be the case:

I went to https://www.cpdl.org/wiki/index.php?title=Special:ListFiles
as suggested in https://forums.cpdl.org/phpBB3/viewtopic.php?f=1&t=14145
and then clicked on Last Page, and one or two times on Previous Page.
I chanced on "‖ 20:25, 9 October 2005 | Ledoulxbaisir.ly (file) | 9 KB |
Stenand | updated version of the ly file of an edition by the same editor | 1 ‖"
and displayed the file with a Left Click.

The file, https://www.cpdl.org/wiki/images/9/9f/Ledoulxbaisir.ly ,
displays perfectly, eg, subtitle = "Chanson á quatre de Thielman Susato;",
and the File Info is attached (as it's effectively an image). The menu
item I mentioned before:
  > >    View > Repair Text Encoding        as easy as <Alt> V C
is appropriately greyed out.

> Actually, these settings are at the level of the Apache server and/or
> on an .htaccess file at the root of the website, so that when CPDL
> displays files stored on victoria.uma.es website, they will be
> displayed according to the settings of victoria.uma.es, not those of
> cdpl...

As CPDL serves files with different character sets, it wouldn't be
appropriate to set charset=utf-8 "at the root of the website", ie
for all files on CPDL. (BTW, CPDL doesn't serve/display files stored
on the victoria.uma.es website: the victoria.uma.es server does.)

So I think we have to congratulate CPDL on the remarkable job it does,
but lower our expectations a little. Browse files by all means, with
all their imperfections, but use downloaded copies for reliability
(don't cut&paste off the screen¹).

And please, don't let's add any BOMs.

¹ Ironically, cut&paste will probably work when the browser gets the
  character set correct, like with Ledoulxbaisir.ly, but I still
  prefer to download files and use:
    $ recode -v -s ISO-8859-1..UTF-8 Ledoulxbaisir.ly
    Request: ISO-8859-1..ISO-10646-UCS-4..UTF-8
    Recoding Ledoulxbaisir.ly... done
    $ 

> Le 06.06.26 à 17:41, David Wright a écrit :
> > On Tue 02 Jun 2026 at 09:10:00 (-0400), Gabriel Ellsworth wrote:
> > > Recently I have noticed an issue with how .ly files display in my browser
> > > when I view the LilyPond work that other editors kindly share on CPDL.
> > > 
> > > I have documented the issue, with links to several examples, in the CPDL
> > > Forums:
> > > 
> > > Unicode encoding issue for LilyPond engraving files (.ly)
> > > <https://forums.cpdl.org/phpBB3/viewtopic.php?f=1&t=14145>
> > Like you, I see LP files displayed on the screen as Windows-1252.
> > (Normally, I SaveToFile, which works correctly (utf-8), and
> > in the past would download by pasting the link into the
> > commandline, nowadays scuppered by the bots/permissions/etc.)
> > 
> > > The forum moderator wrote:
> > > 
> > > > processors like LilyPond and Frescobaldi show diacritics and … browsers
> > > > don’t show them.
> > I think the most significant line in that conversation might be:
> >    "Just (left)clicking on the link is NOT ever correct."
> > which sounds like a typical computing rationalisation of:
> >    "It's never worked for me so it must be the wrong thing to do."
> > 
> > > But this is incorrect, IIUC. As I mentioned in the forum, there is no
> > > problem with diacritics when I view in my browser the excellent work of
> > > Nancho Álvarez:
> > > https://victoria.uma.es/guerrero/ly/Guerrero-Adios_Verde_Ribera.ly.
> > Perhaps they were able to enforce only utf-8 files from the start.
> > 
> > I use FireFox (preferred) and Chromium on Debian linux. In both,
> > saving from the screen display places the correct contents into
> > the file. (Cut and Paste will copy the incorrect characters.)
> > 
> > FireFox does appear to know what the correct encoding is, as it
> > provides an easy way to correct the screen display:
> >    View > Repair Text Encoding        as easy as <Alt> V C
> > (I've read that this option is greyed out if FF doesn't know what
> > the correct encoding is. I'm also told that you can make this into
> > a button on the toolbar, but I prefer using the keyboard.)
> > 
> > I don't know the equivalent command in Chromium, which I only use
> > for financial sites that refuse FF.)
> > 
> > > I believe that CPDL should be able to display files in UTF-8 within
> > > browsers. But I know almost nothing about either Unicode or websites.
> > Neither do most people. AIUI, browsers could easily fix this, and some
> > used to automatically, but they feel they have to pander to the small
> > population of users who still use computers with what they call the
> > "ANSI" standard, whereas the vast majority of web pages are now served
> > up as UTF-8. There's also a (hopefully) diminishing population of web
> > servers with legacy iso-8859 pages, but charset=utf-8 set site-wide.¹
> > 
> > > My questions for this list:
> > > 
> > > 
> > >     1. By any chance, can anyone figure out *why* CPDL currently can’t
> > >     display diacritics correctly?
> > Because browsers take it upon themselves to guess, and they guess
> > wrongly. (Posh name: heuristics.)
> >    https://en.wikipedia.org/wiki/Charset_detection
> > 
> > >     2. What would be required to fix/improve CPDL?
> > Like others here, I assume that adding charset=utf-8 would fix it.
> > (We can observe that it gets text/plain correct.) The question then
> > becomes: are there ancient text files that are not encoded in utf-8?²
> > Strictly, the charset should be set file-by-file if that's the case;
> > quite a task.
> > 
> > I've read that one can still set a default charset in FF, but also
> > that it only applies to local files, opened asfile:///… . Not
> > something I've tried, as my own systems are utf-8 throughout.
> > 
> > ¹ It might be worth pointing out that our troubles are small: it's
> >    easy to read much of a utf-8 page wrongly encoded as iso-8859 and
> >    see what the problem is. Even the inverse is not too troubling.
> >    But a non-latin page served incorrectly as being in utf-8 becomes
> >    gobbledygook. Have a chuckle reading this related problem:
> >      https://en.wikipedia.org/wiki/Bush_hid_the_facts
> > 
> > ² CPDL goes back to 1998: Debian switched its default encoding for
> >    new installations to utf-8 in 2007.

Cheers,
David.