Weird 8859-1 charset
Robert Elz <[email protected]>
| Newsgroups | gmane.os.netbsd.devel.userlevel |
|---|---|
| Message-ID | <[email protected]> |
Hello all ... Some of you might know I am working on fixing the printf(3) use of non-ascii decimal point and the grouping characters (see PR bin/60496 which really should be lib/...) The fixes for the bugs are relatively straightforward, and made even easier by FreeBSD having already made them many years ago. However, obviously (I hope) I'm not going to commit any fixes without testing them, and while we have locales which have non-ascii grouping characters, we have none, or none I could find, which have a non-ascii decimal point. For local testing I could easily just change some locale without committing that change, run the test using that locale (one I'd never normally use locally, like perhaps the no_NO locale from the PR) and run enough tests to be confident the fixes are working. But Taylor would have my grouping characters for decimal points if I fail to commit ATF tests to test the changes, so we can make sure this problem never happens again, and that can't be done without installing a locale with a non-ascii decimal point (certainly at least one - more than one would be better, as each gets just one decimal point and grouping character, and testing just one variation of them wouldn't be much of a test). So I plan on creating a few new locales, using the (reserved) x* names for country codes. Given who I am, I'll probably have them exclusively using English, but natuarlly anyone would be free to add some variations using other languages, to improve the test coverage (the decimal point and grouping chars need to work with any character encoding that might be used with the locale, and as these will all be fakes, we can make that to include absolutely everything, if someone can do the work to define them). Unfortunately I know nothing at all about locales, so I have been trying to learn. Our doc (unless I missed something) on this isn't great, I haven't (so far) even managed to work out how locales get made (other than message catalogs and the charset files, which aren't the parts I need to change). But I shouldn't really need to know that, just copy existing locales for the files & contents, add the new one(s) to Makefile.locale (a list of all the locale names), and then just allow the build system to do its thing. With that in mind I have been looking at the existing locale definitions so I can work out what I need to create, in order to build a (fake) locale definition which is correct (as a locale) even if containing nonsense as values for a locale (I mean, no'one is likely to really use a CJK ideograph as a decimal point, are they? But that's a good way to test long UTF-8 encodings and non-ascii based charsets.) And now we finally get to the point of this message: While looking, I looked in .../src/share/locale/ctype/charset and for the purposes of this e-mail anyway, the Latin-1 file in particular. In that we see (inter alia): ALPHA 0x40 - 0x56 0x58 - 0x76 0x78 - 0x7f LOWER 0x60 - 0x76 0x78 - 0x7f PUNCT 0x21 - 0x3f 0x57 0x77 UPPER 0x40 - 0x56 0x58 - 0x5f What's with the Ws ??? (0x57 and 0x77) As best I can tell, 0x57 is 'W' and 0x77 is 'w' in Latin-1 (aka 8859-1) despite the original (ancient) Latin not having a 'W' character. If those code points truly are punctuation, used for what? The W's are also issing from the TOUPPER and TOLOWER mappings. Probably now someone will tell me it is all irrelevant, as nothing uses this file (the ctype/en_US.ISO_8859-1.src file has no similar oddities). This possibility is increased given there is no Latin-9 file, which as best I can work out would be the name for an equivalent for ISO 8859-15 (the other possibility, it seems would be Latin-0, which we don't have either). Any idea what is going on here, someone must have a clue? If those files (or at least, that one file) isn't used, why hasn't it just been deleted? And while I am here, two related questions, more just to satisfy my curiosity. First, the Latin-1 file also contains: CHARSET ",A" which a comment suggests is used after an ESC. Used for what? Anything that matters? Who or what assigned the ",A" ? (Other files in that dir have similar definitions, with different values.) And second, I see that along with ctype/en_US.ISO_8859.1.src there are sibling files en_US.ISO_8859-2.src and en_US.ISO_8859-4.src but there is also a file en_US.DIS_8859-15.src What's with the "DIS" ? Surely no-one believes tha 8859-15 is still just a draft standard, do they? Shouldn't that one be changed to en_US.ISO_8859-15.src ? kre