Re: YAML2

Ingy dot Net wrote:
> Hi Devin,
> 
> I added your comments to 
> https://github.com/yaml2/YAML2/wiki/Unicode-Strictness
> 
> This topic seems like it would be more poignant if people actually wrote 
> some test cases about what they were talking about. It's so easy to be 
> misunderstood on this type of issue. But with actual test files, it's 
> much less so.

It is impossible to make a stand-alone YAML test case, as it is 
currently impossible to write invalid UTF-8 byte sequences into a scalar.

An example of a program is to, on Linux, do this:

1. Set your locale to a UTF-8 one, as is default on all modern systems

2. Create GoodFile with non-ASCII UTF-8 characters in the filename

3. Using low-level code, create BadFile with invalid UTF-8 in the 
filename (just add a single 0x80 byte to the end of GoodFile's name, it 
is even better if there is valid UTF-8 in the name as well). TCL is a 
good way to do this, or you can write C code.

4. Now imagine a YAML file that has a structure that has a "filename" 
member. Figure out a way to write a YAML description so that both of 
these files can be named and GoodFile uses the *CORRECT* Unicode 
characters in the YAML file (ie do NOT "double encode" as about 100 
people have suggested), and BadFile at least displays the ASCII 
characters the same so that an uneducated user can at least make a good 
guess as to the which file shown in another list is named in the YAML file.

5. In case you think "oh the non-UTF-8 files are ILLEGAL and therefore I 
will pretend that magic makes them suddenly non-existent when I switch 
to UTF-8", lets imagine the YAML source is to this "magic" program and 
it's job is to rename bad filenames to correct UTF-8. You must provide 
this program with a list of the files to fix. As this program does other 
useful things with valid UTF-8 (perhaps it also changes normalization 
forms) the input must also be readable when the files are valid UTF-8.

------------------------------------------------------------------------------
Get your Android app more play: Bring it to the BlackBerry PlayBook 
in minutes. BlackBerry App World™ now supports Android™ Apps 
for the BlackBerry® PlayBook™. Discover just how easy and simple 
it is! http://p.sf.net/sfu/android-dev2dev