Re: U+FFFF

On 30/07/2013, at 7:29 PM, Jan Wielemaker wrote:
> 
> I don't really recall this thread to reach a conclusion.

Two things were clear:
 (a) The Unicode standard is QUITE explicit about this.
 (b) There is no consensus yet about what WE should do about it.
> 
>> for a program to generate such a thing to begin with.
>> 
>> So how did that U+FFFF get there?
> 
> As you've seen yourself, there are plenty of languages that to not care
> whether the code is a valid Unicode codepoint or not.

That's not actually an answer.  The question is how did *THIS*
U+FFFF get there in *THIS* case and what *SHOULD* it have been?
>> 
> 
> We need an executive summary :-) As far as I'm converned, SWI-Prolog
> basically reads, stores and writes Unicode and should be as little as
> possible involved in understanding how to deal with Unicode characters.
> Of course, in some places this cannot be avoided (e.g., read/1, case
> conversion, etc).

This is certainly a defensible and useful position.
> 
> 
> The only problem I have at the moment is that internal wide character
> handling uses the system's wchar_t*,

When I started processing Unicode, I had to deal with systems in
which wchar_t could be 4 bytes, 2 bytes, or 1 byte.  Some of them
were the same system.  The lesson I drew from this was *NEVER* to
use wchar_t except at the very last step of interfacing with
someone else's code that wanted it.
> 
>  - Change the internal representation to UCS-32 on all
>    platforms.  Then we need conversions (and may have
>    errors) in the C API functions that talk wchar_t*
>    on Windows.

This is definitely the way to go.  As far as I know, it is
still legal for wchar_t to be implemented as 1 byte.  Doing this in
my own C code meant that a world of pain just disappeared.

> I'm tempted to choose the second solution.

That's the painful hard work bug-fountain solution.
Cut the Gordian knot.  Use something you _control_.

> 
> And that is not the only case.  I've been (on the side)
> involved in the W3C standards for SPARQL and the various
> RDF languages and I have not seen any restrictions for
> \uFFFF being proposed.  I want SWI-Prolog to be able to
> process such documents.

Well, does anyone on those committees actually understand
Unicode?  Like have their own copy of the book with the
edges brown from use?  Is there any way to get someone from
the actual Unicode group seconded to those committees?

In the mean time, you have to interoperate with what you
have to interoperate with.

The question remains for someone trying to *interpret* this
stuff, "what on earth was THIS occurrence of U+FFFF supposed
to mean? how did the originator INTEND it to be processed?"