Re: U+FFFF

On 07/31/2013 03:28 AM, Richard A. O'Keefe wrote:

> That's not actually an answer.  The question is how did *THIS*
> U+FFFF get there in *THIS* case and what *SHOULD* it have been?

Nope. As the OP says, it comes from an external tool. That is not in our
control. It is illegal Unicode, so if the input was supposed to be
Unicode, we can raise the flag or just pass it over to the next
processing stage.

>> We need an executive summary :-) As far as I'm concerned, SWI-Prolog
>> basically reads, stores and writes Unicode and should be as little as
>> possible involved in understanding how to deal with Unicode characters.
>> Of course, in some places this cannot be avoided (e.g., read/1, case
>> conversion, etc).
>
> This is certainly a defensible and useful position.

Glad to hear that :-)

>>  - Change the internal representation to UCS-32 on all
>>    platforms.  Then we need conversions (and may have
>>    errors) in the C API functions that talk wchar_t*
>>    on Windows.
>
> This is definitely the way to go.  As far as I know, it is
> still legal for wchar_t to be implemented as 1 byte.  Doing this in
> my own C code meant that a world of pain just disappeared.
>
>> I'm tempted to choose the second solution.
>
> That's the painful hard work bug-fountain solution.
> Cut the Gordian knot.  Use something you _control_.

Thanks for this comment.  A bit of experience is always
welcome before one takes a decision ...

>> And that is not the only case.  I've been (on the side)
>> involved in the W3C standards for SPARQL and the various
>> RDF languages and I have not seen any restrictions for
>> \uFFFF being proposed.  I want SWI-Prolog to be able to
>> process such documents.
>
> Well, does anyone on those committees actually understand
> Unicode?  Like have their own copy of the book with the
> edges brown from use?  Is there any way to get someone from
> the actual Unicode group seconded to those committees?

The discussions are complicated enough without.  I think that
the overall position if the RDF/SPARQL groups is that they just
copy these things from other standards, notably from the XML
world.

> In the mean time, you have to interoperate with what you
> have to interoperate with.
>
> The question remains for someone trying to *interpret* this
> stuff, "what on earth was THIS occurrence of U+FFFF supposed
> to mean? how did the originator INTEND it to be processed?"

I guess there is no meaningful answer to that.  Quite likely
someone converted -1 (EOF) to 16-bit wchar_t ...  Maybe the
intend was to have some special marker in the string.  It
might mean something to a specific processing system and this
system might hope that external tools will just pass it around.
I think we learned there is no guarantee that will happen ...

	Cheers --- Jan