Re: U+FFFF

On 07/29/2013 11:46 PM, Richard A. O'Keefe wrote:
>
> On 30/07/2013, at 12:58 AM, 申远 wrote:
>
>> I have a U+FFFF in my .pl file,which is coded in UTF-8.
>
> It is definitely ungood for a program to silently accept
> non-character codes (we had a long thread about that, in
> I think the standard mailing list).  But it is far worse

I don't really recall this thread to reach a conclusion. I think the
only thing we agreed on is that atom_codes/2 has real codepoints in its
right argument and never surrogates.

> for a program to generate such a thing to begin with.
>
> So how did that U+FFFF get there?

As you've seen yourself, there are plenty of languages that to not care
whether the code is a valid Unicode codepoint or not.

>> There is no error when I consulted it in swi-prolog,
>> where there is a error when I consulted it in tu-prolog.
>
> Getting full Unicode-correctness is appallingly hard.
> The Unicode 6.1 book is about 660 pages long -- bigger
> than the phone book in this city! -- and that does NOT
> include any character tables any more.  It is full to
> overflowing with quirky little details.

We need an executive summary :-) As far as I'm converned, SWI-Prolog
basically reads, stores and writes Unicode and should be as little as
possible involved in understanding how to deal with Unicode characters.
Of course, in some places this cannot be avoided (e.g., read/1, case
conversion, etc).

As far as I'm concerned, Unicode is basically a string with members
0..0x10ffff. UTF-16 (Java, Windows and quit a few others) is just one
multi-byte (2 or 4) encoding that may result in conversion issues, just
like any other multibyte locale (except for UTF-8 and UTF-32).

The only problem I have at the moment is that internal wide character
handling uses the system's wchar_t*, which effectively means UTF-16 on
Windows. That breaks the logic described above. I see two solutions:

  - Change the internal representation to UCS-32 on all
    platforms.  Then we need conversions (and may have
    errors) in the C API functions that talk wchar_t*
    on Windows.
  - Keep the UTF-16 representation and fix
    - atom_codes/2 and friends to (de/en)code UTF-16
    - same for UTF-8 I/O
    - I fear more ...
    This also means that on some systems, strings are
    not 0..0x10ffff, but limited by UTF-16 encoding.

I'm tempted to choose the second solution.

> The Java compiler (javac 1.7.0_13, just tested) will
> let you use '\uFFFF' and "\uFFFF" freely, even at the
> highest "lint" setting.

And that is not the only case.  I've been (on the side)
involved in the W3C standards for SPARQL and the various
RDF languages and I have not seen any restrictions for
\uFFFF being proposed.  I want SWI-Prolog to be able to
process such documents.

	Cheers --- Jan
_______________________________________________
SWI-Prolog mailing list
[email protected]
https://lists.iai.uni-bonn.de/mailman/listinfo.cgi/swi-prolog