Re: Unicode strings and invalid codepoints

On Fri, Mar 20, 2015 at 6:08 PM, Chris Angelico <[email protected]> wrote:
> On Sat, Mar 21, 2015 at 2:24 AM, Henrik Grubbström <[email protected]> wrote:
>>>> Well, as U+D800 is legal in UTF-16 strings (which BTW is why it is
>>>> reserved), I don't see a reason to prohibit it.
>>>>
>>>>> Are there situations in which it's necessary to be able to store these
>>>>> kinds of noncharacters in a string?
>>>>
>>>>
>>>>
>>>> Yes.
>>>
>>>
>>> UTF-16 is represented with individual bytes, though:

For transport, yes. When actually used (cf NT) it is an array of
16-bit integers.

>>>> string_to_unicode("\U00012345");
>>>
>>> (1) Result: "\330\b\337E"
>>>>
>>>> String.string2hex(string_to_unicode("\U00012345"));
>>>
>>> (2) Result: "d808df45"
>>
>>
>> Well, take UCS-2 then.
>
> Which, I believe, disallows U+D800, bringing us back to the start.

Not quite. U+D800 (and the other surrogates) are intended to use to
encode the full Unicode set with 16-bit integers, which is a perfectly
good use of wide strings.

-- 
Henrik Grubbström                                       [email protected]
Roxen Internet Software AB                              [email protected]