Re: Unicode strings and invalid codepoints

On Tue, Mar 10, 2015 at 9:09 PM, Arne Goedeke <[email protected]> wrote:
> I think it could be a compat problem, someone might be using wide
> strings for non unicode data (e.g. as an efficient way to store
> integers for a bitmask). The other issue is that implementing this
> correctly would require checking all chars when hashing the string,
> which would probably make it rather slow for wide strings.

Fair enough. I'm more looking for philosophical arguments; backward
compatibility is, of course, arguing for no change.

> Whats the reasoning for restricting strings in Python? What happens when
> a new language is added to Unicode and the next pike release is still a
> decade away?

Nothing there; unallocated codepoints would be perfectly acceptable,
to ensure forward compatibility. It's only the blocks that are
strictly disallowed (such as surrogates) which would be forbidden.

There's currently no plan to actually make this restriction, just some
broad discussion about concepts. For instance, both Python and Pike
reject an attempt to UTF-8 encode the string "\uDD00"; but the
question was raised, is that string actually itself the problem?
Should the error have been raised earlier? Hence this query.

ChrisA