Re: Unicode strings and invalid codepoints

I think it could be a compat problem, someone might be using wide
strings for non unicode data (e.g. as an efficient way to store
integers for a bitmask). The other issue is that implementing this
correctly would require checking all chars when hashing the string,
which would probably make it rather slow for wide strings.

Whats the reasoning for restricting strings in Python? What happens when
a new language is added to Unicode and the next pike release is still a
decade away?

arne

On Mon, 9 Mar 2015, Chris Angelico wrote:

> There's a bit of a discussion happening on python-list about whether
> or not it should be legal to have codepoints like U+D800 in Unicode
> strings. Currently, both Python and Pike permit them, but reject them
> if you try to, for example, convert to UTF-8. But a suggestion has
> been made that the mere presence of \uD800 in a string literal should
> be a syntax error, and I'm wondering: Has anyone considered and
> rejected this, or is it simply something that nobody's thought to
> disallow?
>
> Are there situations in which it's necessary to be able to store these
> kinds of noncharacters in a string?
>
> ChrisA
>
>