A Jython unicode object is a sequence of ...

Addressing issue 2100 (http://bugs.jython.org/issue2100) is straining my 
understanding of our policy on Unicode, where surrogates are concerned.

Would it be accurate to say that, irrespective of implementation choices:
1. A Jython unicode object is a sequence of point codes.
2. The allowable values are in range(0x110000) excluding range(0xd800, 
0xe000).
3. When a Jython unicode object is converted to a java.lang.String, each 
point code element >0xffff is converted to a surrogate pair as in the 
encoding to UTF-16.
4. When a java.lang.String is converted to a Jython unicode object, each 
surrogate pair is converted to a single point code, as in the decoding 
of UTF-16.
5. When a Java int[] (intended as code points) is converted to a Jython 
unicode object, each surrogate pair is converted to a single point code, 
as in the decoding of UTF-16.
6. Unpaired surrogates encountered during conversion to a Jython unicode 
object raise a ValueError.

I spent some time reminding myself of the capabilities of CPython 
(below): the bottom line is that exclusion of the surrogate range in 
rule 2 is where we diverge from CPython. We can only remove that by a 
different implementation of PyUnicode (not proposed). I trace my 
uncertainty around change to PyUnicode to the fact that we are not 
completely enforcing this exclusion in the code, and then not knowing 
whether that's deliberate. Some variation of rule 2 is imaginable, but 
must be a defined variation if we're to be consistent.

Incidentally, code that builds an index to facilitate quick access to 
non-BMP strings would have to rely on (or enforce) our rule on surrogates.

Jeff

---

Some background observations on CPython:

It is reasonably clear in Python that a unicode object is a sequence of 
code points. Python 3.4:
 >>> u = u"ab\U00010003de"
 >>> len(u)
5
 >>> u
'ab\U00010003de'
 >>> u.encode("UTF-16BE")
b'\x00a\x00b\xd8\x00\xdc\x03\x00d\x00e'

So far, it is possible to ignore the distinction between code point and 
characters. But ...
 >>> v = u"ab\uD800\uDC03de"
 >>> len(v)
6
 >>> v
'ab\ud800\udc03de'
 >>> v.encode("UTF-16BE")
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-16-be' codec can't encode character '\ud800' in 
position 2: surrogates not allowed
 >>> u+v
'ab\U00010003deab\ud800\udc03de'

We see that in CPython 3, code points for surrogates are definitely 
allowed and are different from the code point of character they combine 
to represent. PEP-261 seems to go down that route. 
(http://legacy.python.org/dev/peps/pep-0261/) Nowhere in the 
documentation do we quite shake off the confusion between character and 
code point (even in the error message).

The Unicode standard says: "A process shall not interpret a 
high-surrogate code point or a low-surrogate code point as an abstract 
character." This doesn't quite forbid storing isolated surrogates in a 
unicode object, if you're clear these are code points not characters. 
But they won't have a character interpretation (PEP-383 aside).

In Jython we implement the unicode type using java.lang.String, with a 
good deal of seamless interoperability between the two types. The 
interoperability is highly desirable, yet it seems impossible to have 
that, and exactly the semantics of Python. It won't be possible to 
represent u+v above and distinguish it from u+u or v+v. If we went as 
far as storing code points (int[], not char[]), we would still not know 
whether an incoming java.lang.String was like to be interpreted like u or v.

-- 
Jeff Allen


------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/