Re: A Jython unicode object is a sequence of ...

On Mon, Sep 1, 2014 at 9:12 AM, Jeff Allen <[email protected]> wrote:

> Addressing issue 2100 (http://bugs.jython.org/issue2100) is straining my
> understanding of our policy on Unicode, where surrogates are concerned.
>
> Would it be accurate to say that, irrespective of implementation choices:
> 1. A Jython unicode object is a sequence of point codes.
>

Yes

I prefer the more common nomenclature of code points (sometimes rendered
codepoints, which is how I typically write it myself) -
http://www.unicode.org/glossary/#code_point


> 2. The allowable values are in range(0x110000) excluding range(0xd800,
> 0xe000).
>

Yes, these are the valid Unicode code points, due to the representational
needs of UTF-16

3. When a Jython unicode object is converted to a java.lang.String, each
> point code element >0xffff is converted to a surrogate pair as in the
> encoding to UTF-16.
>

Or it is stored that way, as it is currently. However Python only cares
about code points - outside of using the Java (internals) API directly,
there's no way to access the underlying representation from Python code. So
this is a correct characterization of the user API.


> 4. When a java.lang.String is converted to a Jython unicode object, each
> surrogate pair is converted to a single point code, as in the decoding
> of UTF-16.
>

Factoring out representation choices, yes


> 5. When a Java int[] (intended as code points) is converted to a Jython
> unicode object, each surrogate pair is converted to a single point code,
> as in the decoding of UTF-16.
>

This doesn't sound right as written - such code points in a Java int[]
representation correspond one-to-one to Python code points. In encoding in
UTF-16 for java.lang.String, this will require the use of surrogate pairs
to represent non BMP code points. Such as http://codepoints.net/U+1F37A (an
emoji for a mug of beer) - this codepoint has the int value 0x1F37A
or 127866.


> 6. Unpaired surrogates encountered during conversion to a Jython unicode
> object raise a ValueError.
>
> I spent some time reminding myself of the capabilities of CPython
> (below): the bottom line is that exclusion of the surrogate range in
> rule 2 is where we diverge from CPython. We can only remove that by a
> different implementation of PyUnicode (not proposed).


Agreed. For pure Python code, we could use alternative representations
(presumably in addition to java.lang.String), but any Java method that
expects java.lang.String would see a ValueError (or undefined results) in
__java__ conversion when that method was called with the unpaired
surrogates. That would be quite surprising, I would think, especially since
this could happen very far from when the string was first constructed.


> I trace my
> uncertainty around change to PyUnicode to the fact that we are not
> completely enforcing this exclusion in the code, and then not knowing
> whether that's deliberate. Some variation of rule 2 is imaginable, but
> must be a defined variation if we're to be consistent.
>

Let's be consistent - it's a straightforward check for concatenation, for
example.


>
> Incidentally, code that builds an index to facilitate quick access to
> non-BMP strings would have to rely on (or enforce) our rule on surrogates.
>

+1


- Jim

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/

_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev