Re: A Jython unicode object is a sequence of ...

On 01/09/2014 18:24, Jim Baker wrote:
> On Mon, Sep 1, 2014 at 9:12 AM, Jeff Allen <[email protected] 
> <mailto:[email protected]>> wrote:
>
>     Addressing issue 2100 (http://bugs.jython.org/issue2100) is
>     straining my
>     understanding of our policy on Unicode, where surrogates are
>     concerned.
>
>     Would it be accurate to say that, irrespective of implementation
>     choices:
>     1. A Jython unicode object is a sequence of point codes.
>
>
> Yes
>
> I prefer the more common nomenclature of code points (sometimes 
> rendered codepoints, which is how I typically write it myself) - 
> http://www.unicode.org/glossary/#code_point
Me too: sorry for the mental transposition.

>     2. The allowable values are in range(0x110000) excluding range(0xd800,
>     0xe000).
>
>
> Yes, these are the valid Unicode code points, due to the 
> representational needs of UTF-16
>
>     3. When a Jython unicode object is converted to a
>     java.lang.String, each
>     point code element >0xffff is converted to a surrogate pair as in the
>     encoding to UTF-16.
>
>
> Or it is stored that way, as it is currently. However Python only 
> cares about code points - outside of using the Java (internals) API 
> directly, there's no way to access the underlying representation from 
> Python code. So this is a correct characterization of the user API.
Note the direction in question. This would be a rule independent of our 
implementation, relevant to Java clients. In the intended 
implementation, it's trivial, of course.

...
>
>     5. When a Java int[] (intended as code points) is converted to a
>     Jython
>     unicode object, each surrogate pair is converted to a single point
>     code,
>     as in the decoding of UTF-16.
>
>
> This doesn't sound right as written - such code points in a Java int[] 
> representation correspond one-to-one to Python code points. In 
> encoding in UTF-16 for java.lang.String, this will require the use of 
> surrogate pairs to represent non BMP code points. Such as 
> http://codepoints.net/U+1F37A (an emoji for a mug of beer) - this 
> codepoint has the int value 0x1F37A or 127866.
In this context, I would expect a supplementary character to be 
represented as a single code point (int). But 0xD800-0xDFFF are also 
valid code points, so the question arises what we ought to do if they 
turn up. In view of rule 2, we can't preserve them: I suggest we 
interpret them. (It's actually the zero cost choice.)

>     6. Unpaired surrogates encountered during conversion to a Jython
>     unicode
>     object raise a ValueError.
>
>     ...
>
>
> Agreed. For pure Python code, we could use alternative representations 
> (presumably in addition to java.lang.String), but any Java method that 
> expects java.lang.String would see a ValueError (or undefined results) 
> in __java__ conversion when that method was called with the unpaired 
> surrogates. That would be quite surprising, I would think, especially 
> since this could happen very far from when the string was first 
> constructed.
There could be a no ValueError on conversion to a String. There would be 
one on conversion to a PyUnicode, probably in the constructor, if the 
incoming String contains lone surrogates.  That could still be a 
surprise, but the alternative is to allow them somehow in our unicode 
objects (not proposed).

>     I trace my
>     uncertainty around change to PyUnicode to the fact that we are not
>     completely enforcing this exclusion in the code, and then not knowing
>     whether that's deliberate. Some variation of rule 2 is imaginable, but
>     must be a defined variation if we're to be consistent.
>
>
> Let's be consistent - it's a straightforward check for concatenation, 
> for example.
>
>
>     Incidentally, code that builds an index to facilitate quick access to
>     non-BMP strings would have to rely on (or enforce) our rule on
>     surrogates.
>
>
> +1
>
>
> - Jim
My idea for quick access involves an auxiliary index built by scanning 
the String. If done during the constructor, it is efficient to do the 
rule 2 check at the same time. On the other hand, I also had a plan to 
defer building the index until we need it. I'll work it out.

Jeff

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/

_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev