Re: utf8::valid and \x14_000 - \x1F_0000

[email protected] (Chris Hall) Tue, 11 Mar 2008 21:09:24 +0000
Newsgroups	perl.unicode
Message-ID	<[email protected]>
On Tue, 11 Mar 2008 you wrote
>Chris Hall skribis 2008-03-11 18:48 (+0000):
>> I'm comfortable with the notion that perl characters are unsigned
>> integers that overlap UCS, and happen to be held internally as a
>> superset of UTF-8.
>> I wonder if perl is completely comfortable.

>It isn't. There are some very unfortunate "features".

>> chr(n) throws various runtime warnings where 'n' isn't kosher UCS, and
>> "\x{h...h}" throws the same ones at compile time.
>> (...)I'm not sure I see the point of picking on a few values to warn
>> about.

>I don't see the point, but Perl's warnings are arbitrary in several
>ways. Abigail has a lightning talk about the "interpreted as function"
>warning, that illustrates this.

OK.  In the meantime IMHO chr(n) should be handling utf8 and has no 
business worrying about things which UTF-8 or UCS think aren't 
characters.

Note that chr(n) is whingeing about 0xFFFE, which Encode::en/decode
(UTF-8) are happy with.  Unicode defines 0xFFFE and 0xFFFF as 
non-characters, not just 0xFFFF (which Encode::en/decode do deem 
invalid).

>> In any case, is chr(n) supposed to be utf8 or UTF-8 ?  AFAIKS, it's
>> neither.

>It's supposed to be neither on the outside. Internally, it's utf8.

One can turn off the warnings and then chr(n) will happily take any +ve 
integer and give you the equivalent character -- so the result is utf8, 
but the warnings are some (very) small subset of checking for UTF-8 :-(

I wonder what happens for n >= 2^64.  The encoding runs out at 2^72 !

>>      If chr(-1) doesn't exist, then undef looks like a reasonable
>>      return value -- returning "\x{FFFD}" makes chr(-1)
>>      indistinguishable from chr(0xFFFD) -- where the first is
>>      nonsense and the second is entirely proper.

>0xFFFD is the Unicode equivalent of undef. I think it makse sense in
>this case.

Well...

Unicode says: "REPLACEMENT CHARACTER: used to represent an incoming 
character whose value is unknown or unrepresentable in Unicode".

...so it has plenty to do without being used to represent a value which 
is completely beyond the range for characters, and for which perl has a 
perfectly good convention already.

...besides, if I want to see if chr(n) has worked I have to check that 
(a) the result is not "\xFFFD" and (b) that n is not 0xFFFD.

So we'll have to differ on this :-)

Chris
-- 
Chris Hall               highwayman.com            +44 7970 277 383
signature.asc (application/pgp-signature, 470 B)