Pack and Unpack are Broken for > 0x7FFF_FFFF (in 5.10.0)

[email protected] (Chris Hall) Sun, 16 Mar 2008 15:23:25 +0000
Newsgroups	perl.unicode
Message-ID	<[email protected]>
More confusion about the valid range of characters in Perl.

Both v5.8.8 and v5.10.0 Perl will pack('U', $v) for values of $v which
are > 0x7FFF_FFFF.  The result is the (non-standard) Perl utf8 encoding
for such characters.

v5.8.8 Perl will unpack a string containing the non-standard encoding.

v5.10.0 Perl will not.

Consider:

  use warnings ;

  sub sp {
    my ($v) = @_ ;

    my $p = pack('U', $v) ;
    my @t = unpack('C*', $p) ;

    printf '\x%04X_%04X: ', ($v >> 16), $v & 0xFFFF ;
    print map sprintf('\x%02X', $_), @t ;
    print "\n" ;
  } ;

  sp(0x7FFF_FFFD) ;
  sp(0x8000_0000) ;
  sp(0xFFFF_FFFD) ;

v5.8.8 result:

  \x7FFF_FFFD: \xFD\xBF\xBF\xBF\xBF\xBD
  \x8000_0000: \xFE\x82\x80\x80\x80\x80\x80
  \xFFFF_FFFD: \xFE\x83\xBF\xBF\xBF\xBF\xBD

v5.10.0 result:

  \x7FFF_FFFD: \x7FFFFFFD
  Malformed UTF-8 character (byte 0xfe) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0x82, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0x80, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0x80, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0x80, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0x80, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0x80, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  \x8000_0000: \x00\x00\x00\x00\x00\x00\x00
  Malformed UTF-8 character (byte 0xfe) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0x83, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0xbf, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0xbf, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0xbf, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0xbf, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  Malformed UTF-8 character (unexpected continuation byte 0xbd, with no
    preceding start byte) in unpack at tpbug.pl line 7.
  \xFFFF_FFFD: \x00\x00\x00\x00\x00\x00\x00

And, FWIW, in 64-bit v5.8.8, pack('U', $v) appears to mask the $v value
to unsigned 32-bits before attempting to pack !

-- 
Chris Hall               highwayman.com            +44 7970 277 383
signature.asc (application/pgp-signature, 470 B)