Re: Make Encode.pm support the real UTF-8

Bjoern Hoehrmann <[email protected]> writes:

> * Gisle Aas wrote:
> >As you probably know perl's version of UTF-8 is not the real thing.  I
> >thought I would hack up a patch to support the encoding as defined by
> >Unicode.  That involves rejecting illegal chars (like surrogates,
> >"\x{FFFF}" and "\x{FDD0}), chars above 0x10FFFF, overlong sequences
> >and such.
> 
> I would very much like to have this functionality available in some
> standard module. Though, what do you mean here by rejecting exactly?

It would do the same as it currently does for illegal UTF-8-Perl.  It
falls back to what the CHECK argument ask for.

> For example, by default, I would expect
> 
>   decode("UTF-8" => "Bj\xF6rn")
> 
> to return "Bj\x{FFFD}rn" as documented in `perldoc Encode`; would
> this change (i.e., would it croak instead)?

It would be exactly the same.

More interesting is:

   decode("UTF8", "Bj\xEF\xBF\xBFrn")

where "\xEF\xBF\xBF" is not legal UTF-8 because "\x{FFFF}" is not
legal Unicode.  Either the whole sequence "\xEF\xBF\xBF" is replaced
by "\x{FFFD}" or each bad byte is giving us
"Bj\x{FFFD}\x{FFFD}\x{FFFD}rn".  I think the later will be more sane,
especially when you hit on perl 64-bit extension to UTF-8..

> >Before I do this I would like to get some feedback on the interface.
> >My prefered interface would be to make:
> >
> >   encode("UTF-8", $string)
> >
> >imply the official restricted form and then have
> >
> >   encode("UTF-8-Perl", $string)
> >
> >be used as the name for Perl's relaxed and extended version of the
> >encoding.  The encode_utf8($string) function would continue to be the
> >same as encode("UTF-8-Perl", $string).
> 
> I would prefer there was no semantic overloading of "UTF-8" at all,
> I generally expect that anything called UTF-8 refers to UTF-8 as
> defined in the Unicode standard or RFC 3629. I was for example sur-
> prised that Encode::is_utf8(...) considers sequences UTF-8 that are
> not UTF-8 as defined in those specifications (the documentation
> explicitly states "well-formed UTF-8").

This can be fixed by fixing the documentation.  It might be possible
to get a way by making a distinction between 'utf8' and 'UTF-8'.  The
former being the perl variant while we reserve uppercase form with
dash for the real UTF-8.

> Now that we have this problem, introducing more places where one needs
> to carefully check the documentation what is considered UTF-8 does not
> seem like the best option, having decode_utf8() and decode(utf8=>...)
> mean some- thing different is likely going to cause confusion. Maybe
> this could go the other way round, i.e. introduce a new encoding
> "UTF-8-Strict" or something.

This is certainly more backwards compatible, but do we really want
perl applications to exchange illegal UTF-8 by default?

> >This implies that encode("UTF-8", $string) can start failing while
> >previously it could not.
> 
> As above, by default I do not think it should fail but rather use a
> replacement character instead of croaking.

Yes.  By failing I mean; handle the bad bytes as specified by the
CHECK argument.

>                                              The result should be the
> same as (using RFC-3629-UTF-8 to mean the non-Perl UTF-8)
> 
>   encode("RFC-3629-UTF-8" => decode("RFC-3629-UTF-8" => $string))
> 
> where decode("RFC-3629-UTF-8") would always return a RFC-3629-UTF-8
> string with no illegal sequences (and as that should not fail, the
> above should not fail either). I.e.
> 
>   encode("RFC-3629-UTF-8" => $string) eq
>   encode("RFC-3629-UTF-8" => decode("RFC-3629-UTF-8" => $string))
> 
> would always hold true (assuming that decode("RFC-3629-UTF-8") would
> ignore that the UTF-8 flag on $string is already set and decode
> "again").
> 
> >Other suggestions or comments?
> 
> There should be a corresponding is_foo function that checks whether
> a sequence of octets (or a string with the UTF-8 flag set) is actually
> UTF-8 as defined in the relevant specifications, maybe by adding one
> more argument to Encode::is_utf8 like
> 
>   Encode::is_utf8($string, $perl_utf8_check, $real_utf8_check)

Agree.

Regards,
Gisle