Re: Invalid byte 2 of 4-byte UTF-8 sequence

On 10.12.2020 16:45, Alan Painter wrote:
> I've loaded your example into Martin's excellent online XSLT and you'll 
> see that it manages the unicode chars without any trouble.

The thing is that it probably, as Mike suspected, wasn't UTF-8 in the 
first place. It probably was recoded when you took it from your mail 
client and pasted it into Martin's tool, therefore you thought it was 
Unicode.

The OP's message headers say:

Content-Type: text/plain; charset="iso-8859-1".

This doesn't prove that Matthew's *input document* was encoded as ISO 
8859-1, too. It demonstrates that clandestine encoding conversions take 
place, maybe by the operating system's clipboard.

It his highly likely though, as Mike suspected, that the original XML 
was Windows CP-1252 or ISO 8859-1, too. It could have been created by 
naively putting tags around a CP-1252 dump of database fields, without 
looking at the actual encoding and without stating this encoding in the 
XML declaration.

The problematic character was probably the 'ó' in 'Bayamón'.

The code point for ó is hex F3 or decimal 243, both in ISO 8859-1 and in 
Unicode.

The byte representation of ó in ISO 8859-1 and CP-1252 is 11110011.

The byte representation of ó in UTF-8 is 11000011:10110011.

In UTF-8, the first byte of 4-byte characters is 11110xxx 
(https://en.wikipedia.org/wiki/UTF-8#cite_ref-30). Note that this is 
exactly the bit pattern that the CP-1252 encoding of ó matches.

After the first byte of a 4-byte character, an XML reader that assumed 
the input to be UTF-8 encoded was expecting a byte 2 of the form 10xxxxxx.

However, it was presented with 01101110, the representation of 'n' both 
in UTF-8 and in ISO 8859-1.

This is illegal for a UTF-8 byte 2, which is always 10xxxxxx.

So someone erroneously labeled CP-1252, ISO 8859-1, ISO 8859-15, ... 
content as UTF-8, or they omitted the encoding pseudo-attribute in the 
XML declaration, making the parser assume the default of UTF-8.

The content creators need to make sure that the (implicitly) declared 
encoding matches what they put in. This can be healed with tools such as 
recode or iconv (there are others, and some editors are able to infer 
the encoding correctly if unambiguous).

You can try to process the input as text like this

iconv -f WINDOWS-1252 -t UTF-8 in.xml > out.xml

and then feed it into Saxon.

Alternatively, replace <?xml version="1.0"?> with <?xml version="1.0" 
encoding="windows-1252"?>

Gerrit



_______________________________________________
saxon-help mailing list archived at http://saxon.markmail.org/
[email protected]
https://lists.sourceforge.net/lists/listinfo/saxon-help