icu4c proposal: consistent converter behavior for truncated input

Markus Scherer <[email protected]>
Newsgroups gmane.comp.lib.icu.general
Organization IBM
Message-ID <[email protected]>
Proposal for the ICU4C conversion functions: Make error handling for truncated input work the same 
as for other errors.

ICU4C converters call callback functions when they encounter a byte/UChar
sequence that cannot be converted because it is malformed or represents an
unmappable code. The last such sequence can also be retrieved with the
ucnv_getInvalid[U]Chars() functions.

Problem 1:
Truncation is detected at the end of a stream when flush=TRUE. This also resets
the converter (in this direction), which means that there is not supposed to be
any state for the past stream. Therefore, there are no invalid[U]Chars for
truncated sequences.

Problem 2:
The ICU4C converters also do not call callback functions for truncated
sequences. They only set an error code.

There is no indication how many bytes at the end of the stream form the truncated sequence.

The proposal is to treat truncated input the same as unmappable and otherwise illegal input:

- call callback functions for end-of-stream truncated sequences
- put truncated sequences into the invalid[U]Chars
- do not reset the converter if any error is set

This will simplify the use of the conversion API - and make useful error handling for truncated 
sequences possible in the first place.

In fact, the envisioned code change includes refactoring and simplifying the converter 
implementation as well, which will result in significantly less code to maintain.

Note that truncated input sequences at the end of a stream occur rarely. Most of the time, the last 
character will be a Line Feed, or '>' in HTML/XML, or similar.

For all the gory details please see http://www.jtcsv.com/cgibin/icu-bugs?findid=2449

Sincerely,
markus
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.