icu4c proposal: consistent converter behavior for truncated input
Markus Scherer <[email protected]>
| Newsgroups | gmane.comp.lib.icu.general |
|---|---|
| Organization | IBM |
| Message-ID | <[email protected]> |
Proposal for the ICU4C conversion functions: Make error handling for truncated input work the same as for other errors. ICU4C converters call callback functions when they encounter a byte/UChar sequence that cannot be converted because it is malformed or represents an unmappable code. The last such sequence can also be retrieved with the ucnv_getInvalid[U]Chars() functions. Problem 1: Truncation is detected at the end of a stream when flush=TRUE. This also resets the converter (in this direction), which means that there is not supposed to be any state for the past stream. Therefore, there are no invalid[U]Chars for truncated sequences. Problem 2: The ICU4C converters also do not call callback functions for truncated sequences. They only set an error code. There is no indication how many bytes at the end of the stream form the truncated sequence. The proposal is to treat truncated input the same as unmappable and otherwise illegal input: - call callback functions for end-of-stream truncated sequences - put truncated sequences into the invalid[U]Chars - do not reset the converter if any error is set This will simplify the use of the conversion API - and make useful error handling for truncated sequences possible in the first place. In fact, the envisioned code change includes refactoring and simplifying the converter implementation as well, which will result in significantly less code to maintain. Note that truncated input sequences at the end of a stream occur rarely. Most of the time, the last character will be a Line Feed, or '>' in HTML/XML, or similar. For all the gory details please see http://www.jtcsv.com/cgibin/icu-bugs?findid=2449 Sincerely, markus