icu4c api proposal: fix/change ucnv_getMaxCharSize()
Markus Scherer <[email protected]>
| Newsgroups | gmane.comp.lib.icu.general |
|---|---|
| Organization | IBM |
| Message-ID | <[email protected]> |
ucnv_getMaxCharSize() is currently documented as:
* Returns the maximum length of bytes used by a character. This varies
* between 1 and 4
This is clear and simple for trivial cases like ISO 8859-x and traditional Shift-JIS. However, it is
not at all clear what this function returns, and what to do with it, in more complicated cases, and
the behavior is all over the map for various converter implementations.
Issues:
- max bytes per UChar or per code point?
different for mappings involving supplementaries
like UTF-8
- max bytes in character content, or including
shifting output (SI/SO, escapes, etc.)?
different for stateful converters
like ISO-2022-JP
- max bytes per character or per conversion unit?
different when converting from multiple UChars/code points
like for JIS X 0213
- max bytes per conversion unit, or including
all output that is somehow related to some input?
different when a converter outputs return-to-initial-state bytes
like EBCDIC_STATEFUL,
or initial bytes like the escape to UTF-8 in the generic
ISO-2022 converter
From anecdotal evidence (user complaints), this function is used as part of an estimate to allocate
an output buffer that is guaranteed to be large enough to hold the conversion result for an input
Unicode buffer of a given size.
Proposal:
1.
Add a macro for such a guaranteed-buffer allocation estimate, taking as parameters the length of the
Unicode text (in UChars) and the return value of ucnv_getMaxCharSize():
#define UCNV_GET_MAX_BYTES_FOR_STRING(length, maxCharSize) \
(((length)+10)*(maxCharSize))
The +10 takes care of initial and final bytes that are output without direct association with
specific input.
2.
Modify the implementation of ucnv_getMaxCharSize() so that its return value works with the above
macro. This means:
- max bytes per UChar
- including shifting output
- when converting from multiple Unicode characters:
max bytes per input UChar (rounded up)
- not including initial/final output bytes,
but see comment under 1. about the +10 in the macro
Remaining issues:
- ucnv_getMaxCharSize() may not return what is intuitively expected;
as proposed, it will return 3 not 4 for UTF-8. (See below.)
However, there is no one number that is suitable for both
an abstract unit size and a reasonable buffer size estimate.
- The function will not take into account the size of
callback output. It will therefore provide accurate results if
the stop/skip/substitution callbacks or similar are used,
but not for the escape callback or similar.
Expiration: Tuesday, 2003-nov-04
Examples:
- traditional single-byte charsets: 1
- traditional Shift-JIS: 2
- single-byte charset which maps the ffi-ligature to f+f+i: 3
- EBCDIC_STATEFUL: 3 (worst case: SO+DBCS)
- UTF-8: 3 (4 bytes per supplementary are per 2 UChars,
so worst case is U+0800..U+FFFF)
- ISO-2022-JP: probably 7 (escape+SO+DBCS), need to verify
- hypothetical charset with a mapping from 3 UChars to 17 bytes: 6
(17/3=5.667 rounded up)
Jitterbug: 2949 http://www.jtcsv.com/cgibin/icu-bugs?findid=2949
Sincerely,
markus