icu4c api proposal: fix/change ucnv_getMaxCharSize()

Markus Scherer <[email protected]>
Newsgroups gmane.comp.lib.icu.general
Organization IBM
Message-ID <[email protected]>
ucnv_getMaxCharSize() is currently documented as:
  * Returns the maximum length of bytes used by a character. This varies
  * between 1 and 4

This is clear and simple for trivial cases like ISO 8859-x and traditional Shift-JIS. However, it is 
not at all clear what this function returns, and what to do with it, in more complicated cases, and 
the behavior is all over the map for various converter implementations.

Issues:
- max bytes per UChar or per code point?
     different for mappings involving supplementaries
     like UTF-8
- max bytes in character content, or including
   shifting output (SI/SO, escapes, etc.)?
     different for stateful converters
     like ISO-2022-JP
- max bytes per character or per conversion unit?
     different when converting from multiple UChars/code points
     like for JIS X 0213
- max bytes per conversion unit, or including
   all output that is somehow related to some input?
     different when a converter outputs return-to-initial-state bytes
     like EBCDIC_STATEFUL,
     or initial bytes like the escape to UTF-8 in the generic
     ISO-2022 converter

 From anecdotal evidence (user complaints), this function is used as part of an estimate to allocate 
an output buffer that is guaranteed to be large enough to hold the conversion result for an input 
Unicode buffer of a given size.

Proposal:

1.
Add a macro for such a guaranteed-buffer allocation estimate, taking as parameters the length of the 
Unicode text (in UChars) and the return value of ucnv_getMaxCharSize():

#define UCNV_GET_MAX_BYTES_FOR_STRING(length, maxCharSize) \
     (((length)+10)*(maxCharSize))

The +10 takes care of initial and final bytes that are output without direct association with 
specific input.

2.
Modify the implementation of ucnv_getMaxCharSize() so that its return value works with the above 
macro. This means:
- max bytes per UChar
- including shifting output
- when converting from multiple Unicode characters:
   max bytes per input UChar (rounded up)
- not including initial/final output bytes,
   but see comment under 1. about the +10 in the macro


Remaining issues:
- ucnv_getMaxCharSize() may not return what is intuitively expected;
   as proposed, it will return 3 not 4 for UTF-8. (See below.)
   However, there is no one number that is suitable for both
   an abstract unit size and a reasonable buffer size estimate.
- The function will not take into account the size of
   callback output. It will therefore provide accurate results if
   the stop/skip/substitution callbacks or similar are used,
   but not for the escape callback or similar.

Expiration: Tuesday, 2003-nov-04

Examples:
- traditional single-byte charsets: 1
- traditional Shift-JIS: 2
- single-byte charset which maps the ffi-ligature to f+f+i: 3
- EBCDIC_STATEFUL: 3 (worst case: SO+DBCS)
- UTF-8: 3 (4 bytes per supplementary are per 2 UChars,
             so worst case is U+0800..U+FFFF)
- ISO-2022-JP: probably 7 (escape+SO+DBCS), need to verify
- hypothetical charset with a mapping from 3 UChars to 17 bytes: 6
   (17/3=5.667 rounded up)

Jitterbug: 2949  http://www.jtcsv.com/cgibin/icu-bugs?findid=2949

Sincerely,
markus
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.