RE: ICU4C proposal: Formalizing the ustdio format specifiers

But...

> -----Original Message-----
> From: George Rhoten [mailto:[email protected]]
> Sent: Thursday, July 10, 2003 1:01 PM
> To: Robert Buck
> Cc: [email protected]
> Subject: RE: ICU4C proposal: Formalizing the ustdio format specifiers
>
>
> 1) Sadly a long isn't always 4 bytes.  It can also be 8 bytes on a few
> other platforms (Alpha is one of them).  This is why I'm not
> using long or
> long long in the format specification, and I'm only using int*_t
> types.  I
> have no plans to support ll or I64.  The reason for this decision was to
> follow a similar philosophy of Java where types always have the same size
> on all platforms.

The Java philosophy holds little weight to C/C++ programmers, and again,
we are talking about a C/C++ based product. Consistency with the C or C++
standard bears far more weight with C/C++ programmers. What you are
proposing
is essentially to dispense with an official ISO standard in favor of a non-
standard (Sun).

Now to say that you prefer to use sized types, as specified by C99, is
entirely reasonable. But the C99 _library_ spec is not at all expressed in
terms of the sized types. Many places it indicates the language types to be
used,
but not sized types.

How the language types map into sized types is a matter of platform or
compiler convention - which is basically what your platform specific
header files do for those platforms that don't provide stdint.h or
inttypes.h.

I hardly think it would be too hard to determine what the major compiler
mfr's do on each platform with respect to these types and format specifiers
(how wide types are, and what format specifiers they map into ), then
write the code in such a manner that it emulates what C/C++ developers
expect. In other words:

- if the user specifies 'l' with d, i, or u, then do integral promotion to
  an int32_t, int32_t, or uint32_t.
- if the user specifies 'll' with d, i, or u, then do integrat promotion to
  an int64_t, int64_t, or uint64_t.
- if the user specifies 'h' with d, i, or u, then cast to a int16_t,
int16_t, or
  an uint16_t.
- if the user specifies 'hh' with d, i, or u, then cast to a int8_t, int8_t,
or
  an uint8_t.

> I figure it would be much easier to say that h means short format, and l
> means long format as opposed to 'h means short int and l means long int.'

That is not at all what the C99 spec says. So why should we
want to befuddle developers using the ICU library?
The spec clearly states, and let me quote:

"The length modifiers and their meanings are:

"'hh' Specifies that a following d, i, o, u, x, or X conversion specifier
applies to a
'signed char' or 'unsigned char' argument (the argument will have
been promoted according to the integer promotions, but its value shall be
converted to 'signed char' or 'unsigned char' before printing); or that
a following n conversion specifier applies to a pointer to a signed char
argument.

"'h' Specifies that a following d, i, o, u, x, or X conversion specifier
applies to a
'short int' or 'unsigned short int' argument (the argument will
have been promoted according to the integer promotions, but its value shall
be converted to 'short int' or 'unsigned short int' before printing);
or that a following n conversion specifier applies to a pointer to a short
int argument."

The first paragraph is paraphrased:  'hh' implies susequent argument is by
convention an 8-bit integer, either int8_t or uint8_t.

The second paragraph is paraphrased:  'h' implies susequent argument is by
convention an 16-bit integer, either int16_t or uint16_t.

One may deduce/guess then that with the lack of a h or hh optional modifier,
with a %i, %d, or %u, that the subsequent argument is a int32_t or uint32_t.
Yes, this last point it's not spelled out in huge bold print, but is
implied.

So as you see, they do not say it "means short format", they clearly state
specific types, as in "unsigned short int", or unsigned char".

So what you are proposing is clearly way out on a limb, and non-standards
conformant. At least with respect to 'h'.

The 'l' (ell) issue is a bit more vague in the spec, because there is no
clear description of what the difference is between 'long int' and 'int' is.
You get the point. So, in these cases the best you can do for ICU users is
to follow platform convention - not terribly difficult to do in the
formatter
code. The chart I previously provided indicates some reasonable defaults.

> If you don't want l (ell) to mean int64_t, I guess I'll hold off on
> supporting 64-bit types in ustdio.  This topic can be brought up again at
> a later date.

No, I'd like to see both l (ell) and ll (ell-ell) put in. It's not too much
more work to do both.

> 2) I'm not using ints.  %S and %C on several platforms means that wchar_t
> should be used.  On Windows, this is 16-bits, but %S can also mean that
> char * is used instead depending on the API used.  On AIX, it can
> be 16 or
> 32-bits depending on the architecture.  On Linux, it's usually 32-bits,
> but it can also be 16-bits at times.  wchar_t sometimes means
> that Unicode
> is used, but sometimes it isn't.  I'm trying to stay away from using int,
> long or wchar_t types.  I'd really rather prefer to implement it
> the way I
> originally suggested.

I think it is reasonable to say that with most current operating systems,
or even most any produced in the past ten years, that an int is 32-bits.
To base your code upon [u]int32_t for what 'C' regards is a reasonable
choice.
Again, go back to the discussion regarding h vs hh, and l vs ll. The basis
of 'h' or 'hh' is an int, which on most reasonable OSs is 32-bits. Again,
it's all a matter of being conformant to the letter, or intent, of the
standard.

I really couldn't care less about wchar_t, because there are issues
that surround its use. But my basic objection was the use of _both_ 'l'
and 'h'. There is no precedent to use 'h' with a basis of 'short int',
only with an int.

> 3) I was planning on explicitly not supporting %ls and %lc for
> now. Markus
> had a great aversion to adding wchar_t support to ustdio right now
> (http://oss.software.ibm.com/icu/docs/papers/unicode_wchar_t.html)
> , so I'm
> reserving it for possible future implementation when someone has the need
> or time to implement wchar_t.  It's easier to add it in the future, but
> it's very difficult to remove it.

Okay.

>
> 4) I'll let you submit an RFE on it.  I don't want to formalize any
> callbacks yet, but I do see an advantage.  My main goal right now is to
> make a ustdio that works.  Many of the callbacks will be changing
> internally.  Special features like this are beyond what I can do for 2.8.

We use it in the message formatting and localization facilities.
It is quite useful.

Bob Buck

>
> George Rhoten
> IBM Globalization Center of Competency/ICU  San Jose, CA, USA
>

> "Robert Buck" <[email protected]>
> 07/10/2003 07:22 AM
>         To:     George Rhoten/San Jose/IBM@IBMUS,
> <[email protected]>
>         cc:
>         Subject:        RE: ICU4C proposal: Formalizing the
> ustdio format specifiers
> Great. I look forward to the changes. Esp support for %ld.
>
> A couple questions though...
>
> ##################################################################
> ##########
> ##
> 1.
>
> >From what I am reading in the C99 spec before me (Ch 7.19.6.1 article 4
> and
> Ch 7.19.6.2 article 11):
>
> , the conversion specifier must be present
> , the length modifer is optional
> , behavior of a specification using a length modifier by itself is
> undefined
>
> Does your spec suggest that '%l' is equivalent to '%ld' ? If so, please
> no.
>
> Also, the spec says...
>
> ", l (ell) Specifies that a following d, i, o, u, x, or X conversion
> specifier
> applies to a long int or unsigned long int argument; that a following n
> conversion specifier applies to a pointer to a long int argument; that a
> following c conversion specifier applies to a wint_t argument; that a
> following s conversion specifier applies to a pointer to a wchar_t
> argument; or has no effect on a following a, A, e, E, f, F, g, or G
> conversion
> specifier.
>
> ", ll (ell-ell) Specifies that a following d, i, o, u, x, or X conversion
> specifier applies to a long long int or unsigned long long int argument;
> or that a following n conversion specifier applies to a pointer to a long
> long int
> argument."
>
> It would seem to me that, apart from those outlier compilers that do not
> support 'long int' or 'long long int', that the 'l' and 'll' should behave
> as though the modifer indicates that the subsequent argument is at most
> the
> platform width of 'long int' for 'l', and the platform width of 'long long
> int'
> for 'll'. Taking a poll on five OSs, this means these would map to:
>
> type / sizeof   windows-ia32-vc7  linux-ia32  solaris-64  macosx  hpux-11
> --------------------------------------------------------------------------
> 'long long int'   unsupported          8         8           8        8
> 'long int'             4               4         4           4        4
>
> For win32 it would be reasonable to say that 'll' regards 64-bit types.
>
> I think it would be reasonable to say from evidence that %ld is used for
> int32_t,
> and %lld is used for int64_t. Similarly for corresponding unsigned
> quantities as well,
> %lu and %llu. In fact, if you use the '%ld' format on hpux for a 64 bit
> type,
> and a value of INT64_MAX, you get an error, which seems to further support
> my
> argument.
>
> printf("sizeof long int: %lld\n", INT64_MAX); // yields
> 9223372036854775807
> printf("sizeof long int: %ld\n", INT64_MAX); // yields 1001; an error of
> course
>
> Other platforms that support the spec for 'll' will yield similar errors.
>
> So in summary, %ld is _not_ for int64_t, nor is %lu for uint64_t according
> to the spec,
> or according to practice. Please do not introduce this error to the ICU
> api.
>
> ##################################################################
> ##########
> ##
> 2.
>
> Also, regarding the 'h' modifier and its use with regards to unicode
> strings,
> another alternative would be:
>
> > %S  UChar32 *  Null terminated UTF-16 string
> > %hS UChar *    Null terminated UTF-16 string
> > %hhS char *    Null terminated UTF-32 string
>
> and,
>
> > %C UChar32     32-bit Unicode code unit
> > %hC  UChar     16-bit Unicode code unit
> > %hhC char      8-bit Unicode code unit
>
> The point is that the conversion type for an 'h' modifier is presumed to
> be
> promoted to sizeof int. The int type is more naturally associated to
> UChar32 than UChar16. So one could argue that the above is more
> consistent with the intent of the C99 Spec on the basis that 'h' is
> meant to demote a type from an int to a type of short, and 'hh' is meant
> to demote an int to a char. Your use of 'h' is inconsistent with the
> spec as it demotes a short to a char. See Ch 7.24.2.1 article 7.
>
> ##################################################################
> ##########
> ##
> 3.
>
> > %ls N/A         (Unimplemented) Reserved for future implementation
>
> Actually %ls is already taken by the C99 Spec, and refers to a wchar_t *.
> So please consider dropping this from your spec, but support it according
> to platform/compiler conventions, and the C99 spec.
>
> > %lc N/A         (Unimplemented) Reserved for future implementation
>
> And in the spec this refers to a wint_t. Same as above.
>
> ##################################################################
> ##########
> ##
> 4.
>
> A nifty thing we use regularly in MATLAB is function holes. It essentially
> does a callback to a function that is allowed to put N characters into
> the output buffer. For us it is represented as a naked %U, so leaving out
> lots of details:
>
> switch (conversion specifier hole type) {
> case FUNCTION_HOLE_TYPE:
> {
> va_list   hole_ap = msg->hole_infos[holeIdx].hole.hole_ap;
> fn_fmtfcn fmtfcn  = va_arg(hole_ap, fn_fmtfcn);
> num_chars_written = (fmtfcn)(putsn, dest_buf_or_file_ptr, &hole_ap);
> break;
> }
> }
>
> typedef int (*fn_fmtfcn)(  /* returns the number of chars putsn'd after
> processing*/
> fn_putsn putsn,
> void *x,             /* destination info, e.g., FILE *, or &buffer */
> va_list *ap);        /* var args list to custom format specifier, on
> return
> this is advanced past the custom format varargs */
>
> Would it be possible to augment the icu api to support this sort of
> callback?
>
> -Bob
>
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]]On Behalf Of George
> > Rhoten
> > Sent: Wednesday, July 09, 2003 7:05 PM
> > To: [email protected]
> > Subject: ICU4C proposal: Formalizing the ustdio format specifiers
> >
> >
> > Deadline for comments: July 16, 2003
> >
> >
> > Introduction
> >
> > For ICU 2.8 we are making ustdio a fully supported library.  Basically,
> > this means that I will be fixing many broken things in the ustdio
> > library.
> >  Part of this fixing process is to formalize how the format
> specification
> > string works for fscanf, fprintf and related functions.
> >
> > As a reminder, the ustdio library has been marked as draft for a while,
> > which means that the API may change at any time.  We have also
> > stated that
> > ustdio is a broken unsupported library for a long time.  I'd like to
> make
> > the ustdio library more compatible with the ANSI C stdio format
> > specification.  Backwards compatibility with the old ustdio is not a
> goal
> > of this proposal, since that would involve keeping many bugs from the
> old
> > ustdio around for eternity.
> >
> > This is only a partial proposal.  A full proposal would be many pages
> > long, and most people wouldn't read it.  So I'm only sending out
> > a "small"
> > proposal for part of my ustdio work at this time so that it is easier to
> > read (hopefully).  More proposals will appear later on, like on how some
> > of the function APIs will look.
> >
> >
> > Actual Proposal
> >
> > Here are the format specifications that I plan to use in the new
> > supported
> > ustdio library.  These formats apply to u_fscanf, u_fprintf, u_sscanf,
> > u_sprintf and all related functions in ustdio.h.
> >
> > printf
> > fmt type        Comment
> > %E  double      Scientific with an uppercase exponent
> > %e  double      Scientific with a lowercase exponent
> > %G  double      Use %E or %f for best format
> > %g  double      Use %e or %f for best format
> > %f  double      Simple floating point without the exponent
> > %X  int32_t     ustdio special uppercase hex radix formatting
> > %x  int32_t     ustdio special lowercase hex radix formatting
> > %d  int32_t     Decimal format
> > %i  int32_t     Same as %d
> > %n  int32_t     count (write the number of chars written)
> > %o  int32_t     octal ustdio special octal radix formatting
> > %u  uint32_t    Decimal format
> > %p  void *      Prints the pointer value
> > %s  char *      Use default converter or specified converter from fopen
> > %hs char *      Use invariant converter
> > %ls N/A         (Unimplemented) Reserved for future implementation
> > %c  char        Use default converter or specified converter from fopen
> > %hc char        Use invariant converter
> > %lc N/A         (Unimplemented) Reserved for future implementation
> > %S  UChar *     Null terminated UTF-16 string
> > %hS char *      Null terminated UTF-8 string
> > %lS UChar32 *   Null terminated UTF-32 string
> > %C  UChar       16-bit Unicode code unit
> > %hC char        8-bit Unicode code unit
> > %lC UChar32     32-bit Unicode code unit
> > %%  N/A         Show a percent sign
> >
> > scanf differences from printf
> > %[] UChar *     Scanset needs to be replaced with ICU UnicodeSet.
> >                 Doesn't contain s, S or any other string format.
> >
> > Format modifiers
> > %l    int64_t   long format for %d, %i, %o, %x (usually this means long
> > int)
> > %h    int16_t   short format for %d, %i, %o, %x (usually this means
> short
> > int)
> > %l    uint64_t  long format for %u (usually this means long int)
> > %h    uint16_t  short format for %u (usually this means short int)
> > %-    N/A       Left justify
> > %+    N/A       Always show the plus or minus sign. Needs data for plus
> > sign.
> > %     N/A       Instead of a "+" output a blank character for positive
> > numbers.
> > %#    N/A       Precede octal value with 0, hex with 0x and show the
> > decimal point for floats.
> > %num  N/A       Width of input/output. num is an actual number from 0 to
> > some large number.
> > %.num N/A       Significant digits precision. num is an actual
> > number from
> > 0
> >                 to some large number. Currently can only specify
> precision
> >                 before or after decimal, and not total precision.
> >
> > printf modifier
> > %*  int32_t     Next argument after this one specifies the width (need
> to
> > implement)
> >
> > scanf modifier
> > %*  N/A         This field is scanned, but not stored (need to
> implement)
> >
> > Special extensions reserved for future consideration
> > %P  double      Percent format
> > %V  double      Spellout format
> > %b  char *      HP-UX and glibc extension which allows you to unescape a
> > string
> > %B  UChar *     Similar to %b
> > % $ N/A         positional formatting modifier like in MessageFormat
> from
> > the "Single Unix Specification".
> >
> > Formats removed due to other formats or functions.
> > %T  UDate       Time. Replaced by strftime/strptime.
> > %D  UDate       Date. Replaced by strftime/strptime.
> > %K  UChar       Source or target is UChar. Replaced by %C.
> > %U  UChar *     Source or target is UChar *. Replaced by %S.
> > %M  double      Currency format. Replaced by strfmon. strpmon may be
> > implemented in the future.
> >
> >
> >
> > Other related changes
> >
> > Functions removed
> > u_getcx()       %b will replace this function in the future
> >
> > The char * format specification will now be converted with
> > u_charsToUChars
> > instead of the default converter.  If you want to display non-ASCII
> > characters, you need to use the functions that use a const UChar *
> format
> > argument.  This is being done because:
> >
> > 1) It's much faster
> > 2) The format specification argument is usually a static char *, but the
> > default converter may convert the string differently depending on the
> > platform ICU is being run on.  This is generally bad, and that's why the
> > UChar * format argument should be used for internationalized text
> instead
> > of the API that takes a char * formatting argument.  The %s argument
> will
> > still use the default converter since the source of the string is much
> > more likely to come from an outside source.
> >
> >
> >
> > Thank you for taking time to read this proposal.
> >
> > George Rhoten
> > IBM Globalization Center of Competency/ICU  San Jose, CA, USA
> > _______________________________________________
> > icu mailing list
> > [email protected]
> > http://oss.software.ibm.com/developerworks/oss/mailman/listinfo/icu
>
>