Re: ICU4C proposal: Formalizing the ustdio format specifiers

> - if the user specifies 'l' with d, i, or u, then do integral
promotion to
>   an int32_t, int32_t, or uint32_t.
> - if the user specifies 'll' with d, i, or u, then do integrat
promotion to
>   an int64_t, int64_t, or uint64_t.

We can't do this in scanf. It's got to be exactly the right type.

Mark
__________________________________
http://www.macchiato.com
►  “Eppur si muove” ◄

----- Original Message ----- 
From: "Robert Buck" <[email protected]>
To: "George Rhoten" <[email protected]>
Cc: <[email protected]>
Sent: Thursday, July 10, 2003 11:03
Subject: RE: ICU4C proposal: Formalizing the ustdio format specifiers


> But...
>
> > -----Original Message-----
> > From: George Rhoten [mailto:[email protected]]
> > Sent: Thursday, July 10, 2003 1:01 PM
> > To: Robert Buck
> > Cc: [email protected]
> > Subject: RE: ICU4C proposal: Formalizing the ustdio format
specifiers
> >
> >
> > 1) Sadly a long isn't always 4 bytes.  It can also be 8 bytes on a
few
> > other platforms (Alpha is one of them).  This is why I'm not
> > using long or
> > long long in the format specification, and I'm only using int*_t
> > types.  I
> > have no plans to support ll or I64.  The reason for this decision
was to
> > follow a similar philosophy of Java where types always have the
same size
> > on all platforms.
>
> The Java philosophy holds little weight to C/C++ programmers, and
again,
> we are talking about a C/C++ based product. Consistency with the C
or C++
> standard bears far more weight with C/C++ programmers. What you are
> proposing
> is essentially to dispense with an official ISO standard in favor of
a non-
> standard (Sun).
>
> Now to say that you prefer to use sized types, as specified by C99,
is
> entirely reasonable. But the C99 _library_ spec is not at all
expressed in
> terms of the sized types. Many places it indicates the language
types to be
> used,
> but not sized types.
>
> How the language types map into sized types is a matter of platform
or
> compiler convention - which is basically what your platform specific
> header files do for those platforms that don't provide stdint.h or
> inttypes.h.
>
> I hardly think it would be too hard to determine what the major
compiler
> mfr's do on each platform with respect to these types and format
specifiers
> (how wide types are, and what format specifiers they map into ),
then
> write the code in such a manner that it emulates what C/C++
developers
> expect. In other words:
>
> - if the user specifies 'l' with d, i, or u, then do integral
promotion to
>   an int32_t, int32_t, or uint32_t.
> - if the user specifies 'll' with d, i, or u, then do integrat
promotion to
>   an int64_t, int64_t, or uint64_t.
> - if the user specifies 'h' with d, i, or u, then cast to a int16_t,
> int16_t, or
>   an uint16_t.
> - if the user specifies 'hh' with d, i, or u, then cast to a int8_t,
int8_t,
> or
>   an uint8_t.
>
> > I figure it would be much easier to say that h means short format,
and l
> > means long format as opposed to 'h means short int and l means
long int.'
>
> That is not at all what the C99 spec says. So why should we
> want to befuddle developers using the ICU library?
> The spec clearly states, and let me quote:
>
> "The length modifiers and their meanings are:
>
> "'hh' Specifies that a following d, i, o, u, x, or X conversion
specifier
> applies to a
> 'signed char' or 'unsigned char' argument (the argument will have
> been promoted according to the integer promotions, but its value
shall be
> converted to 'signed char' or 'unsigned char' before printing); or
that
> a following n conversion specifier applies to a pointer to a signed
char
> argument.
>
> "'h' Specifies that a following d, i, o, u, x, or X conversion
specifier
> applies to a
> 'short int' or 'unsigned short int' argument (the argument will
> have been promoted according to the integer promotions, but its
value shall
> be converted to 'short int' or 'unsigned short int' before
printing);
> or that a following n conversion specifier applies to a pointer to a
short
> int argument."
>
> The first paragraph is paraphrased:  'hh' implies susequent argument
is by
> convention an 8-bit integer, either int8_t or uint8_t.
>
> The second paragraph is paraphrased:  'h' implies susequent argument
is by
> convention an 16-bit integer, either int16_t or uint16_t.
>
> One may deduce/guess then that with the lack of a h or hh optional
modifier,
> with a %i, %d, or %u, that the subsequent argument is a int32_t or
uint32_t.
> Yes, this last point it's not spelled out in huge bold print, but is
> implied.
>
> So as you see, they do not say it "means short format", they clearly
state
> specific types, as in "unsigned short int", or unsigned char".
>
> So what you are proposing is clearly way out on a limb, and
non-standards
> conformant. At least with respect to 'h'.
>
> The 'l' (ell) issue is a bit more vague in the spec, because there
is no
> clear description of what the difference is between 'long int' and
'int' is.
> You get the point. So, in these cases the best you can do for ICU
users is
> to follow platform convention - not terribly difficult to do in the
> formatter
> code. The chart I previously provided indicates some reasonable
defaults.
>
> > If you don't want l (ell) to mean int64_t, I guess I'll hold off
on
> > supporting 64-bit types in ustdio.  This topic can be brought up
again at
> > a later date.
>
> No, I'd like to see both l (ell) and ll (ell-ell) put in. It's not
too much
> more work to do both.
>
> > 2) I'm not using ints.  %S and %C on several platforms means that
wchar_t
> > should be used.  On Windows, this is 16-bits, but %S can also mean
that
> > char * is used instead depending on the API used.  On AIX, it can
> > be 16 or
> > 32-bits depending on the architecture.  On Linux, it's usually
32-bits,
> > but it can also be 16-bits at times.  wchar_t sometimes means
> > that Unicode
> > is used, but sometimes it isn't.  I'm trying to stay away from
using int,
> > long or wchar_t types.  I'd really rather prefer to implement it
> > the way I
> > originally suggested.
>
> I think it is reasonable to say that with most current operating
systems,
> or even most any produced in the past ten years, that an int is
32-bits.
> To base your code upon [u]int32_t for what 'C' regards is a
reasonable
> choice.
> Again, go back to the discussion regarding h vs hh, and l vs ll. The
basis
> of 'h' or 'hh' is an int, which on most reasonable OSs is 32-bits.
Again,
> it's all a matter of being conformant to the letter, or intent, of
the
> standard.
>
> I really couldn't care less about wchar_t, because there are issues
> that surround its use. But my basic objection was the use of _both_
'l'
> and 'h'. There is no precedent to use 'h' with a basis of 'short
int',
> only with an int.
>
> > 3) I was planning on explicitly not supporting %ls and %lc for
> > now. Markus
> > had a great aversion to adding wchar_t support to ustdio right now
> > (http://oss.software.ibm.com/icu/docs/papers/unicode_wchar_t.html)
> > , so I'm
> > reserving it for possible future implementation when someone has
the need
> > or time to implement wchar_t.  It's easier to add it in the
future, but
> > it's very difficult to remove it.
>
> Okay.
>
> >
> > 4) I'll let you submit an RFE on it.  I don't want to formalize
any
> > callbacks yet, but I do see an advantage.  My main goal right now
is to
> > make a ustdio that works.  Many of the callbacks will be changing
> > internally.  Special features like this are beyond what I can do
for 2.8.
>
> We use it in the message formatting and localization facilities.
> It is quite useful.
>
> Bob Buck
>
> >
> > George Rhoten
> > IBM Globalization Center of Competency/ICU  San Jose, CA, USA
> >
>
> > "Robert Buck" <[email protected]>
> > 07/10/2003 07:22 AM
> >         To:     George Rhoten/San Jose/IBM@IBMUS,
> > <[email protected]>
> >         cc:
> >         Subject:        RE: ICU4C proposal: Formalizing the
> > ustdio format specifiers
> > Great. I look forward to the changes. Esp support for %ld.
> >
> > A couple questions though...
> >
> > ##################################################################
> > ##########
> > ##
> > 1.
> >
> > >From what I am reading in the C99 spec before me (Ch 7.19.6.1
article 4
> > and
> > Ch 7.19.6.2 article 11):
> >
> > , the conversion specifier must be present
> > , the length modifer is optional
> > , behavior of a specification using a length modifier by itself is
> > undefined
> >
> > Does your spec suggest that '%l' is equivalent to '%ld' ? If so,
please
> > no.
> >
> > Also, the spec says...
> >
> > ", l (ell) Specifies that a following d, i, o, u, x, or X
conversion
> > specifier
> > applies to a long int or unsigned long int argument; that a
following n
> > conversion specifier applies to a pointer to a long int argument;
that a
> > following c conversion specifier applies to a wint_t argument;
that a
> > following s conversion specifier applies to a pointer to a wchar_t
> > argument; or has no effect on a following a, A, e, E, f, F, g, or
G
> > conversion
> > specifier.
> >
> > ", ll (ell-ell) Specifies that a following d, i, o, u, x, or X
conversion
> > specifier applies to a long long int or unsigned long long int
argument;
> > or that a following n conversion specifier applies to a pointer to
a long
> > long int
> > argument."
> >
> > It would seem to me that, apart from those outlier compilers that
do not
> > support 'long int' or 'long long int', that the 'l' and 'll'
should behave
> > as though the modifer indicates that the subsequent argument is at
most
> > the
> > platform width of 'long int' for 'l', and the platform width of
'long long
> > int'
> > for 'll'. Taking a poll on five OSs, this means these would map
to:
> >
> > type / sizeof   windows-ia32-vc7  linux-ia32  solaris-64  macosx
hpux-11
>
> --------------------------------------------------------------------
------
> > 'long long int'   unsupported          8         8           8
8
> > 'long int'             4               4         4           4
4
> >
> > For win32 it would be reasonable to say that 'll' regards 64-bit
types.
> >
> > I think it would be reasonable to say from evidence that %ld is
used for
> > int32_t,
> > and %lld is used for int64_t. Similarly for corresponding unsigned
> > quantities as well,
> > %lu and %llu. In fact, if you use the '%ld' format on hpux for a
64 bit
> > type,
> > and a value of INT64_MAX, you get an error, which seems to further
support
> > my
> > argument.
> >
> > printf("sizeof long int: %lld\n", INT64_MAX); // yields
> > 9223372036854775807
> > printf("sizeof long int: %ld\n", INT64_MAX); // yields 1001; an
error of
> > course
> >
> > Other platforms that support the spec for 'll' will yield similar
errors.
> >
> > So in summary, %ld is _not_ for int64_t, nor is %lu for uint64_t
according
> > to the spec,
> > or according to practice. Please do not introduce this error to
the ICU
> > api.
> >
> > ##################################################################
> > ##########
> > ##
> > 2.
> >
> > Also, regarding the 'h' modifier and its use with regards to
unicode
> > strings,
> > another alternative would be:
> >
> > > %S  UChar32 *  Null terminated UTF-16 string
> > > %hS UChar *    Null terminated UTF-16 string
> > > %hhS char *    Null terminated UTF-32 string
> >
> > and,
> >
> > > %C UChar32     32-bit Unicode code unit
> > > %hC  UChar     16-bit Unicode code unit
> > > %hhC char      8-bit Unicode code unit
> >
> > The point is that the conversion type for an 'h' modifier is
presumed to
> > be
> > promoted to sizeof int. The int type is more naturally associated
to
> > UChar32 than UChar16. So one could argue that the above is more
> > consistent with the intent of the C99 Spec on the basis that 'h'
is
> > meant to demote a type from an int to a type of short, and 'hh' is
meant
> > to demote an int to a char. Your use of 'h' is inconsistent with
the
> > spec as it demotes a short to a char. See Ch 7.24.2.1 article 7.
> >
> > ##################################################################
> > ##########
> > ##
> > 3.
> >
> > > %ls N/A         (Unimplemented) Reserved for future
implementation
> >
> > Actually %ls is already taken by the C99 Spec, and refers to a
wchar_t *.
> > So please consider dropping this from your spec, but support it
according
> > to platform/compiler conventions, and the C99 spec.
> >
> > > %lc N/A         (Unimplemented) Reserved for future
implementation
> >
> > And in the spec this refers to a wint_t. Same as above.
> >
> > ##################################################################
> > ##########
> > ##
> > 4.
> >
> > A nifty thing we use regularly in MATLAB is function holes. It
essentially
> > does a callback to a function that is allowed to put N characters
into
> > the output buffer. For us it is represented as a naked %U, so
leaving out
> > lots of details:
> >
> > switch (conversion specifier hole type) {
> > case FUNCTION_HOLE_TYPE:
> > {
> > va_list   hole_ap = msg->hole_infos[holeIdx].hole.hole_ap;
> > fn_fmtfcn fmtfcn  = va_arg(hole_ap, fn_fmtfcn);
> > num_chars_written = (fmtfcn)(putsn, dest_buf_or_file_ptr,
&hole_ap);
> > break;
> > }
> > }
> >
> > typedef int (*fn_fmtfcn)(  /* returns the number of chars putsn'd
after
> > processing*/
> > fn_putsn putsn,
> > void *x,             /* destination info, e.g., FILE *, or &buffer
*/
> > va_list *ap);        /* var args list to custom format specifier,
on
> > return
> > this is advanced past the custom format varargs */
> >
> > Would it be possible to augment the icu api to support this sort
of
> > callback?
> >
> > -Bob
> >
> > > -----Original Message-----
> > > From: [email protected]
> > > [mailto:[email protected]]On Behalf Of
George
> > > Rhoten
> > > Sent: Wednesday, July 09, 2003 7:05 PM
> > > To: [email protected]
> > > Subject: ICU4C proposal: Formalizing the ustdio format
specifiers
> > >
> > >
> > > Deadline for comments: July 16, 2003
> > >
> > >
> > > Introduction
> > >
> > > For ICU 2.8 we are making ustdio a fully supported library.
Basically,
> > > this means that I will be fixing many broken things in the
ustdio
> > > library.
> > >  Part of this fixing process is to formalize how the format
> > specification
> > > string works for fscanf, fprintf and related functions.
> > >
> > > As a reminder, the ustdio library has been marked as draft for a
while,
> > > which means that the API may change at any time.  We have also
> > > stated that
> > > ustdio is a broken unsupported library for a long time.  I'd
like to
> > make
> > > the ustdio library more compatible with the ANSI C stdio format
> > > specification.  Backwards compatibility with the old ustdio is
not a
> > goal
> > > of this proposal, since that would involve keeping many bugs
from the
> > old
> > > ustdio around for eternity.
> > >
> > > This is only a partial proposal.  A full proposal would be many
pages
> > > long, and most people wouldn't read it.  So I'm only sending out
> > > a "small"
> > > proposal for part of my ustdio work at this time so that it is
easier to
> > > read (hopefully).  More proposals will appear later on, like on
how some
> > > of the function APIs will look.
> > >
> > >
> > > Actual Proposal
> > >
> > > Here are the format specifications that I plan to use in the new
> > > supported
> > > ustdio library.  These formats apply to u_fscanf, u_fprintf,
u_sscanf,
> > > u_sprintf and all related functions in ustdio.h.
> > >
> > > printf
> > > fmt type        Comment
> > > %E  double      Scientific with an uppercase exponent
> > > %e  double      Scientific with a lowercase exponent
> > > %G  double      Use %E or %f for best format
> > > %g  double      Use %e or %f for best format
> > > %f  double      Simple floating point without the exponent
> > > %X  int32_t     ustdio special uppercase hex radix formatting
> > > %x  int32_t     ustdio special lowercase hex radix formatting
> > > %d  int32_t     Decimal format
> > > %i  int32_t     Same as %d
> > > %n  int32_t     count (write the number of chars written)
> > > %o  int32_t     octal ustdio special octal radix formatting
> > > %u  uint32_t    Decimal format
> > > %p  void *      Prints the pointer value
> > > %s  char *      Use default converter or specified converter
from fopen
> > > %hs char *      Use invariant converter
> > > %ls N/A         (Unimplemented) Reserved for future
implementation
> > > %c  char        Use default converter or specified converter
from fopen
> > > %hc char        Use invariant converter
> > > %lc N/A         (Unimplemented) Reserved for future
implementation
> > > %S  UChar *     Null terminated UTF-16 string
> > > %hS char *      Null terminated UTF-8 string
> > > %lS UChar32 *   Null terminated UTF-32 string
> > > %C  UChar       16-bit Unicode code unit
> > > %hC char        8-bit Unicode code unit
> > > %lC UChar32     32-bit Unicode code unit
> > > %%  N/A         Show a percent sign
> > >
> > > scanf differences from printf
> > > %[] UChar *     Scanset needs to be replaced with ICU
UnicodeSet.
> > >                 Doesn't contain s, S or any other string format.
> > >
> > > Format modifiers
> > > %l    int64_t   long format for %d, %i, %o, %x (usually this
means long
> > > int)
> > > %h    int16_t   short format for %d, %i, %o, %x (usually this
means
> > short
> > > int)
> > > %l    uint64_t  long format for %u (usually this means long int)
> > > %h    uint16_t  short format for %u (usually this means short
int)
> > > %-    N/A       Left justify
> > > %+    N/A       Always show the plus or minus sign. Needs data
for plus
> > > sign.
> > > %     N/A       Instead of a "+" output a blank character for
positive
> > > numbers.
> > > %#    N/A       Precede octal value with 0, hex with 0x and show
the
> > > decimal point for floats.
> > > %num  N/A       Width of input/output. num is an actual number
from 0 to
> > > some large number.
> > > %.num N/A       Significant digits precision. num is an actual
> > > number from
> > > 0
> > >                 to some large number. Currently can only specify
> > precision
> > >                 before or after decimal, and not total
precision.
> > >
> > > printf modifier
> > > %*  int32_t     Next argument after this one specifies the width
(need
> > to
> > > implement)
> > >
> > > scanf modifier
> > > %*  N/A         This field is scanned, but not stored (need to
> > implement)
> > >
> > > Special extensions reserved for future consideration
> > > %P  double      Percent format
> > > %V  double      Spellout format
> > > %b  char *      HP-UX and glibc extension which allows you to
unescape a
> > > string
> > > %B  UChar *     Similar to %b
> > > % $ N/A         positional formatting modifier like in
MessageFormat
> > from
> > > the "Single Unix Specification".
> > >
> > > Formats removed due to other formats or functions.
> > > %T  UDate       Time. Replaced by strftime/strptime.
> > > %D  UDate       Date. Replaced by strftime/strptime.
> > > %K  UChar       Source or target is UChar. Replaced by %C.
> > > %U  UChar *     Source or target is UChar *. Replaced by %S.
> > > %M  double      Currency format. Replaced by strfmon. strpmon
may be
> > > implemented in the future.
> > >
> > >
> > >
> > > Other related changes
> > >
> > > Functions removed
> > > u_getcx()       %b will replace this function in the future
> > >
> > > The char * format specification will now be converted with
> > > u_charsToUChars
> > > instead of the default converter.  If you want to display
non-ASCII
> > > characters, you need to use the functions that use a const UChar
*
> > format
> > > argument.  This is being done because:
> > >
> > > 1) It's much faster
> > > 2) The format specification argument is usually a static char *,
but the
> > > default converter may convert the string differently depending
on the
> > > platform ICU is being run on.  This is generally bad, and that's
why the
> > > UChar * format argument should be used for internationalized
text
> > instead
> > > of the API that takes a char * formatting argument.  The %s
argument
> > will
> > > still use the default converter since the source of the string
is much
> > > more likely to come from an outside source.
> > >
> > >
> > >
> > > Thank you for taking time to read this proposal.
> > >
> > > George Rhoten
> > > IBM Globalization Center of Competency/ICU  San Jose, CA, USA
> > > _______________________________________________
> > > icu mailing list
> > > [email protected]
> > >
http://oss.software.ibm.com/developerworks/oss/mailman/listinfo/icu
> >
> >
>
> _______________________________________________
> icu mailing list
> [email protected]
> http://oss.software.ibm.com/developerworks/oss/mailman/listinfo/icu
>