RE: ICU4C proposal: Formalizing the ustdio format specifiers

1) Sadly a long isn't always 4 bytes.  It can also be 8 bytes on a few 
other platforms (Alpha is one of them).  This is why I'm not using long or 
long long in the format specification, and I'm only using int*_t types.  I 
have no plans to support ll or I64.  The reason for this decision was to 
follow a similar philosophy of Java where types always have the same size 
on all platforms.

I figure it would be much easier to say that h means short format, and l 
means long format as opposed to 'h means short int and l means long int.' 
If you don't want l (ell) to mean int64_t, I guess I'll hold off on 
supporting 64-bit types in ustdio.  This topic can be brought up again at 
a later date.

I withdraw the l (ell) format modifier for meaning anything on %d, %i, %o, 
%x, %X or %u.

I agree with you about the optional type specifier.  The type must be 
specified after the l or h in ustdio.

2) I'm not using ints.  %S and %C on several platforms means that wchar_t 
should be used.  On Windows, this is 16-bits, but %S can also mean that 
char * is used instead depending on the API used.  On AIX, it can be 16 or 
32-bits depending on the architecture.  On Linux, it's usually 32-bits, 
but it can also be 16-bits at times.  wchar_t sometimes means that Unicode 
is used, but sometimes it isn't.  I'm trying to stay away from using int, 
long or wchar_t types.  I'd really rather prefer to implement it the way I 
originally suggested.

3) I was planning on explicitly not supporting %ls and %lc for now. Markus 
had a great aversion to adding wchar_t support to ustdio right now 
(http://oss.software.ibm.com/icu/docs/papers/unicode_wchar_t.html), so I'm 
reserving it for possible future implementation when someone has the need 
or time to implement wchar_t.  It's easier to add it in the future, but 
it's very difficult to remove it.

4) I'll let you submit an RFE on it.  I don't want to formalize any 
callbacks yet, but I do see an advantage.  My main goal right now is to 
make a ustdio that works.  Many of the callbacks will be changing 
internally.  Special features like this are beyond what I can do for 2.8.

George Rhoten
IBM Globalization Center of Competency/ICU  San Jose, CA, USA




"Robert Buck" <[email protected]>
07/10/2003 07:22 AM

 
        To:     George Rhoten/San Jose/IBM@IBMUS, <[email protected]>
        cc: 
        Subject:        RE: ICU4C proposal: Formalizing the ustdio format specifiers



Great. I look forward to the changes. Esp support for %ld.

A couple questions though...

############################################################################
##
1.

From what I am reading in the C99 spec before me (Ch 7.19.6.1 article 4 
and
Ch 7.19.6.2 article 11):

, the conversion specifier must be present
, the length modifer is optional
, behavior of a specification using a length modifier by itself is 
undefined

Does your spec suggest that '%l' is equivalent to '%ld' ? If so, please 
no.

Also, the spec says...

", l (ell) Specifies that a following d, i, o, u, x, or X conversion
specifier
applies to a long int or unsigned long int argument; that a following n
conversion specifier applies to a pointer to a long int argument; that a
following c conversion specifier applies to a wint_t argument; that a
following s conversion specifier applies to a pointer to a wchar_t
argument; or has no effect on a following a, A, e, E, f, F, g, or G
conversion
specifier.

", ll (ell-ell) Specifies that a following d, i, o, u, x, or X conversion
specifier applies to a long long int or unsigned long long int argument;
or that a following n conversion specifier applies to a pointer to a long
long int
argument."

It would seem to me that, apart from those outlier compilers that do not
support 'long int' or 'long long int', that the 'l' and 'll' should behave
as though the modifer indicates that the subsequent argument is at most 
the
platform width of 'long int' for 'l', and the platform width of 'long long
int'
for 'll'. Taking a poll on five OSs, this means these would map to:

type / sizeof   windows-ia32-vc7  linux-ia32  solaris-64  macosx  hpux-11
--------------------------------------------------------------------------
'long long int'   unsupported          8         8           8        8
'long int'             4               4         4           4        4

For win32 it would be reasonable to say that 'll' regards 64-bit types.

I think it would be reasonable to say from evidence that %ld is used for
int32_t,
and %lld is used for int64_t. Similarly for corresponding unsigned
quantities as well,
%lu and %llu. In fact, if you use the '%ld' format on hpux for a 64 bit
type,
and a value of INT64_MAX, you get an error, which seems to further support
my
argument.

printf("sizeof long int: %lld\n", INT64_MAX); // yields 
9223372036854775807
printf("sizeof long int: %ld\n", INT64_MAX); // yields 1001; an error of
course

Other platforms that support the spec for 'll' will yield similar errors.

So in summary, %ld is _not_ for int64_t, nor is %lu for uint64_t according
to the spec,
or according to practice. Please do not introduce this error to the ICU 
api.

############################################################################
##
2.

Also, regarding the 'h' modifier and its use with regards to unicode
strings,
another alternative would be:

> %S  UChar32 *  Null terminated UTF-16 string
> %hS UChar *    Null terminated UTF-16 string
> %hhS char *    Null terminated UTF-32 string

and,

> %C UChar32     32-bit Unicode code unit
> %hC  UChar     16-bit Unicode code unit
> %hhC char      8-bit Unicode code unit

The point is that the conversion type for an 'h' modifier is presumed to 
be
promoted to sizeof int. The int type is more naturally associated to
UChar32 than UChar16. So one could argue that the above is more
consistent with the intent of the C99 Spec on the basis that 'h' is
meant to demote a type from an int to a type of short, and 'hh' is meant
to demote an int to a char. Your use of 'h' is inconsistent with the
spec as it demotes a short to a char. See Ch 7.24.2.1 article 7.

############################################################################
##
3.

> %ls N/A         (Unimplemented) Reserved for future implementation

Actually %ls is already taken by the C99 Spec, and refers to a wchar_t *.
So please consider dropping this from your spec, but support it according
to platform/compiler conventions, and the C99 spec.

> %lc N/A         (Unimplemented) Reserved for future implementation

And in the spec this refers to a wint_t. Same as above.

############################################################################
##
4.

A nifty thing we use regularly in MATLAB is function holes. It essentially
does a callback to a function that is allowed to put N characters into
the output buffer. For us it is represented as a naked %U, so leaving out
lots of details:

switch (conversion specifier hole type) {
case FUNCTION_HOLE_TYPE:
{
va_list   hole_ap = msg->hole_infos[holeIdx].hole.hole_ap;
fn_fmtfcn fmtfcn  = va_arg(hole_ap, fn_fmtfcn);
num_chars_written = (fmtfcn)(putsn, dest_buf_or_file_ptr, &hole_ap);
break;
}
}

typedef int (*fn_fmtfcn)(  /* returns the number of chars putsn'd after
processing*/
fn_putsn putsn,
void *x,             /* destination info, e.g., FILE *, or &buffer */
va_list *ap);        /* var args list to custom format specifier, on
return
this is advanced past the custom format varargs */

Would it be possible to augment the icu api to support this sort of
callback?

-Bob

> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]]On Behalf Of George
> Rhoten
> Sent: Wednesday, July 09, 2003 7:05 PM
> To: [email protected]
> Subject: ICU4C proposal: Formalizing the ustdio format specifiers
>
>
> Deadline for comments: July 16, 2003
>
>
> Introduction
>
> For ICU 2.8 we are making ustdio a fully supported library.  Basically,
> this means that I will be fixing many broken things in the ustdio
> library.
>  Part of this fixing process is to formalize how the format 
specification
> string works for fscanf, fprintf and related functions.
>
> As a reminder, the ustdio library has been marked as draft for a while,
> which means that the API may change at any time.  We have also
> stated that
> ustdio is a broken unsupported library for a long time.  I'd like to 
make
> the ustdio library more compatible with the ANSI C stdio format
> specification.  Backwards compatibility with the old ustdio is not a 
goal
> of this proposal, since that would involve keeping many bugs from the 
old
> ustdio around for eternity.
>
> This is only a partial proposal.  A full proposal would be many pages
> long, and most people wouldn't read it.  So I'm only sending out
> a "small"
> proposal for part of my ustdio work at this time so that it is easier to
> read (hopefully).  More proposals will appear later on, like on how some
> of the function APIs will look.
>
>
> Actual Proposal
>
> Here are the format specifications that I plan to use in the new
> supported
> ustdio library.  These formats apply to u_fscanf, u_fprintf, u_sscanf,
> u_sprintf and all related functions in ustdio.h.
>
> printf
> fmt type        Comment
> %E  double      Scientific with an uppercase exponent
> %e  double      Scientific with a lowercase exponent
> %G  double      Use %E or %f for best format
> %g  double      Use %e or %f for best format
> %f  double      Simple floating point without the exponent
> %X  int32_t     ustdio special uppercase hex radix formatting
> %x  int32_t     ustdio special lowercase hex radix formatting
> %d  int32_t     Decimal format
> %i  int32_t     Same as %d
> %n  int32_t     count (write the number of chars written)
> %o  int32_t     octal ustdio special octal radix formatting
> %u  uint32_t    Decimal format
> %p  void *      Prints the pointer value
> %s  char *      Use default converter or specified converter from fopen
> %hs char *      Use invariant converter
> %ls N/A         (Unimplemented) Reserved for future implementation
> %c  char        Use default converter or specified converter from fopen
> %hc char        Use invariant converter
> %lc N/A         (Unimplemented) Reserved for future implementation
> %S  UChar *     Null terminated UTF-16 string
> %hS char *      Null terminated UTF-8 string
> %lS UChar32 *   Null terminated UTF-32 string
> %C  UChar       16-bit Unicode code unit
> %hC char        8-bit Unicode code unit
> %lC UChar32     32-bit Unicode code unit
> %%  N/A         Show a percent sign
>
> scanf differences from printf
> %[] UChar *     Scanset needs to be replaced with ICU UnicodeSet.
>                 Doesn't contain s, S or any other string format.
>
> Format modifiers
> %l    int64_t   long format for %d, %i, %o, %x (usually this means long
> int)
> %h    int16_t   short format for %d, %i, %o, %x (usually this means 
short
> int)
> %l    uint64_t  long format for %u (usually this means long int)
> %h    uint16_t  short format for %u (usually this means short int)
> %-    N/A       Left justify
> %+    N/A       Always show the plus or minus sign. Needs data for plus
> sign.
> %     N/A       Instead of a "+" output a blank character for positive
> numbers.
> %#    N/A       Precede octal value with 0, hex with 0x and show the
> decimal point for floats.
> %num  N/A       Width of input/output. num is an actual number from 0 to
> some large number.
> %.num N/A       Significant digits precision. num is an actual
> number from
> 0
>                 to some large number. Currently can only specify 
precision
>                 before or after decimal, and not total precision.
>
> printf modifier
> %*  int32_t     Next argument after this one specifies the width (need 
to
> implement)
>
> scanf modifier
> %*  N/A         This field is scanned, but not stored (need to 
implement)
>
> Special extensions reserved for future consideration
> %P  double      Percent format
> %V  double      Spellout format
> %b  char *      HP-UX and glibc extension which allows you to unescape a
> string
> %B  UChar *     Similar to %b
> % $ N/A         positional formatting modifier like in MessageFormat 
from
> the "Single Unix Specification".
>
> Formats removed due to other formats or functions.
> %T  UDate       Time. Replaced by strftime/strptime.
> %D  UDate       Date. Replaced by strftime/strptime.
> %K  UChar       Source or target is UChar. Replaced by %C.
> %U  UChar *     Source or target is UChar *. Replaced by %S.
> %M  double      Currency format. Replaced by strfmon. strpmon may be
> implemented in the future.
>
>
>
> Other related changes
>
> Functions removed
> u_getcx()       %b will replace this function in the future
>
> The char * format specification will now be converted with
> u_charsToUChars
> instead of the default converter.  If you want to display non-ASCII
> characters, you need to use the functions that use a const UChar * 
format
> argument.  This is being done because:
>
> 1) It's much faster
> 2) The format specification argument is usually a static char *, but the
> default converter may convert the string differently depending on the
> platform ICU is being run on.  This is generally bad, and that's why the
> UChar * format argument should be used for internationalized text 
instead
> of the API that takes a char * formatting argument.  The %s argument 
will
> still use the default converter since the source of the string is much
> more likely to come from an outside source.
>
>
>
> Thank you for taking time to read this proposal.
>
> George Rhoten
> IBM Globalization Center of Competency/ICU  San Jose, CA, USA
> _______________________________________________
> icu mailing list
> [email protected]
> http://oss.software.ibm.com/developerworks/oss/mailman/listinfo/icu