Re: FreeTDS and UTF-8 support

On Fri, Jul 25, 2008 at 10:03:33AM +0200, Sebastien FLAESCH wrote:
> >> 3) Before sending on TDS protocol, convert from current charset to
> >> WideChar (UTF-16/UCS-2) with mbstowcs() and when receiving TDS responses,
> >> convert from WideChar to current charset with wcstombs().

> > So mbtowc() and wctomb() only convert between widechar and multibyte
> > representations *of the same charset*.  That makes them insufficient for
> > converting to "the current charset", because you always want your WC
> > representation to be UCS-2 but you want your mb representation to be in the
> > current locale, which may not be a Unicode locale.

> mbtowc(), wctomb(), mbstowcs() and wcstombs() can convert Wide Char to/from
> *the current* SB or MB charset defined by a setlocale() call.

>  From the man pages of wctomb:

> "The behavior of wctomb depends on the LC_CTYPE category of the current locale."

Yes.  The issue is that a mbstowcs() conversion is *not* guaranteed to give
you UCS-4 encoding; it just means that you get a widechar representation of
whatever your multibyte charset was, which may or may not have been unicode
before, and may or may not be in UCS-4 afterwards.

> We use that in our VM to convert from the current charset to wchar_t in order
> to implement upshift/downshift functions with towupper / towlower (you cannot
> use toupper/tolower when current charset is MB ...

towupper / towlower are locale-sensitive as well; from towupper(3) on my
glibc-based system:

NOTES
       The behavior of towupper() depends on the LC_CTYPE category of the
       current locale.

       This  function  is not very appropriate for dealing with Unicode
       characters, because Unicode knows about three cases: upper, lower and
       title case.

That makes it pretty clear that wchar_t does not imply Unicode.

> >>  From my understanding, a C application on UNIX can be written in:

> >> A- Pure ASCII (LC_ALL=POSIX)
> >> B- A specific single-byte charset (LC_ALL=fr_FR.iso88591)
> >> C- A specific multi-byte charset (LC_ALL=ja_JP.eucjp or en_US.utf8)
> >> D- Wide Char (UTF-16/UCS-2), using the wchar_t type.
> > 
> > I don't think you can be very effective writing a C application on Unix
> > using a widechar representation internally.  All string manipulation
> > functions in the standard library, and in other common libraries built on
> > top of them, are implemented using char* as the argument type.

> Honestly I have no idea...

> I was just guessing there are WC apps out there on UNIX because I know
> about some WC libc functions, you have for example fwprintf(), getwchar().

Yes, there are some libc functions that are defined to have WC equivalents;
but in general, you won't find A/W pairs in other libraries on Unix the way
you will on Windows, so there are plenty of cases where you'll have to
convert (as opposed to only having to convert from mb to wc when you're
doing character-wise operations, such as towupper() or wcslen()).

> http://blake.ism.u-bordeaux1.fr/doc_link/C/a_doc_lib/aixprggd/genprogc/multi-byte_widechar_subr.htm

> Note I found no equivalent for fopen()... probably because UNIX file
> systems don't have an explicit charset? (filenames are just a set of bytes,
> you can have one file name in UTF-8 and another in ISO-8859-1, AFAIK).

Well, an fopenW() would require either a syscall that can take a wchar_t*
argument, or a libc implementation that converts it to a C string for
passing to the syscall.  Neither of these make sense to implement; the only
difference would be who's responsible for converting the wchar_t* to char*
for the on-disk comparison, so there's no reason why it shouldn't just be
the caller's responsibility.

"explicit charset" is less the issue than that all Unix filesystems use C
strings as filenames.  (And the non-Unix filesystems that don't, such as
NTFS, don't use Unix wchar_t either...)

Cheers,
-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
[email protected]                                     [email protected]