Re: FreeTDS and UTF-8 support

Hello Steve,

Basically we agree, some comments below.

Seb

Steve Langasek wrote:
> On Thu, Jul 24, 2008 at 09:16:38AM +0200, Sebastien FLAESCH wrote:
>>>> We agree; we're just tripping over terminology.  By "multibyte characters"
>>>> I meant pre-Unicode encodings, such as input to mbtowc(3).  I'm happy to
>>>> support UTF-8.  
> 
>>> FWIW, UTF-8 is a multibyte character set, and is understood by mbtowc() from
>>> glibc.
> 
>> That's true.
> 
>> I think FreeTDS could support all single-byte or multi-byte character sets
>> if:
>>
>> 1) The application (main) calls setlocale() to define the current charset
>> for libc functions like mbtowc, wctomb, mblen (that would include UTF-8
>> support).
> 
>> 2) Any (char*) string manipulation in FreeTDS lib uses mblen() to know the
>> size in bytes of the characters (= no byte-based parsers/lexers).
> 
>> 3) Before sending on TDS protocol, convert from current charset to
>> WideChar (UTF-16/UCS-2) with mbstowcs() and when receiving TDS responses,
>> convert from WideChar to current charset with wcstombs().
> 
> So mbtowc() and wctomb() only convert between widechar and multibyte
> representations *of the same charset*.  That makes them insufficient for
> converting to "the current charset", because you always want your WC
> representation to be UCS-2 but you want your mb representation to be in the
> current locale, which may not be a Unicode locale.

mbtowc(), wctomb(), mbstowcs() and wcstombs() can convert Wide Char to/from
*the current* SB or MB charset defined by a setlocale() call.

 From the man pages of wctomb:

"The behavior of wctomb depends on the LC_CTYPE category of the current locale."

> 
> Hence iconv().

 From my understanding iconv will be used, but if only WC to MB/SB conversion
is concerned, the standard libc functions are sufficient IMHO.

We use that in our VM to convert from the current charset to wchar_t in order
to implement upshift/downshift functions with towupper / towlower (you cannot
use toupper/tolower when current charset is MB ...

> 
>> ...
> 
>>  From my understanding, a C application on UNIX can be written in:
> 
>> A- Pure ASCII (LC_ALL=POSIX)
>> B- A specific single-byte charset (LC_ALL=fr_FR.iso88591)
>> C- A specific multi-byte charset (LC_ALL=ja_JP.eucjp or en_US.utf8)
>> D- Wide Char (UTF-16/UCS-2), using the wchar_t type.
> 
> I don't think you can be very effective writing a C application on Unix
> using a widechar representation internally.  All string manipulation
> functions in the standard library, and in other common libraries built on
> top of them, are implemented using char* as the argument type.

Honestly I have no idea...

I was just guessing there are WC apps out there on UNIX because I know
about some WC libc functions, you have for example fwprintf(), getwchar().

http://blake.ism.u-bordeaux1.fr/doc_link/C/a_doc_lib/aixprggd/genprogc/multi-byte_widechar_subr.htm

Note I found no equivalent for fopen()... probably because UNIX file
systems don't have an explicit charset? (filenames are just a set of bytes,
you can have one file name in UTF-8 and another in ISO-8859-1, AFAIK).

I known also that in ou VM we have to convert to WC in some cases because
functions like isalpha(), toupper() work only for SBCS.

...



> 
>> Maybe FreeTDS could work in wchar_t internally and do charset conversion
>> at the ODBC API level? I think this is how MS ODBC drivers work... no?
> 
> This is how MS ODBC drivers work, because MS is wchar_t-centric.  Unix is
> not.  Using wchar_t internally and having to either convert to char for
> every string manipulation, or reimplement your own wchar-libc, == pain.

Ok I understand.

> 
>> I mean (casts are just there for understanding):
> 
>> SQLPrepare(hp, (char *) "my SQL string in current SBCD or MBCS charset",
>> SQL_NTS);
> 
>> would be mapped to:
> 
>> SQLPrepareW(hp, (wchar_t *) "my SQL string in UCS-2", SQL_NTS);
> 
>> and just use the mbstowcs() standard C lib function to convert current
>> charset to wide char...
> 
>> WARNING: wchar_t on UNIX is 4 bytes while UCS-2 and wchar_t on Windonws
>> is 2 bytes.
> 
> Right, that means you can't even use mbtowc() to do the conversions, because
> your wchar_t internal representation no longer matches the output format of
> *either* of your APIs, so... what's the point in using it as the internal
> representation?

Ok forget it.

Thanks
Seb