Re: FreeTDS and UTF-8 support

Steve Langasek wrote:
> On Wed, Jul 23, 2008 at 01:59:54PM -0400, James K. Lowden wrote:
>> Sebastien FLAESCH wrote:
>>>> I see no reason support multibyte characters.  I have no problem
>>>> converting to UCS-2, UTF-8, etc.  But I see no value in using
>>>> pre-iconv functions or supporting weird encodings.  Unicode is making
>>>> them obsolete.
>> ...
>>> So the idea is to make FreeTDS support multi-byte characters sets, even
>>> if it's just UTF-8:
> 
>> Hi Sebastien, 
> 
>> We agree; we're just tripping over terminology.  By "multibyte characters"
>> I meant pre-Unicode encodings, such as input to mbtowc(3).  I'm happy to
>> support UTF-8.  
> 
> FWIW, UTF-8 is a multibyte character set, and is understood by mbtowc() from
> glibc.
> 

That's true.

I think FreeTDS could support all single-byte or multi-byte character sets if:

1) The application (main) calls setlocale() to define the current charset for
libc functions like mbtowc, wctomb, mblen (that would include UTF-8 support).

2) Any (char*) string manipulation in FreeTDS lib uses mblen() to know the size
in bytes of the characters (= no byte-based parsers/lexers).

3) Before sending on TDS protocol, convert from current charset to WideChar
(UTF-16/UCS-2) with mbstowcs() and when receiving TDS responses, convert from
WideChar to current charset with wcstombs().

...

 From my understanding, a C application on UNIX can be written in:

A- Pure ASCII (LC_ALL=POSIX)
B- A specific single-byte charset (LC_ALL=fr_FR.iso88591)
C- A specific multi-byte charset (LC_ALL=ja_JP.eucjp or en_US.utf8)
D- Wide Char (UTF-16/UCS-2), using the wchar_t type.

Ideally, FreeTDS should allow all of that... no?

In cases A, B, C, FreeTDS can determine the current charset with nl_langinfo().

Maybe FreeTDS could work in wchar_t internally and do charset conversion
at the ODBC API level? I think this is how MS ODBC drivers work... no?

I mean (casts are just there for understanding):

SQLPrepare(hp, (char *) "my SQL string in current SBCD or MBCS charset", SQL_NTS);

would be mapped to:

SQLPrepareW(hp, (wchar_t *) "my SQL string in UCS-2", SQL_NTS);

and just use the mbstowcs() standard C lib function to convert current charset
to wide char...

WARNING: wchar_t on UNIX is 4 bytes while UCS-2 and wchar_t on Windonws is 2 bytes.

?

Cheers,
Seb