Re: iconv madness

On Sat, 15 Feb 2014 17:32:53 +0000
Frediano Ziglio <[email protected]> wrote:

> I'm going probably to refactor a bit the iconv stuff.

Hi Freddy, 

Since the iconv mess is mostly my fault, I have a few suggestions in
hindsight.  

There are only two character-set conversion requirements:

1.  On the wire, to convert TDS 7+ SQL text to/from UCS-2.  

2.  In the client, to convert character data to/from the client's
character set for bound buffers (whether dbbind, dbrpcbind or
bcp_bind).  

The policy we adopted was to convert all character data "at the wire".
The read/write functions performed the conversion, and all character
data were kept in the client's preferred encoding.  

That policy was a mistake.  Better not to touch the data unless and
until it's necessary, because that simplifies some things that should
be simple, like binary bindings.  Only when the encoding must be
changed -- for the protocol, or for the client -- should the conversion
take place.  

It's too bad the iconv standard API doesn't include I/O functions,
because for our purposes functions modelled on printf/scanf would be
easier to use.  I suppose that's what tds_convert_stream is for.  

Apart from protocol requirements, character-set conversion is entirely
a client library function.  The user expects characters to be
faithfully rendered on the screen, no matter how they're stored in
database or transmitted on the wire.  In effect, all
*client-provided* character buffers implicitly use the client's
encoding.  That means dbdata() returns the unconverted buffer and 
dbbind() enlists conversion (if need be).  

There is an exception: bcp.  Sybase's bcp utility supports a -j option
that defines the file's encoding, which may be different from the
encoding in the user's environment.  The user may well have a file --
or require a file -- in a "foreign" encoding, say a UCS-2 file
destined for a Windows machine but created on a utf-8 Linux box.  
There is no reason freebcp can't support that, but the current design
makes it more difficult than it should be.  

It would also be nice if, as a byproduct of your work, freebcp could
read from a pipe.  I created the misbegotten requirement that the
stream be seekable to avoid memory pressure.  With today's larger
memories, freebcp should just realloc as needed to hold a column
buffer.  

HTH & regards, 

--jkl