RE: utf8 in QB

Hi Taras, 

> Does that mean that when typing queries into the edit pane, 
> the characters are stored somewhere in memory in UTF-16?

Yes, on Windows Unicode strings are usually stored in UTF-16 format in
memory. All Unicode APIs use UTF-16 (e.g. for text output). The key point
behind this is that UTF-16 is a format that has a constant size per
character (2 bytes, if you don't count surrogates) which makes processing it
very effective. For transport, however, UTF-8 is better suited, for (at
least) two reasons: 1) it avoids redundancies for scripts which require only
one byte (where the upper byte would always be 0, e.g. standard latin
alphabet) and 2) it is byte order neutral and can be handled like a stream
of single bytes, allowing so its transport over 8bit channels.
 
> character_set_client: the encoding the client is using.
> 
> The last one is the one I'm having confusion with. The way I 
> understand this variable's role is that if a file for 
> example, saved in UCS-2, was piped into the mysql command 
> line client, this variable would have to be set to indicate 
> the encoding to mysql. If the connection variable was set as 
> UTF-8, then a conversion from UCS-2 to UTF-8 would have to 
> take place before sending the command to mysql.

The client is the MySQL client library (the DLL on Windows) not the
application. Communication with the client library is done with UTF-8
encoding. QB and the underlying shared libraries for our tools convert back
and forth where necessary. This happens transparently and does not affect
the actual server communication. When QB starts a session it immediately
sets the charset to UTF-8 (SET NAMES) and from this point on expects and
sends data exclusively in UTF-8.
 
> So the way I view (maybe incorrectly) QB is that the data 
> displayed in the edit pane has an encoding, and this encoding 
> is indicated by character_set_client. The client here is 
> query browser. When I press ctrl-enter this data is 
> transcoded to character_set_connection (if required), and 
> sent to the DB.

No, the text in the UI is UTF-16 (on Windows, this differs to other
platforms) and is converted to UTF-8 before it is given to the client
library (e.g. mysql_query). The library then handles the rest. Since
everything is set to UTF-8 no further conversion is needed, *except* for the
actual storage on the server. For instance, if your table has a varchar
column with Big5 as encoding then the server takes care to convert the
incoming UTF-8 to Big5 (and vice versa when reading out). Everything else is
still UTF-8. That is the base idea behind using UTF-8 all the way to the
server. We have full Unicode support in the tools, without caring about the
actual encodings. We have picked UTF-8 over UCS-2 (which is the ISO version
of UTF-16) for mainly two reasons: 1) UCS-2 support in the server is not as
complete as for UTF-8 (at least for time when we did the decision) and 2)
all of the used support libs (libxml, glib etc.) work with UTF-8 (and on
Linux Unicode support means UTF-8).

Hope this better clarifies things for you.

Mike
--
Mike Lischke, Software Engineer GUI
MySQL AB, www.mysql.com


-- 
MySQL GUI Tools Mailing List
For list archives: http://lists.mysql.com/gui-tools
To unsubscribe:    http://lists.mysql.com/[email protected]