Re: AW: PyDS RPC fix

Hi!

Bauer, Georg wrote:
> You can't store unicode strings - to store them you will allways have to
> encode them somehow. And that encoding is either iso-8859-1 if you don't
> need unicode strings or utf-8. There are other unicode encodings (utf-16 and
> utf-32 and some more obscure onese), but those aren't supported in most
> librarties. utf-8 is the best choice there is. Unicode strings in Python can
> only exist during runtime. There is no way to store them directly without
> encoding them (even pickling will do some encoding first).

Of course,
Unicode cannot be treated in the library where usual I/O is attended as
you say, and I think that it is desirable that it is converted into utf-8.

However, the operation with such utf-8 should be concealed and be
abstracted. It is made not to appear in a usual program code, and I
think that it is the best method of do store as Unicode making confusion
a minimum in the variable in the application code.

Phillip Pearson wrote:
> Do you know what utf- format Python uses internally?  I think Windows
> NT uses utf-16, but I'm not an expert here.  It would always be
> possible to switch to utf-16 instead of utf-8, but that would be a
> pain, and it would be harder to manually look at databases. 

Python use UCS-2 or UCS-4 as unicode for internally.
(not UTF-8, see PEP100, PEP261)
This depends on compile time configuration of python.

UCS-2 and UCS-4 are fixed size character string.
UCS-2 : 2byte (0 - 0xffff)   ; same as ISO/IEC 10646 BMP
UCS-4 : 4byte (0 - 0x10ffff) ; full set of 10646

utf-7, utf-8 and utf-16 are represent as variable-length sequences
And
utf-8 : covers all UCS-4 set
utf-16 : covers UCS-2 and part of UCS-4 (not full set)

bye, tomo.