Re: Re: Strings again

Mark Hahn wrote:
> I'll be honest.  I know nothing about locale and it's issues.  Can
> anyone get me started?  Any urls for what Python does about locales?

[Apologies for the delay in responding.  I had to move house and wait
for the phone company to hook me up again]

There are several locale related issues.  The one most people
are aware of is formatting.  For example some locales use
dots as thousand seperators and commas as the decimal point
in numbers.  Another is dealing with dates where the US
uniquely uses M/D/Y and various other locales use the
other (more logical :-) orderings.

Usually that sort of thing is handled by the system library.
On startup you do a setlocale or similarly named call, and
pick up the defaults from the system or the environment.
Calls to functions like scanf, printf and strftime then
all do the right thing.

The other major locale issue is to do with string formatting.
For example doing a string sort is different based on the locale.
In particular think about accented characters such as the many
variations of the letter 'e' with marks in various directions
around it.  If you had a whole bunch of words that all started
with those, the sort ordering is dependent on the locale.

For example, in some locales è comes before é and in others
it is the other way round.

Not remembering the locale can also destroy information.
For example the German ß character is similar to 'ss'.
If you made the string uppercase and then lower case
again, the locale information would be useful to converting
the upper case characters into lower case again and deciding
whether or not to use it.

The other area of issue is in the oriental character sets.
Unicode lumped together characters that looked similar
from Chinese, Japanese and Korean, but they aren't the same
character.

If you cast your mind back 20 years you will remember how
most typewriters printed '1' (the digit one) and 'l' (lower
case L) identically.  Imagine some character set committee
decided they were the same thing since they were rendered
identically.  Imagine how annoyed you would get now :-)
If you remember the locale of the character then you won't
do harm since you will know which nation it originally came
from.

The locale stuff gets even more complicated since they can
be combined.  Not only do you have to distinguish between
French in France, but also French in Canada.  Then you get
nice hybrids such as an American living in Japan.  (Which
date format do you think they want and which language do
they want messages in?)

Then you have to look at the scope of the locale information.
IIRC Win32 lets it be different per thread.  Imagine a web
server sending stuff to clients all over the world to see
why this is necessary.  UNIX is per process last time I
used (to great frustration when a server process is in a
different locale than the client you are talking to :-)

Roger