Re: Locale Encoding & Filenames

On Apr 9, 2010, at 5:26 PM, Lawrence D'Oliveiro wrote:

> David Huang wrote:
> 
>> The distro-wide policy should be to use the locale...
> 
> That’s fine. That’s why I provided the option to build the code that way,
> if you want it.

The default should be to preserve the previous behavior, and if you think it's useful, there can be a compile-time option to choose your new behavior. I still don't know what problem this is trying to solve though...

> Previously you wrote:
> 
>> Sure, *NTFS* uses Unicode, but the Windows API has two versions of every
>> function that takes a string: the "ANSI" version, which uses the system's
>> locale setting, and the "Wide" version, which uses UTF-16 (not UTF-8). I
>> believe Windows dvdauthor binaries are usually built with mingw32, which
>> I'm not too familiar with, but I get the impression that as with MS Visual
>> C++, the standard library functions, such as open(), take strings encoded
>> with the system's locale.
> 
> Since that’s inconsistent with the behaviour on POSIX systems, I’m pretty sure
> that would be wrong.


I don't know POSIX off the top of my head, but as far as I know, it doesn't say anything about the encoding of open()'s path argument. It does mention that "For a filename to be portable across implementations conforming to IEEE Std 1003.1-2001, it shall consist only of the portable filename character set as defined in Portable Filename Character Set", where the Portable Filename Character set is [A-Za-z0-9._-]. So any national characters in filenames are already outside the realm of portability...

However, the test(1) utility mentions that the LC_CTYPE environment variable is used to "Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments)." http://www.opengroup.org/onlinepubs/000095399/utilities/test.html : there is precedent for interpreting filenames/pathnames based on the encoding specified in LC_CTYPE.

Windows doesn't have a per-user setting for how char* strings are encoded (barring the use of utilities such as AppLocale, which most people haven't heard of); it just has a system-wide setting. While I don't know for certain whether mingw32 behaves this way, I assure you that both the Win32 "ANSI" API (e.g., CreateFileA()) and MS Visual C++'s open() interprets the path using the character set specified by the system locale (i.e., the "Language for non-Unicode programs" setting in the Regional and Language Options control panel).
-- 
Name: Dave Huang         |  Mammal, mammal / their names are called /
INet: khym-+fkPqM/[email protected] |  they raise a paw / the bat, the cat /
FurryMUCK: Dahan         |  dolphin and dog / koala bear and hog -- TMBG
Dahan: Hani G Y+C 34 Y++ L+++ W- C++ T++ A+ E+ S++ V++ F- Q+++ P+ B+ PA+ PL++


------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev