Re: Locale Encoding & Filenames

On Apr 9, 2010, at 4:10 PM, Lawrence D'Oliveiro wrote:
> Previously, filenames in the XML file would be converted to locale encoding.

That sounds like the correct behavior to me...

> Actually, they were always used as is, and that hasn’t changed.

Used as-is is basically equivalent to using the current locale. If my LC_CTYPE is set to en_US.UTF-8, my terminal emulator (xterm or whatever) had better be sending UTF-8 characters if I expect things to work.

> That’s why I provided that build option. The way I see it, there should 
> be a distro-wide policy: either all applications treat filenames as 
> UTF8-encoded, or not. There should not be a mixture where some apps do 
> and some apps don’t. Hence there should be no configuration option to 
> control this; the code should simply be built that way.

The distro-wide policy should be to use the locale... if you force UTF-8 even if the user's locale is set to something non-UTF-8, you're making dvdauthor different from everything else out there.

>> You want someone to rename their files to be compatible with dvdauthor,
>> and incompatible with all the other applications they use?
> 
> What other applications would be doing locale conversions on filenames?

None that I know of... which is why I think it's wrong for dvdauthor to do a locale conversion on filenames. Yes, you say that what you did was *remove* the locale conversion, but if libxml2 did a conversion to UTF-8 behind the scenes, it makes sense for dvdauthor to change it back.

Say I'm Chinese and like to use the "Big5" encoding popular in Taiwan. You're saying that if my XML has a <?xml version="1.0" encoding="big5" ?> in it, and contains a filename with the bytes: a7 da aa ba b9 71 bc 76 2e 6d 70 67 (the Big5 encoding of 我的電影.mpg), libxml2 will give that to dvdauthor as the bytes: e6 88 91 e7 9a 84 e9 9b bb e5 bd b1 2e 6d 70 67 (the UTF-8 encoding of that same string)? And then dvdauthor will try to open that? That certainly seems to violate the principle of least surprise to me.

I guess I'm unclear on what problem you're trying to solve with this... in an earlier email, you mention:

> But this causes problems when you try to interchange data with other 
> people. Consider two different people logging into the same server from 
> remote locations in two different parts of the world, each using locale 
> settings appropriate to their own region, but being unable to even refer 
> to the same files because their filename encodings don’t agree.


However, I don't see that as a problem that it's dvdauthor's job to solve. ls and cat have the same problem. If person A uses iso-8859-2 and has a file "jak_se_máš.mpg" (6a 61 6b 5f 73 65 5f 6d e1 b9 2e 6d 70 67), and person B, who uses UTF-8, logs in to the same machine and simply does an ls on that dir, that filename will be garbled.

Bottom line is if I've chosen a locale, my filenames are going to be encoded using that locale's charset encoding, and any programs I use should honor that... if two people are trying to share the same file, but they don't use the same encoding, they need to take care of that issue themselves--dvdauthor isn't the least of their problems. I disagree with this change.

>> Have you actually tried your change under Windows?
> 
> I have no access to Microsoft Windows.


I've never tried building dvdauthor on Windows (I always find someone else's binary somewhere), but assuming it's not too difficult to do, I suppose I can give it a try, unless someone else volunteers... my suspicion is that filenames with characters that are encoded differently between UTF-8 and the system locale not are not going to work. (Then again, I have no idea if that worked in older versions).

-- 
Name: Dave Huang         |  Mammal, mammal / their names are called /
INet: [email protected] |  they raise a paw / the bat, the cat /
FurryMUCK: Dahan         |  dolphin and dog / koala bear and hog -- TMBG
Dahan: Hani G Y+C 34 Y++ L+++ W- C++ T++ A+ E+ S++ V++ F- Q+++ P+ B+ PA+ PL++


------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Dvdauthor-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dvdauthor-users