Re: Re: character sets and filenames

On Wed, 2004-03-10 at 01:40, Robert Hart wrote:
> The same problem exists with ID3 tags, and is solved in the following
> way:
> 
> if a tag is valid UTF-8 assume it *IS* UTF-8.
>   This covers the case of:
> 	- UTF-8 (well, duh)
> 	- ASCII (which is a subset of UTF-8)
> 
> else assume the tag is ISO-8859-1 (or some other user-specified charset)
>   This covers the case of:
> 	- ISO-8859-1 (most likely non-UTF8 charset)
> 
> The cases not covered are:
> - Tag is not UTF-8 but happens to be valid.
> - Tag is not ISO-8859-1.
> 
> There is nothing we can do about this, because - like the filesystem -
> the tag is charset agnostic, and some kind of 'specification' was only
> added to the spec at a later date as an afterthought.
> 
> See .zinf/preferences : 
> DefaultID3Charset: ISO-8859-1
> 
> and plm/metadata/id3lib/id3lib.cpp

Great, I'll give this a go.   Seems like we are going to keep
facing this issue for a while.. Maybe this conversion code should 
be centralized and placed in base/src/utility.cpp   so it can 
be used by the FileToURL and URLtoFileRoutine
or even in the utf8 library

> How do your files show up in a file-selector?

Hmm.  they don't even show up.

Presumably they are not valid in the current encoding
and therefore fail to convert to UTF8 for display.  
This is pretty unfortunate.  
> 
> There was some discussion about autodetecting the character set based on
> letter frequency distributions and the like. Apparently there is a
> Russian version of xmms that does this (because there are two fairly
> common Russian character sets) but that seems like more work than the
> benefit.

Neat idea, but I guess all of this is temporary until the rippers
write UTF-8 strings for metadata, and filenames.


> On Wed, 2004-03-10 at 02:10, Sam Varshavchik wrote:
> > Kristian G. Kvilekval writes:
> > 
> > > On Tue, 2004-03-09 at 17:49, Sam Varshavchik wrote:
> > > 
> > >> The user used an encoding other than the "encoding used by the operating 
> > >> system."
> > >> 
> > >> By fiat, Unix filenames have been decreed to be using UTF-8; still plenty of 
> > >> stuff creates files using iso-8859-1, et al…
> > >> 
> > >> There is no easy solution for this one.
> > > 
> > > How about a tool to guess the encoding and re-write the filenames
> > > as UTF8.  Would this sufficient?
> > 
> > It's the lesser of all evils.  I think the following heuristics are probably 
> > the best one can do:
> > 
> > A)  Try to see if the filename parses in UTF-8.  Stuff that's not UTF-8 is 
> > very unlikely to be parsable as valid UTF-8.
> > 
> > B)  Try nl_langinfo(CODESET).  Not all platforms have that call.  You'll 
> > need to probe if this is implemented, in configure.ac.
> > 
> > C)  ????
-- 
email:[email protected] office:(805)893-4276 http://www.cs.ucsb.edu/~kris



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click