Re: display undeclared iso-charset in UTF environment

Hi Alain,

* Alain Bench on Friday, September 08, 2006 at 02:55:28 +0200:
>  On Thursday, September 7, 2006 at 16:59:43 +0200, Christian Ebert wrote:
>> %% charset that will be fixed if not declared
>> private variable fix_charset = "ISO-8859-15";
> 
>     Assuming that unlabelled articles are in Latin-9 seems not a good
> choice. Experience shows that, for westerners, the vast majority of such
> articles really are in CP-1252 or Latin-1. A small minority is in UTF-8.
> And peanuts for all other charsets, including Latin-9.

But I want to see Euros, loads of 'em! -- Ok, you're right, as
always. I just didn't think of cp-1252, and had no trouble up to
now.

>     So, for westerners, the best choice is probably either a fixed
> CP-1252, or some auto-sensing between UTF-8 and CP-1252 (I mean a
> Vim-like fileencodings="ucs-bom:utf-8:latin1").
> 
>     Then there are other populations, and a Chinese or Polish guy
> probably prefers some other default charset (or auto-sensing list).
> Should be configurable.

I agree, and noted that already in my comment.

>> private variable hibits = "\\C[ßäöüáéíàèçôœëñåøµ©®¢£€]";
> 
>     I don't understand the interest of grepping specifically those
> chars.

Those were the "detection" chars that came to my mind/need.  And
I have no knowledge of charsets. My motivation was just: I want
to read these articles.

> Why not just convert all unlabelled articles?

No need to touch those in us-ascii. And there are actually quite
a lot of them (from my very subjective impressions).

> Or grep for any char above 0x80?

Much better. But how do I do that in s-lang?

>> % check whether we are in an UTF environment
>> private variable utf_env = getenv ("LANG");
>> if (NULL == utf_env)
>>   utf_env = getenv ("LC_CTYPE");
>> if (NULL != utf_env and 0 == string_match (utf_env, "\\Cutf-8", 1))
>>   utf_env = NULL;
> 
>     Is there no better way to get current CODESET then parsing env vars?

I'm sure there is. Only I don't know it. For instance, on MacOS I
can't do

$ locale

> It seems to me you get parsing reversed: The active value is LC_ALL if
> defined, otherwise LC_CTYPE, otherwise LANG.

Thanks for clearing this up.

I've renamed utf_env to lc_env and the check looks like so now:

private variable lc_env;
foreach lc_env (["LC_ALL", "LC_CTYPE", "LANG"])
{
  lc_env = getenv (lc_env);
  if (NULL != lc_env)
  {
    if (0 == string_match (lc_env, "\\Cutf-8", 1))
      lc_env = NULL;
    break;
  }
}

> Finally, once you have the current CODESET, why limit the
> fixing to UTF-8? It could be also usefull in other locales.

I didn't have problems reading with LANG=en_US.ISO-8859-15. The
need to translate just arose with LANG=en_US.UTF-8.

>> iconv = sprintf ("/sw/bin/iconv -c -f %s -t UTF-8", fix_charset);
> 
>| $ /sw/bin/iconv
>| bash: /sw/bin/iconv: No such file or directory

Oops; was to busy censoring even more embarrassing stuff ...

>     BTW you could simply remove "-t UTF-8" so iconv itself converts to
> the current locale's charset.

Right.

> Bye!	Alain.

Thanks for you comments, Alain.

> -- 
> 	And now, ladies and gentlemen, Silly Footer announces:

heh, I have

/Using Tomcat but need to do more?/,/http:\/\/sel\.as-us\.falkag\.net\// d

in my sourceforge display_filter ;)

c
-- 
_B A U S T E L L E N_ lesen! --->> <http://www.blacktrash.org/baustellen.html>

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642