Re: UTF-8/Unicode support

Hi,

> Another option is to use the facilities of the c library
> without reference to unicode. Basically it means that
> you get as good i18n support as the os offers.
> By default one writes text files using the locale encoding, whatever it is.
> For portability reasons, one might also want to write text files in UTF8,
> but this problem is slowly disappearing because everybody 
> is slowly transitionning to UTF-8 locales anyway.
> 

Right.

> So I believe that we only would have to write a small string
> library that does the basic string operations using
> the wide character interface from the c library.
> There are some little performance issues (counting character positions)
> that might benefit from a reasonable caching scheme.
> 

I can now get UTF-8 in a console (e.g. Japanese), but Lush cannot handle this
as a string (always vomits on the out of range UTF-8 bytes 0x8x and above).

So even entering "ユーザーマニュアル" straight at the toplevel results in an error
regardless of the consoles ability to display the charmap.

? "ユーザーマニュアル"

*** read : illegal character : 0x83


Check this UTF-8/unicode FAQ for more info: http://www.cl.cam.ac.uk/~mgk25/unicode.html

There is a section entitled 'C support for Unicode and UTF-8' with an example C program
to test UTF-8. I ran that from the same console Lush runs from (same locale, charset, etc)
and it prints out UTF-8 mapped to the correct characters.

The printout results from

#include <stdio.h>
#include <locale.h>

int main()
{
  if( !setlocale( LC_CTYPE, "" ) ) 
  {
    fprintf( stderr, "Can't set the specified locale! "
             "Check LANG, LC_CTYPE, LC_ALL.\n" );

    return( 1 );
  }

  printf( "%ls\n", L"Schöne Grüße" );

  return( 0 );
}

giving on stdout "Schöne Grüße" as expected. Then in Lush:

? (printf "%ls\n" L"Schöne Grüße")

*** read : illegal character : 0x9f

The C test automatically uses multibyte conversion to handle UTF-8 forward and 
backwards based on having the locale set/detected as using UTF-8.

So the upfront problem might have to do with locale setup/detection so that UTF-8 
strings are automatically processed. Either manipulating the locale in Lush when 
running or compiling from a properly set locale might work. This would take things
at least part way to fixed if it works.

I'm not sure where in Lush I can determine the locale it is using in order to the 
verify this, but I haven't seen setlocale() anywhere yet. 

What do you think?

Cheers.

Raymond

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642