Re: UTF-8/unicode support

Hi Leon,

On Tuesday 21 February 2006 01:03, Leon Bottou wrote:
> On Monday 20 February 2006 02:50 pm, Raymond Martin wrote:
> > You mentioned that a number of string functions are needed for Unicode (UTF-8), 
> > this is still the case I suspect. Are you able to enumerate which operations are 
> > needed so I can look into it?
> 
> They are all in
> - string.c (for the interpreter) 
> - lsh/compiler/dh-macro.lsh (for the compiler)
> 

Okay.

> The main problem is that there are lush programs
> that use strings as arrays of bytes and assume
> a latin1 representation.  The worst offender is 
> Yann's JSS software (http://jssindex.sf.net).
> 
> Changing these functions breaks such 
> software unless one uses the latin1 locale.
> But after all Yann could add a 
>   LC_CTYPE="C" ; export LC_CTYPE 
> before calling lush.
> 

It is actually the responsibility of the program to specify and use the locale
it needs, assumptions about the environment are not good. So if Lush is doing
things right with strings (recognizing UTF-8 or ASCII subset) it is other programs
that are not up to standard and need to change.

> Among the functions in string.c,
> the most difficult change happens in
> the regular expressions.
> Handling "." is not too hard.
> Handling ranges "[..]" requires a 
> change of the bytecode.
> But we might simply specify that ranges
> are only defined on ascii characters.
> We could also define predefined ranges 
> such as [:uppercase:] [:lowercase:] 
> and use the wide character classification functions
> to handle this.
> 

> So if you want to give it a try ...

I see. Different alphabets will have different ranges and these need to be taken
care of on-the-fly when parsing for regular expressions. I'm working on another
project that does multilingual text processing (with UTF-8) so I can look around
in there for how the search/matching algorithm is implemented.

> 
> The Libs documentation tells a lot about the wide character functions.
> Just type "info libc" and check 'Character Handling' and 'Character Set Handling'.
> 

I'll read up.

Thanks.

Raymond


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642