Re: UTF-8/unicode support

On Monday 20 February 2006 02:50 pm, Raymond Martin wrote:
> You mentioned that a number of string functions are needed for Unicode (UTF-8), 
> this is still the case I suspect. Are you able to enumerate which operations are 
> needed so I can look into it?

They are all in
- string.c (for the interpreter) 
- lsh/compiler/dh-macro.lsh (for the compiler)

The main problem is that there are lush programs
that use strings as arrays of bytes and assume
a latin1 representation.  The worst offender is 
Yann's JSS software (http://jssindex.sf.net).

Changing these functions breaks such 
software unless one uses the latin1 locale.
But after all Yann could add a 
  LC_CTYPE="C" ; export LC_CTYPE 
before calling lush.

Among the functions in string.c,
the most difficult change happens in
the regular expressions.
Handling "." is not too hard.
Handling ranges "[..]" requires a 
change of the bytecode.
But we might simply specify that ranges
are only defined on ascii characters.
We could also define predefined ranges 
such as [:uppercase:] [:lowercase:] 
and use the wide character classification functions
to handle this.

So if you want to give it a try ...

The Libs documentation tells a lot about the wide character functions.
Just type "info libc" and check 'Character Handling' and 'Character Set Handling'.

- L.


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642