Re: string overhaul

Hi,

> The representation of string objects by the interpreter
> and by the compiler is different at the moment. For
> the interpreter it's an object of builtin class STRING
> and it maintains a pointer to a C-style (null-terminated)
> string, the compiler stores strings in ubyte storages
> (with a redundant \0 terminator to be compatible with C
> functions).

Last year Leon made a start at adding Unicode support, so I would say that
if anything is to be changed at the lower level an eye should be kept on 
internationalization support as a main concern. That is to say, if anything
is changed in terms of strings then internationalization is a concern to be 
dealt with at the same time. It should not be left until later if possible.

> 
> It's ugly because the string data is being copied back
> and forth each time the line of execution crosses the
> boundary between interpreter and compiled code (it seems).
> I would like to unify the representation and use storages
> on both sides. I would introduce a new storage class
> "character", say, which internally is equivalent to
> ubyte (a "wide-character" class could be added later),
> so that strings are still distinguishable from ubyte
> storages. These character storages would be read-only,
> but it's conceivable to allow some string functions to
> work with ubyte storages as well, so that we effectively
> would have mutable and immutable strings.

A suggestion I have made previously is to use the m17n library for strings
(http://www.m17n.org). It is included in major Linux distros. There is also
UIMA from IBM, but it is very large.

In the end, some kind of support or compatibility for external standard string libraries
should be worked towards so that Lush can be more easily used in natural language
processing and other text oriented applications. 

Raymond




__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV