Re: string overhaul

Ralf,

On 11 March 2007 21:08:17 you wrote:

> I'm still trying to get a grasp of this whole issue, please correct me
> where I'm wrong.
> 
> When we read a text file in lush, with 'read-lines' say, then it is 
> assumed that the text is in ASCII. ASCII encodes only 128 characters,
> and in a way that the "code points" (integers) fit in a C char. That's
> why we can go with the conventional C representation of character
> strings (arrays of chars).

Depends on locale set in shell. If it is set to something like en_US then
it will be using ASCII. If it is set to UTF-8 then wide characters will be used
internally after conversion.

> 
> Both, the interpreter and the compiler use this C string representation,
> both also assume/make sure that the character strings are null-terminated.
> They only differ in what objects they use as "handles" for C strings (the
> interpreter uses objects of builtin type 'string_class' (in header.h),
> the compiler also uses a dedicated type but in the code it produces,
> storage objects are used as "handles". I'm not quite sure why it's
> necessary but unlike array data, string data is being copied when the
> interpreter passes a string to a compiled (dh) function, or when a
> a compiled function returns a character string to the interpreter. In
> my last email I said I would like to get rid of that copying and thought,
> in order to do that we'd need to unify the way interpreter and compiler
> "handle" character strings. But I'm not even sure anymore that this
> is the case.

I don't know all the details, but there are definitely portability concerns
at play when going to compile strings.

> Anyway, there is another reason you may want to change the 
> representation of character strings: In Haskell, for instance, strings
> are just lists of type 'Char'. The list datatype is the central datatype
> in this language, and there are tons of functions that deal with lists,
> just as in Lisp. Now, since strings are just lists you may apply all
> the generic list machinery to strings. The situation is  similar in 
> Python. In Python many builtin types adhere to a "sequence protocol",
> and you can apply a set of generic functions to those kind of objects
> (functions for slicing, splicing, iteration, etc). On top of that
> there are, of course, many functions just for strings (regex etc.)

Some of that already exists in Lush.

> In Lush, if strings where just vectors of characters, we could do
> many things with strings by using generic array functions and wouldn't
> need special functions like 'concat', 'len', 'mid', etc.
> Strings would have a constant-time length and we could do without 
> the inefficient 'strlen', but would have to find replacements for
> some C functions like 'printf'.
> 
> Now, when it comes to making Lush fit for other character set encodings
> than ASCII this idea of strings as vectors won't work when we want
> to use multi-byte encodings. The alternative that I can think of would
> be to have three kinds of strings:
> * vectors of 8-bit values to represent encoded text where all code
>    points fit in 8 bits per character
> * vectors of 16-bit values ...
> * vectors of 32-bit values ...
> 
> Frankly, this doesn't look like a good idea to me. Can anybody think
> of a simpler 'strings as vectors' solution that would also be fit for
> internationalization?
> 

To unify strings might require a flexible mapping, perhaps something like what
is seen with UTF-8, where characters range between a number of bytes. Specific
character sets could map to certain bytes.

Or maybe one large map and a number of functions to down convert on-the-fly,
truncating as needed.

One way to get an idea how to do this is to look at other languages that already
have internationalization fully implemented (Java, Python, ?).

> To enable dealing with arbitrary character encodings the simplest
> way forward would probably be to stick with a dedicated character
> string type but to switch to the commonly used utf-8 encoding
> internally (where we can map every encoding to (?)).
> We would need to adapt/rewrite all string functions, I believe, and
> find replacements for things like printf and scanf. It sounds like
> the m17n library contains such replacements. On the downside, we'd
> add another dependency (perhaps we could just incorporate the core
> of m17n if we would go that way).

m17n has all its own data types for strings and is designed to replace
all the C string functions. The basic library is small enough, not much of a 
dependency issue (some of its dependencies were added with the w_char 
support: freetype, xft, etc). There is also a database part which is over 1MB. 
Not sure about how necessary it is. That could be added as a Lush package
maybe.
 
Overall, I think Lush should keep its basic string functions and add a few
more for w_char versions or make the basic ones w_char aware. Almost
all popular languages have functions like those in Lush, I'm not sure what
they could be replaced with. Most programmers expect to see these functions.

m17n seems like one way to go. It can be sort of dropped in place to test before
committing to it.


Raymond


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV