Re: string overhaul

On Sun, 4 Mar 2007, Raymond Martin wrote:

> Last year Leon made a start at adding Unicode support, so I would say that
> if anything is to be changed at the lower level an eye should be kept on
> internationalization support as a main concern. That is to say, if anything
> is changed in terms of strings then internationalization is a concern to be
> dealt with at the same time. It should not be left until later if possible.
>
> A suggestion I have made previously is to use the m17n library for strings
> (http://www.m17n.org). It is included in major Linux distros. There is also
> UIMA from IBM, but it is very large.

Thanks, I took a cursory look at it (just to say that I don't really know 
it), looks certainly interesting.

I'm still trying to get a grasp of this whole issue, please correct me
where I'm wrong.

When we read a text file in lush, with 'read-lines' say, then it is 
assumed that the text is in ASCII. ASCII encodes only 128 characters,
and in a way that the "code points" (integers) fit in a C char. That's
why we can go with the conventional C representation of character
strings (arrays of chars).

Both, the interpreter and the compiler use this C string representation,
both also assume/make sure that the character strings are null-terminated.
They only differ in what objects they use as "handles" for C strings (the
interpreter uses objects of builtin type 'string_class' (in header.h),
the compiler also uses a dedicated type but in the code it produces,
storage objects are used as "handles". I'm not quite sure why it's
necessary but unlike array data, string data is being copied when the
interpreter passes a string to a compiled (dh) function, or when a
a compiled function returns a character string to the interpreter. In
my last email I said I would like to get rid of that copying and thought,
in order to do that we'd need to unify the way interpreter and compiler
"handle" character strings. But I'm not even sure anymore that this
is the case.

Anyway, there is another reason you may want to change the 
representation of character strings: In Haskell, for instance, strings
are just lists of type 'Char'. The list datatype is the central datatype
in this language, and there are tons of functions that deal with lists,
just as in Lisp. Now, since strings are just lists you may apply all
the generic list machinery to strings. The situation is  similar in 
Python. In Python many builtin types adhere to a "sequence protocol",
and you can apply a set of generic functions to those kind of objects
(functions for slicing, splicing, iteration, etc). On top of that
there are, of course, many functions just for strings (regex etc.)

In Lush, if strings where just vectors of characters, we could do
many things with strings by using generic array functions and wouldn't
need special functions like 'concat', 'len', 'mid', etc.
Strings would have a constant-time length and we could do without 
the inefficient 'strlen', but would have to find replacements for
some C functions like 'printf'.

Now, when it comes to making Lush fit for other character set encodings
than ASCII this idea of strings as vectors won't work when we want
to use multi-byte encodings. The alternative that I can think of would
be to have three kinds of strings:
* vectors of 8-bit values to represent encoded text where all code
   points fit in 8 bits per character
* vectors of 16-bit values ...
* vectors of 32-bit values ...

Frankly, this doesn't look like a good idea to me. Can anybody think
of a simpler 'strings as vectors' solution that would also be fit for
internationalization?

To enable dealing with arbitrary character encodings the simplest
way forward would probably be to stick with a dedicated character
string type but to switch to the commonly used utf-8 encoding
internally (where we can map every encoding to (?)).
We would need to adapt/rewrite all string functions, I believe, and
find replacements for things like printf and scanf. It sounds like
the m17n library contains such replacements. On the downside, we'd
add another dependency (perhaps we could just incorporate the core
of m17n if we would go that way).


Enough sermon for today.
Cheers,
ralf



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV