Unicode support (Re: macho 0.4 released)

On Mon, Nov 10, 2003 at 03:15:08PM -0800, Miles Egan wrote:
> So what is the status of Unicode support in CL?  Is it purely up to the
> implementation?  The string handling functions seems to be abstracted
> over character type but I don't see anything in the hyperspec explicitly
> about Unicode.

There are basically two approaches:

* use plain octet strings containing UTF-8 encoded character streams.

That does not need any support from the language for simple I/O, but
you have to be careful when doing operations on the text because a single
character may consist of multiple octets.

Code will get the wrong result if it tries to calculate display widths
via string lengths, and pattern matches or searches based on octet
sequences may get false positive hits if they don't respect the
boundaries of multi-octet characters. But if you're essentially just
using strings as octet streams without caring about the characters,
that's not a problem.

* use wide characters internally, and convert input and output.

That's the more robust solution, but it has the disadvantage of wasting
memory (16 bits aren't enough, so you'll be using 32 bits per character),
and it's also ugly if your language doesn't support it directly. You'd
usually expand whatever strings you read into an array of wide
characters, and (usually) convert to the more compact UTF-8 encoding on
output.

Lisp being Lisp, wide character Unicode support should be possible to add
as a library without any core language changes, but that's way beyond my
current Lisp skills...

Sorting strings properly is difficult in either case, because the rules
for "alphabetical order" get really weird depending on your locale. Let's
not get into that...

I think the first approach would be easier to implement for Macho but
that's not well researched, in fact I haven't even looked at the code yet
:-)

-Klaus