Re: Unicode -- issues and recommendations

On Tue, 2005-09-06 at 13:41 +0100, [email protected] wrote:
> Jonathan S. Shapiro wrote:
> > Character Size:
> > 
> > I don't think that C-- can take a position on character size.
> 
> I don't see why we would want it to. C-- defines types sufficient
> to represent Unicode code units (bits8 for UTF-8, bits16 for UTF-16,
> bits32 for UTF-32) and code points (bits32).

Up to this point, I agree. In fact, I said so!

> Strings are a higher-level
> issue that should be left to front-ends and libraries....

When we get to complex representations, I agree. If we are talking about
code point vectors whose code points are of like representation, I do
not.

While it is feasible for the front end to encode characters using
ordinal byte values, it is not feasible for humans to validate that
output. Decoding the utf-8 (or whichever) encoding can be done in the
C-- lexer without any knowledge of specific languages, and there is no
more difficulty in range checking the code points than there is in range
checking integer literals.

In all honesty, I wouldn't find my own argument compelling were it not
for the fact that one wants to support Unicode identifiers in any case.
The current C-- specification, for example, cannot support Java
identifiers unless they are mangled to ASCII by the front end. The
post-mangled names are not link-compatible with existing code.

Speaking as someone who has been looking at a lot of such mangled
identifiers lately (BitC permits some characters in identifiers that C
does not), let me assure you that reading through the mangle is
incredibly irritating and bug prone.

If the lexer is going to support unicode identifiers, then it needs to
decode UTF-8 anyway. At that point, it seems sensible to reuse the code
for string literals.

> I would not add any additional character literals; it is sufficient to
> use integers.

That is exceptionally North American of you. :-) Especially so when one
stops to think just how little support is required for quoted UTF-8
character literals:

  Lexer must use a 32-bit quantity to report the code point to
  the parser.

  Lexer must know the UTf-8 encoding scheme, but NOT any details
  about unicode characters beyond what it already knows.

  Range check that is already being used for integers must now also
  be applied when the character literal is type qualified.

If this is all it takes to do Unicode character literals, then I have to
say that NOT doing them is pretty silly. The "readable in editor"
argument alone is sufficient to justify this level of effort.

In practical terms, the level of effort required to support unicode
identifiers is about the same. The only added complexity is the need to
pick up the character class tables from somewhere (which is why I
pointed to ICU). I believe they are available in machine-readable form
at www.unicode.org.


shap