Re: Unicode in C--: a proposal

Doug Moen scripsit:

> - Unicode is a big can of worms,
> - The Unicode standard is more complicated than you think,
>   even if you have spent some time reading the documents at unicode.org.
> - Don't assume that other authors who have added unicode support to
>   their programming languages know what they are doing--some of the
>   designs out there seem pretty broken.

I absolutely agree with these points.  If it helps, I am an individual
member of the Unicode Consortium, and I've been around Unicode for
more than five years, and I am the editor of XML 1.1, which updates
XML 1.0 to allow the use of recent Unicode characters in XML markup.

> > Non-ASCII characters
> > are treated by C-- as syntactically equivalent to the ASCII letters,
> > and so may appear freely in names.
> 
> This means that non-ascii control characters, non-ascii white space,
> and even non-ascii newline characters are treated as letters, and
> can appear in identifiers.  Not a good idea.
> 
> It would be better to follow the guidelines in the Unicode standard
> for the syntax of identifiers in programming languages, than to do this.

The trouble then is that you have to track the Unicode Standard and
issue a new version every time Unicode adds new letters or digits, which
happens frequently as new scripts are encoded.  C-- is typically to be
generated rather than written by hand, and I don't see anything wrong
with expecting front ends to do their own enforcement.  Making C--
maximally permissive allows it to be a superset of all front-end
languages now existing or specified in future.

There is a trend, best exemplified by Java, to allow programmers to
use more and more characters in identifiers.  In addition, non-mainstream
programing languages such as Scheme have allowed non-letter-non-digit
identifiers for many years, and the wealth of new math operators in
Unicode can only accelerate this trend.  Permissive treatment of
Unicode characters means that name mangling by front ends can be
restricted to the well-understood ASCII set.

It's certainly true that identifiers with invisible formatting
characters in them are undesirable.  As an assembler, however, C--
can afford to push off considerations of undesirability to the
front ends.

> Since C-- is a compiler back-end, I assume that the set of
> characters legal in an identifier is determined by the need to be
> compatible with existing object file formats, linkers, and external
> libraries that a C-- program might need to link to.

Linkers on *ix usually treat external names as uninterpreted byte
sequences.  I don't know what the Windows linker does.

-- 
Newbies always ask:                             John Cowan
  "Elements or attributes?                      http://www.ccil.org/~cowan
Which will serve me best?"                      http://www.reutershealth.com
  Those who know roar like lions;               [email protected]
  Wise hackers smile like tigers.                   --a tanka, or extended haiku