icu api proposal: apis for normalization data

This is a proposal to externalize via new APIs some normalization data that ICU uses internally and 
that is shaping up to be useful for ICU users as well. All of these kinds of data more or less 
relate to segmenting text into adjacent pieces that do not affect each other in normalization 
operations.

These properties are relatively easily derived from standard properties, but the derivation in some 
cases requires looking at all Unicode characters. It is more efficient to have them directly, and 
allows writing UnicodeSet patterns that then get used in Transforms, break iteration, regex, etc.

* Expiration: Thu, 2004-apr-22

I will only list proposed short and long property names like Unicode's PropertyAliases.txt file. 
These properties will be available via the usual functions [u_]hasBinaryProperty() and 
[u_]getIntPropertyValue() in both ICU4C and ICU4J, with the UProperty enum names following the long 
property names below, as well as via the property name functions and UnicodeSet.

* The following are already in the current CVS snapshot (we needed these for other work):

# lccc(c)=ccc(NFD(c)[0])
# tccc(c)=ccc(NFD(c)[last])
lccc; Lead_Canonical_Combining_Class
tccc; Trail_Canonical_Combining_Class

In other words, these are the combining classes of the leading and trailing code points of c's 
canonical decomposition. They are used in "FCD checks" (see UTN #5) which means that they are used 
to check if a string would need canonical reordering. For each of these two properties, the values 
are 8-bit combining class values and their common string names.

* NF*Skippable

A character c is skippable in a normalization form if it appears at or near the end of a string and 
there is nothing that can be appended to the string that affects the normalization form of c. This 
is different for each normalization form, therefore there are 4 boolean properties here. For details 
of how this is computed, see unormimp.h (search for "NF* Skippable").

These are used in our Transliterator framework to find out which part of a buffer is safe to be 
normalized during incremental normalization.

nfdskip; NFD_Skippable
nfkdskip; NFKD_Skippable
nfcskip; NFC_Skippable
nfkcskip; NFKC_Skippable

Feedback on names welcome. We might want to talk about "safe trailing character" or something 
similar. The names above are simply what we used internally for years.

* Last (for this time), we have a boolean property for whether a character is a starter for 
canonical closure. ICU has a class that enumerates canonically equivalent strings for some input 
string, and the first steps are to NFD the input string and then segment it into pieces, each of 
which gets its closure set of strings computed. (This is used for canonical closure of collation 
tables.) The boolean property indicates the first characters of each segment.

I am struggling with a good name here. Internally we talk about "safe starter for canonical closure" 
which I think is not saying enough. (Tell me if you disagree.)

The characters with this property are starters of non-overlapping segments of text relevant for 
canonical equivalence -> how about

eqsegstart; Equivalence_Segment_Starter

This single boolean property is internally (by gennorm) computed as the _opposite_ of "not safe 
starters", which are the union of (ccc!=0) and (occurs in non-initial position of the canonical 
decomposition of any character [like " in NFD(ä) and a Jamo T in an NFD(Hangul LVT)]).


Best regards,
markus