Re: General ICU BreakIterator usage model - relevant locale IDs

> From:
> "Elisha Berns" <[email protected]>
> ...
> One of my main discontents concerns the creation of the BreakIterators:
> You need a Locale object to instantiate a BreakIterator even though in
> several cases the Locale object has no bearing whatsoever on the
> creation nor the performance of the BreakIterator.  Since I cache any
> BreakIterators I create, depending on whether a new or different Locale
> is required, I can easily create unnecessary iterators.  The issue
> perhaps has more to do with the primitiveness of the Locale class in
> that you don't know if you need a new iterator object or not when you
> have 2 different Locales.  So it would be an optimization from my point
> of view to have some mechanism that tests whether indeed you actually
> need to create a new iterator. ...

The Locale class is just a container for a locale ID. It does not know anything about which service 
supports which locale IDs with distinct functionality.

However, your point is valid. We have plans to make two improvements:

1. Provide a more precise list of locale IDs that are distinctly supported per service. This will be 
done with a function like BreakIterator::getAvailable(). Currently, all services report that all 
locale IDs are supported that simply have ICU resource bundles behind them. The improvement will be 
to truly return the list of locale IDs that are relevant.

This will probably require a reorganization of our internal data into multiple "trees" of data 
files, one "tree" per service.

2. Provide a mechanism and data for break iteration for more locales.

We also plan to do the opposite of 2.: To make the Thai word/line break behavior part of the 
default, removing the special handling currently associated only with the Thai language.

> But if one of the languages is
> Turkish, then you do need 2 different iterators, at least that's my
> impression.  But there is no way of knowing that from any ICU API, it's
> just wisdom gained from experience with Unicode.

Currently, ICU only has one single word BreakIterator implementing the default Unicode word breaks. 
In the future, 1. above will tell you that, and 2. may add language-specific tailorings.

String case mappings - other than finding the titlecase positions - depends on the language, but is 
ultimately controlled by Unicode's SpecialCasing.txt. The locale-sensitive and context-sensitive 
cases are hardcoded, so ICU could tell you which locale IDs are relevant, but it is possible to look 
it up in the Unicode data file as well. Fortunately, that file is quite stable now.

Best regards,
markus