Re: PROBLEM with word iterator

Ruslan Zasukhin asks:
 > In other words it [ICU's word break] consider "aa.dd" as single word.
 > Why is this ???

Because the Unicode Consortium says so :-)

ICU uses the boundary rules defined in the Unicode Text Boundaries 
document, http://www.unicode.org/reports/tr29/#Word_Boundaries

This document defines three classes of punctuation characters that do 
not cause breaks in words or numbers - MidLetter, MidNumLet, and MidNum. 
   '.' is in the MidNumLet class, meaning that it does not cause a break 
when it appears in the interior of either a letter or number.

The intent is not to put breaks within things like 3.14159

 > The same is for any punctuation characater  , / = - + *

This shouldn't be.  Only some punctuation falls into the midLetter / 
Number category.

Another thing that people often find surprising with the Unicode word 
rules is that runs of punctuation outside of words do group together. 
Each punctuation character stands alone as its own "word".

-- 
      Andy Heninger
      [email protected]


Ruslan Zasukhin wrote:
> Hi All,
> 
> I have made word iterator
> 
>     UErrorCode status = U_ZERO_ERROR;
>     mpIter = BreakIterator::createWordInstance( *pLocale, status );
> 
> 
> 
> And I do e.g. Counting of words:
> 
> ulong Index_String::CountWords( const UChar* inWords )
> {
>     ulong res = 0;
> 
>     mpIter->setText( inWords );
> 
>     long start = mpIter->first();
>     if( start == BreakIterator::DONE )
>         return res;
>     
>     while( true ) 
>     {
>         long next = mpIter->next();
>         
>         if( next == BreakIterator::DONE )
>             return res;
>         
>         if( u_isalnum(inWords[start]) )
>             ++res;
> 
>         start = next;
>     }
> 
>     // never reached.
>     return 0;    
> }
> 
> 
> --------------------------------
> PROBLEM IS:
> 
> If I send string "aa.dd bbb"
> 
> Then BreakIterator's next() return me the boundaries  0 and 5
> 
> In other words it consider "aa.dd" as single word.
> 
> Why is this ???
> 
> The same is for any punctuation characater  , / = - + *
> 
> I use ICU 3.0 btw.
> 
> 
>