Re: PROBLEM with word iterator

I want to point out that the UTC did change the status of '.' in particular;
that change will be showing up in Unicode 4.1.

‎Mark

----- Original Message ----- 
From: "Andy Heninger" <[email protected]>
To: "Ruslan Zasukhin" <[email protected]>
Cc: <[email protected]>
Sent: Wednesday, November 10, 2004 09:43
Subject: Re: PROBLEM with word iterator


> Ruslan Zasukhin asks:
>  > In other words it [ICU's word break] consider "aa.dd" as single word.
>  > Why is this ???
>
> Because the Unicode Consortium says so :-)
>
> ICU uses the boundary rules defined in the Unicode Text Boundaries
> document, http://www.unicode.org/reports/tr29/#Word_Boundaries
>
> This document defines three classes of punctuation characters that do
> not cause breaks in words or numbers - MidLetter, MidNumLet, and MidNum.
>    '.' is in the MidNumLet class, meaning that it does not cause a break
> when it appears in the interior of either a letter or number.
>
> The intent is not to put breaks within things like 3.14159
>
>  > The same is for any punctuation characater  , / = - + *
>
> This shouldn't be.  Only some punctuation falls into the midLetter /
> Number category.
>
> Another thing that people often find surprising with the Unicode word
> rules is that runs of punctuation outside of words do group together.
> Each punctuation character stands alone as its own "word".
>
> -- 
>       Andy Heninger
>       [email protected]
>
>
> Ruslan Zasukhin wrote:
> > Hi All,
> >
> > I have made word iterator
> >
> >     UErrorCode status = U_ZERO_ERROR;
> >     mpIter = BreakIterator::createWordInstance( *pLocale, status );
> >
> >
> >
> > And I do e.g. Counting of words:
> >
> > ulong Index_String::CountWords( const UChar* inWords )
> > {
> >     ulong res = 0;
> >
> >     mpIter->setText( inWords );
> >
> >     long start = mpIter->first();
> >     if( start == BreakIterator::DONE )
> >         return res;
> >
> >     while( true )
> >     {
> >         long next = mpIter->next();
> >
> >         if( next == BreakIterator::DONE )
> >             return res;
> >
> >         if( u_isalnum(inWords[start]) )
> >             ++res;
> >
> >         start = next;
> >     }
> >
> >     // never reached.
> >     return 0;
> > }
> >
> >
> > --------------------------------
> > PROBLEM IS:
> >
> > If I send string "aa.dd bbb"
> >
> > Then BreakIterator's next() return me the boundaries  0 and 5
> >
> > In other words it consider "aa.dd" as single word.
> >
> > Why is this ???
> >
> > The same is for any punctuation characater  , / = - + *
> >
> > I use ICU 3.0 btw.
> >
> >
> >
>
>
> _______________________________________________
> icu mailing list
> [email protected]
> http://oss.software.ibm.com/developerworks/oss/mailman/listinfo/icu
>