Re: Word boundaries

On Mon, Mar 26, 2012 at 12:57 PM, Lars D=C9=AA=E1=B4=87=E1=B4=84=E1=B4=8B=
=E1=B4=8F=E1=B4=A1 =E8=BF=AA=E6=8B=89=E6=96=AF <[email protected]> wrote:
> Let the regex engine help you advance the character counter.
>
> =C2=A0 =C2=A0$ cat langs
> =C2=A0 =C2=A0=CE=95=CE=BB=CE=BB=CE=B7=CE=BD=CE=B9=CE=BA=CE=ACEnglish=ED=
=95=9C=EA=B5=AD=EC=96=B4=E6=97=A5=E6=9C=AC=E8=AA=9E=D0=A0=D1=83=D1=81=D1=81=
=D0=BA=D0=B8=D0=B9=E0=B9=84=E0=B8=97=E0=B8=A2
>
> ----
>
> =C2=A0 =C2=A0$ cat langs.pl
> =C2=A0 =C2=A0use 5.010;
> =C2=A0 =C2=A0use strictures;
> =C2=A0 =C2=A0use Unicode::UCD qw(charinfo);
>
> =C2=A0 =C2=A0sub script {
> =C2=A0 =C2=A0 =C2=A0 =C2=A0return charinfo(ord substr($_[0], 0, 1))->{scr=
ipt}
> =C2=A0 =C2=A0};
>
> =C2=A0 =C2=A0# necessary because pos() magic is tracked on the scalar.
> =C2=A0 =C2=A0my $copy =3D $_;
> =C2=A0 =C2=A0while (/(\X)/g) {
> =C2=A0 =C2=A0 =C2=A0 =C2=A0my $script =3D script $1;
> =C2=A0 =C2=A0 =C2=A0 =C2=A0my ($part) =3D $copy =3D~ /(\p{$script}+)/;
> =C2=A0 =C2=A0 =C2=A0 =C2=A0say $part;
> =C2=A0 =C2=A0 =C2=A0 =C2=A0pos($_) =3D pos($_) + length($part);
> =C2=A0 =C2=A0}

Thanks a lot!

Here is the first version of my tokenizer based on this idea:


use Lingua::ZH::MMSEG;

sub tokenize {
    my $text =3D shift;
    my @tokens;
    while ( $text =3D~ /(\X)/g ) {
        my $part =3D $1;
        my $script =3D charinfo( ord $1)->{script};
        $text=3D~ /(\p{$script}*)/g;
        next if $script eq 'Common';
        $part .=3D $1;
        if( $script eq 'Han' ){
            push @tokens, mmseg( $part );
        }
        else{
            push @tokens, $part;
        }
    }
    return @tokens;
}

And the surprise - this works even without further splitting because
space and other dots all get the 'Common' script and are not matched
by \p{Latin}.

--=20
Zbigniew Lukasiak
http://brudnopis.blogspot.com/
http://perlalchemy.blogspot.com/