RE: Stripping out Unicode combining characters (diacritics)

Hi Leif,

> This is what I do. You can try that.
> See if it helps:
>=20
> Encode::_utf8_on($str);  # <<<
> $str =3D~ s/\pM*//g;

That works!  I will gladly buy the beers Leif, should we ever meet in =
person.

> I mean - have you for instance tried running your cgi scripts=20
> in tainted mode (-T)?

No, I do not run my CGI scripts in tainted mode (although I realize that =
I probably should). =20

Thanks (once again) for your help.

-- Michael

# Michael Doran, Systems Librarian
# University of Texas at Arlington
# 817-272-5326 office
# 817-688-1926 mobile
# [email protected]
# http://rocky.uta.edu/doran/
=20

> -----Original Message-----
> From: Leif Andersson [mailto:[email protected]]=20
> Sent: Tuesday, May 06, 2008 3:33 AM
> To: Doran, Michael D
> Subject: Re: Stripping out Unicode combining characters (diacritics)
>=20
> Oh, now I see your REAL question.
>=20
> This is what I do. You can try that.
> See if it helps:
>=20
> Encode::_utf8_on($str);  # <<<
> $str =3D~ s/\pM*//g;
>=20
> You are not the only one having problems with Unicode.
> Esp. in web programming it can be very confusing.
>=20
> I am quite surprised that there are not more discussions of this kind.
> Not even in the "official" channels.
>=20
> I mean - have you for instance tried running your cgi scripts=20
> in tainted mode (-T)?
>=20
> I had all my scripts set up that way. Before Unicode.
> But basic Unicode stuff became broken with -T enabled.
> Have they fixed that now?
> I have at least seen no mentioning of it.
>=20
> And screen scraping. If you want to mess around with=20
> javascript embedded in an HTML page, you may find that the=20
> content encoding is mixed. And Perl gets very confused=20
> getting mixed character encodings.
> And so do I.
>=20
> You may also have to deal with mixed encodings doing SQL=20
> against the Voyager database.
>=20
> What would we do if we could not fall back on "use bytes"
> every now and then! ;-)
>=20
> Leif
>=20
> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Leif Andersson, Systems Librarian
> Stockholm University Library
> SE-106 91 Stockholm
> SWEDEN
> Phone : +46 8 162769
> Mobile: +46 70 6904281
>=20
>=20
> -----Ursprungligt meddelande-----
> Fr=E5n: Doran, Michael D [mailto:[email protected]]
> Skickat: den 6 maj 2008 04:13
> Till: Mike Rylander
> Kopia: [email protected]; Perl4lib
> =C4mne: RE: Stripping out Unicode combining characters (diacritics)
>=20
> Hi Mike,
>=20
> I appreciate the quick reply.  I am familiar with the=20
> Unicode::Normalize module (and will also be using that), but=20
> I left it out of this question because it's not relevant to=20
> the problem I'm currently trying to solve.  The text I'm=20
> trying to strip diacritics out of does not have precomposed=20
> accented characters.
>=20
> -- Michael
>=20
> # Michael Doran, Systems Librarian
> # University of Texas at Arlington
> # 817-272-5326 office
> # 817-688-1926 cell
> # [email protected]
> # http://rocky.uta.edu/doran/
>=20
>=20
>=20
> -----Original Message-----
> From: Mike Rylander [mailto:[email protected]]
> Sent: Mon 5/5/2008 8:52 PM
> To: Doran, Michael D
> Cc: [email protected]; Perl4lib
> Subject: Re: Stripping out Unicode combining characters (diacritics)
> =20
> On Mon, May 5, 2008 at 8:26 PM, Doran, Michael D=20
> <[email protected]> wrote:
> [snip]
> >
> >  I'm pulling my hair out on this... so any help would be=20
> appreciated.  If there's any other info I can provide, let me know.
> >
>=20
> You'll want to transform the text to NFD format (nominally,=20
> base characters plus combining marks) instead of NFC (precombined
> characters) using Unicode::Normalize:
>=20
>  use Unicode::Normalize;
>=20
>  my $text =3D NFD($original);
>  $text =3D~ s/\pM+//go;
>=20
> Hope that helps.
>=20
> --
> Mike Rylander
>  | VP, Research and Design
>  | Equinox Software, Inc. / The Evergreen Experts  | phone:=20
> 1-877-OPEN-ILS (673-6457)  | email: [email protected]  |=20
> web: http://www.esilibrary.com
>=20
>=20