RE: Stripping out Unicode combining characters (diacritics) -
[email protected] ("Doran, Michael D") Wed, 7 May 2008 08:51:02 -0500
| Newsgroups | perl.perl4lib,perl.i18n |
|---|---|
| Message-ID | <[email protected]> |
SSByZWNlaXZlZCBhIG51bWJlciBvZiBoZWxwZnVsIHN1Z2dlc3Rpb25zIGFuZCBzb2x1dGlvbnMu ICBUaGUgYXBwcm9hY2ggSSBkZWNpZGVkIHRvIGFkb3B0IGluIG15IGxhcmdlciBzY3JpcHQgaXMg dG8gJ2RlY29kZScgYWxsIHRoZSBpbmNvbWluZyBmb3JtIGlucHV0IGFzIFVURi04IGFzIHdlbGwg YXMgdGhlIGlucHV0IGZyb20gdGhlIGRhdGFiYXNlIHRoYXQgSSdsbCBiZSBtYXRjaGluZyB0aGUg Zm9ybSBpbnB1dCBhZ2FpbnN0LiAgVGhpcyBzZWVtcyB0byBhbGxvdyB0aGUgJ1xwe019JyBzeW50 YXggdG8gd29yayBhcyBleHBlY3RlZCBpbiBhIFBlcmwgcmVnZXhwLiAgSW4gbXkgdGVzdC5jZ2kg c2NyaXB0IGZvciBmb3JtIGlucHV0IGl0IHdvdWxkIGxpa2UgbGlrZSB0aGlzOg0KDQojIS91c3Iv bG9jYWwvYmluL3BlcmwNCnVzZSBzdHJpY3Q7DQp1c2UgQ0dJOw0KdXNlIEVuY29kZTsNCm15ICRx dWVyeSA9IENHSTo6bmV3KCk7DQpteSAkc2VhcmNoX3Rlcm0gPSBkZWNvZGUoJ1VURi04JywkcXVl cnktPnBhcmFtKCd0ZXh0JykpOw0KbXkgJHNhbnNfZGlhY3JpdGljcyAgPSAkc2VhcmNoX3Rlcm07 DQokc2Fuc19kaWFjcml0aWNzID1+IHMvXHBNKi8vZzsNCnByaW50IHFxKENvbnRlbnQtdHlwZTog dGV4dC9wbGFpbjsgY2hhcnNldD11dGYtOA0KDQpzZWFyY2hfdGVybSAgICAgaXMgJHNlYXJjaF90 ZXJtDQpzYW5zX2RpYWNyaXRpY3MgaXMgJHNhbnNfZGlhY3JpdGljcw0KKTsNCmV4aXQoMCk7DQoN CkknbSBzbG93bHkgZmlndXJpbmcgb3V0IGhvdyB0byB3b3JrIHdpdGggVW5pY29kZSBpbiBteSB3 ZWIgc2NyaXB0cywgYnV0IHN0aWxsIGhhdmUgYSBsb3QgdG8gbGVhcm4uICBUaGFua3MgZm9yIGFs bCB0aGUgaGVscC4gOi0pDQoNCi0tIE1pY2hhZWwNCg0KIyBNaWNoYWVsIERvcmFuLCBTeXN0ZW1z IExpYnJhcmlhbg0KIyBVbml2ZXJzaXR5IG9mIFRleGFzIGF0IEFybGluZ3Rvbg0KIyA4MTctMjcy LTUzMjYgb2ZmaWNlDQojIDgxNy02ODgtMTkyNiBtb2JpbGUNCiMgZG9yYW5AdXRhLmVkdQ0KIyBo dHRwOi8vcm9ja3kudXRhLmVkdS9kb3Jhbi8NCiANCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2Ut LS0tLQ0KPiBGcm9tOiBEb3JhbiwgTWljaGFlbCBEIFttYWlsdG86ZG9yYW5AdXRhLmVkdV0gDQo+ IFNlbnQ6IE1vbmRheSwgTWF5IDA1LCAyMDA4IDc6MjcgUE0NCj4gVG86IHBlcmwtaTE4bkBwZXJs Lm9yZw0KPiBDYzogUGVybDRsaWINCj4gU3ViamVjdDogU3RyaXBwaW5nIG91dCBVbmljb2RlIGNv bWJpbmluZyBjaGFyYWN0ZXJzIChkaWFjcml0aWNzKQ0KPiANCj4gSSdtIHRyeWluZyB0byBzdHJp cCBvdXQgY29tYmluaW5nIGRpYWNyaXRpY3MgZnJvbSBzb21lIGZvcm0gDQo+IGlucHV0IHVzaW5n IHRoaXMgY29kZToNCj4gDQo+IDxoZWFkPg0KPiAgICAgPE1FVEEgaHR0cC1lcXVpdj0iQ29udGVu dC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IA0KPiBjaGFyc2V0PVVURi04Ij4gPC9oZWFkPiA8 Ym9keT4NCj4gICA8Zm9ybSBhY3Rpb249InRlc3QuY2dpIiBhY2NlcHQtY2hhcnNldD0iVVRGLTgi IG1ldGhvZD0iZ2V0Ij4NCj4gICAgIDxpbnB1dCB0eXBlPSJ0ZXh0IiBuYW1lPSJ0ZXh0IiB2YWx1 ZT0iIiBzaXplPSIxMCI+DQo+ICAgICA8aW5wdXQgdHlwZT0ic3VibWl0IiB2YWx1ZT0ic3VibWl0 Ij4NCj4gICA8L2Zvcm0+DQo+IDwvYm9keT4NCj4gPC9odG1sPg0KPiANCj4gIyEvdXNyL2xvY2Fs L2Jpbi9wZXJsDQo+IHVzZSBDR0k7DQo+ICRxdWVyeSA9IENHSTo6bmV3KCk7DQo+ICRzZWFyY2hf dGVybSA9ICRxdWVyeS0+cGFyYW0oJ3RleHQnKTsNCj4gJHNhbnNfZGlhY3JpdGljcyAgPSAkc2Vh cmNoX3Rlcm07DQo+ICRzYW5zX2RpYWNyaXRpY3MgID1+IHMvXHB7TX0qLy9nOw0KPiAjJHNhbnNf ZGlhY3JpdGljcyAgPX4gcy9vLy9nOw0KPiBwcmludCBxcShDb250ZW50LXR5cGU6IHRleHQvcGxh aW47IGNoYXJzZXQ9dXRmLTgNCj4gDQo+ICRzYW5zX2RpYWNyaXRpY3MNCj4gKTsNCj4gZXhpdCgw KTsNCj4gDQo+IA0KPiBJbiB0aGUgZm9ybSwgSSdtIGlucHV0dGluZyB0aGUgc3RyaW5nICJCYXJ0 b8yBayIgd2l0aCB0aGUgDQo+IGFjY2VudGVkIGNoYXJhY3RlciBiZWluZyBhIGJhc2UgY2hhcmFj dGVyIChzbWFsbCBMYXRpbiBsZXR0ZXIgDQo+ICJvIikgZm9sbG93ZWQgYnkgYSBjb21iaW5pbmcg YWN1dGUgYWNjZW50LiAgSG93ZXZlciwgd2hlbiBJIA0KPiBwcmludCAodG8gdGhlIHdlYikgJHNh bnNfZGlhY3JpdGljcywgSSBnZXQgbXkgaW5wdXQgd2l0aCBubyANCj4gY2hhbmdlIC0tIHRoZSBj b21iaW5pbmcgZGlhY3JpdGljIGlzIHN0aWxsIHRoZXJlLiAgSSBrbm93IA0KPiB0aGF0IG15IGlu cHV0IGlzIG5vdCBhIHByZWNvbXBvc2VkIGFjY2VudGVkIGNoYXJhY3RlciwgDQo+IGJlY2F1c2Ug SSBjYW4gc3RyaXAgb3V0IHRoZSBiYXNlICJvIiBhbmQgdGhlIGNvbWJpbmluZyBhY2NlbnQgDQo+ IGVpdGhlciBzdGFuZHMgYWxvbmUgb3IganVtcHMgdG8gYW5vdGhlciBjaGFyYWN0ZXIgWzJdLg0K PiANCj4gVGhlICJccHtNfSIgaXMgYSBVbmljb2RlIGNsYXNzIG5hbWUgZm9yIHRoZSBjaGFyYWN0 ZXIgY2xhc3MgDQo+IG9mIFVuaWNvZGUgJ21hcmtzJywgZm9yIGV4YW1wbGUgYWNjZW50IG1hcmtz IFsxXS4gIEkndmUgdHJpZWQgDQo+IHRoZXNlIHZhcmlhdGlvbnMgKGFuZCBtYW55IG90aGVycykg YW5kIG5vbmUgc2VlbSB0byBiZSBkb2luZyANCj4gd2hhdCBJIHdhbnQ6DQo+IA0KPiAgICAgICAg JHNhbnNfZGlhY3JpdGljcyA9fiBzI1tccHtNYXJrfV0qIyNnOw0KPiAgICAgICAgJHNhbnNfZGlh Y3JpdGljcyA9fiB0ciNbXHB7SW5Db21iaW5pbmdEaWFjcml0aWNhbE1hcmtzfV0jIzsNCj4gICAg ICAgICRzYW5zX2RpYWNyaXRpY3MgPX4gdHIjW1xwe019XSMjOw0KPiAgICAgICAgJHNhbnNfZGlh Y3JpdGljcyA9fiBzL1xwe019Ki8vZzsNCj4gICAgICAgICRzYW5zX2RpYWNyaXRpY3MgPX4gcyNb XHB7TX1dIyNnOw0KPiAgICAgICAgJHNhbnNfZGlhY3JpdGljcyA9fiBzI1x4ezAzMDF9IyNnOw0K PiAgICAgICAgJHNhbnNfZGlhY3JpdGljcyA9fiBzI1x4ezAwNkZ9XHh7MDMwMX0jI2c7DQo+ICAg ICAgICAkc2Fuc19kaWFjcml0aWNzID1+IHMjW1x4ezAzMDB9LVx4ezAzNkZ9XSojI2c7DQo+IA0K PiBJJ20gcHVsbGluZyBteSBoYWlyIG91dCBvbiB0aGlzLi4uIHNvIGFueSBoZWxwIHdvdWxkIGJl IA0KPiBhcHByZWNpYXRlZC4gIElmIHRoZXJlJ3MgYW55IG90aGVyIGluZm8gSSBjYW4gcHJvdmlk ZSwgbGV0IG1lIGtub3cuDQo+IA0KPiBNeSBQZXJsIHZlcnNpb24gaXMgNS44LjggYW5kIHRoZSBz Y3JpcHQgaXMgcnVubmluZyBvbiBhIA0KPiBzZXJ2ZXIgcnVubmluZyBTb2xhcmlzIDkuDQo+IA0K PiAtLSBNaWNoYWVsDQo+IA0KPiBbMV0gcGVyIGh0dHA6Ly9wZXJsZG9jLnBlcmwub3JnL3Blcmxy ZXR1dC5odG1sIGFuZCBvdGhlciBkb2N1bWVudGF0aW9uDQo+IA0KPiBbMl0gdXNpbmcgJHNhbnNf ZGlhY3JpdGljcyAgPX4gcy9vLy9nOw0KPiANCj4gIyBNaWNoYWVsIERvcmFuLCBTeXN0ZW1zIExp YnJhcmlhbg0KPiAjIFVuaXZlcnNpdHkgb2YgVGV4YXMgYXQgQXJsaW5ndG9uDQo+ICMgODE3LTI3 Mi01MzI2IG9mZmljZQ0KPiAjIDgxNy02ODgtMTkyNiBtb2JpbGUNCj4gIyBkb3JhbkB1dGEuZWR1 DQo+ICMgaHR0cDovL3JvY2t5LnV0YS5lZHUvZG9yYW4vDQo+IA0K