Re: Tokenization of words containing hyphens
[email protected] (Carlo Traverso) Thu, 27 Jun 2013 18:59:15 +0200 (CEST)
| Newsgroups | gmane.comp.gnu.aspell.devel |
|---|---|
| Message-ID | <[email protected]> |
Pj4+Pj4gImNpYXJhbiIgPT0gPT9pc28tODg1OS0xP0I/UTJsaGN1RnVJTk1nUkhWcFltanRiZz09 Pz0gIDxpc28tODg1OS0xPiB3cml0ZXM6CgogICAgY2lhcmFuPiBMYW5ndWFnZXMgbWF5IGhhdmUg d29yZHMgY29udGFpbmluZyBhbiBpbnRlcm5hbCBoeXBoZW4sCiAgICBjaWFyYW4+IGJ1dCB3aXRo IHRoZSBjb21wb25lbnRzIG5vdCBiZWluZyB0aGVtc2VsdmVzIHdvcmRzIG9mCiAgICBjaWFyYW4+ IHRoZSBsYW5ndWFnZSAoYSBwb3NzaWJsZSBFbmdsaXNoIGV4YW1wbGUgaXMKICAgIGNpYXJhbj4g aG90Y2gtcG90Y2gpLiAgSW4gc3VjaCBsYW5ndWFnZXMgaXQgaXMgd2VsbCB0byBhbGxvdyBhCiAg ICBjaWFyYW4+IHdvcmQtaW50ZXJuYWwgaHlwaGVuIGluICouZGF0IGFuZCBwdXQgc3VjaCAiY29t cG91bmRzIiBpbgogICAgY2lhcmFuPiB0aGUgZGljdGlvbmFyeS4gIE5vIG5ldyBjb2RlIGlzIHJl cXVpcmVkIGZvciB0aGlzLgogICAgY2lhcmFuPiBIb3dldmVyLCB3aXRoIHRoZSBjaGFuZ2UgaW4g c3RhdHVzIG9mIHRoZSBoeXBoZW4sIGFsbAogICAgY2lhcmFuPiBoeXBoZW5hdGVkIGNvbXBvdW5k cyBub3QgZXhwbGljaXRseSBpbmNsdWRlZCBpbiB0aGUKICAgIGNpYXJhbj4gZGljdGlvbmFyeSB3 aWxsIG5vdyBiZSByZWplY3RlZCwgZXZlbiBpZiB0aGVpcgogICAgY2lhcmFuPiBjb21wb25lbnRz IGFyZSBhbGwgaW4gdGhlIGRpY3Rpb25hcnkuICBUbyBhdm9pZCB0aGlzLCBuZXcKICAgIGNpYXJh bj4gY29kZSBpcyBuZWVkZWQsIGZvciBsYW5ndWFnZXMgc3VwcG9ydGluZyBpbnRlcm5hbAogICAg Y2lhcmFuPiBoeXBoZW4sIHRvIGV4YW1pbmUgYSByZWplY3RlZCB3b3JkLCBhbmQgaWYgaXQgY29u dGFpbnMKICAgIGNpYXJhbj4gYW4gaW50ZXJuYWwgaHlwaGVuLCB0byBjaGVjayB0aGUgY29tcG9u ZW50cyBzZXBhcmF0ZWx5LgogICAgY2lhcmFuPiBJZiBhbGwgdGhlIGNvbXBvbmVudHMgYXJlIGFj Y2VwdGVkLCBzbyBpcyB0aGUgY29tcG91bmQuCiAgICBjaWFyYW4+IFRoZSBoeXBoZW4gaXRzZWxm IHdpbGwgbm90IGJlIGluY2x1ZGVkIGluIHRoZSBzZXBhcmF0ZQogICAgY2lhcmFuPiBjb21wb25l bnRzIG9uIGVpdGhlciBzaWRlIG9mIGl0LgoKSSBkb24ndCB0aGluayB0aGF0IHRoaXMgaXMgYSBn b29kIGlkZWEgZm9yIGV4YW1wbGUgZm9yIEZyZW5jaCAodGhlCmxhbmd1YWdlIHRoYXQgSSBrbm93 IHRoYXQgYWxsb3dzIHdvcmQtaW50ZXJuYWwgaHlwaGVuIGluICouZGF0KS4gCgpDb21wb3Npbmcg d29yZHMgd2l0aCBoeXBoZW5zIGlzIHN1YmplY3QgdG8gc2V2ZXJhbCBydWxlcywgc28gbGlzdGlu Zwp0aGUgYWxsb3dlZCBwYWlycyBpcyBiZXR0ZXIuIEZvciBleGFtcGxlLCBiZWF1LCBiZWxsZSwg ZnLDqHJlLCBzb2V1cgphcmUgYWxsb3dlZCB3b3JkcywgYXMgd2VsbCBhcyBiZWF1LWZyw6hyZSwg YmVsbGUtc29ldXIgKGJyb3RoZXItCihzaXN0ZXIpLWluLWxhdykuIEJ1dCBiZWF1LXNvZXVyIGlz IHdyb25nIChhcyBpcyAiYmVhdSBzb2V1ciIgd2l0aG91dApoeXBoZW4gYnR3KSBiZWNhdXNlIHlv dSBuZWVkIHRvIG1hdGNoIGdlbnJlcywgbWFzY3VsaW5lIHdpdGggbWFzY3VsaW5lCmV0Yy4gQW5k IG1vc3Qgd29yZCBwYWlycywgZXZlbiBpZiB0aGUgZ2VucmUgYW5kIGNhc2UgYXJlIE9LLCBhcmUK d3JvbmcuIFRoZSBjb3JyZWN0IG9uZXMgc2hvdWxkIGJlIGluY2x1ZGVkIGluIHRoZSBkaWN0aW9u YXJ5LCB0aGUKb3RoZXIgb25lcyBjb25zaWRlcmVkIHdyb25nLgoKSW4gR2VybWFuLCBmb3IgZXhh bXBsZSwgeW91IGNvbXBvc2Ugd29yZHMgd2l0aG91dCBoeXBoZW5zLiBZb3Ugd291bGQKbmV2ZXIg YWxsb3cgYSB3b3JkIGp1c3QgYmVjYXVzZSBpdCBpcyBvYnRhaW5lZCBjb21wb3NpbmcgdHdvIHdv cmRzLiAKCk15IGZlZWxpbmcgbWlnaHQgZGVwZW5kIGZyb20gbXkgdXNhZ2Ugb2YgYSBzcGVsbC1j aGVja2VyLCB0aGF0IGlzIHRvCmNoZWNrIE9DUiByZXN1bHRzIG9mIHZpbnRhZ2UgdGV4dHMsIGhl bmNlIEkgcHJlZmVyIGZhbHNlIHBvc2l0aXZlcwoobWFyayBhcyBlcnJvbmVvdXMgd29yZHMgdGhh dCBhcmUgT0spIHRvIGZhbHNlIG5lZ2F0aXZlcywgYW5kIE9DUgpxdWl0ZSBlYXNpbHkgaW5zZXJ0 cyBhIGh5cGhlbiBpbiBwbGFjZSBvZiBhIHNwZWNrbGUgYmV0d2VlbiB3b3Jkcywgc28KdGhpcyBt aWdodCBsZWFkIHRvIGFjY2VwdCBhIGZldyB3cm9uZyB3b3JkcyB3aXRob3V0IGFjY2VwdGluZyBh CnN1YnN0YW50aWFsIG51bWJlciBvZiB3b3JkcyB0aGF0IGN1cnJlbnRseSB3b3VsZCBiZSBPSyBi dXQgZmxhZ2dlZC4KClNvIGlmIHlvdSByZWFsbHkgd2FudCB0byBhY2NlcHQgd29yZHMgdGhhdCBh cmUgY29tcG9zZWQgZnJlZWx5LCB0aGUKZmVhdHVyZSBzaG91bGQgYmUgbGFuZ3VhZ2Utc3BlY2lm aWMgKEVuZ2xpc2ggSSBiZWxpZXZlIG1pZ2h0IGJlIHRoZQpvbmx5IG9uZSkgaGVuY2UgaW5jbHVk ZWQgaW4gKi5kYXQgZmlsZXMuIEZvciBGcmVuY2ggaXQgaXMgT0sgYXMgaXMsCmRvbid0IGZpeCB3 aGF0IGlzbid0IGJyb2tlbi4KCgpDYXJsbyBUcmF2ZXJzbwoKX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KQXNwZWxsLWRldmVsIG1haWxpbmcgbGlzdApBc3Bl bGwtZGV2ZWxAZ251Lm9yZwpodHRwczovL2xpc3RzLmdudS5vcmcvbWFpbG1hbi9saXN0aW5mby9h c3BlbGwtZGV2ZWwK