Re: Tokenization of words containing hyphens

SGkuCgpJIGFncmVlIHdpdGggQ2FybG8gdGhhdCB0aGVyZSBpcyBhIHN0cm9uZyBjYXNlIGZvciBj
b250aW51aW5nIHRvIHRva2VuaXplIApGcmVuY2ggYXMgYXQgcHJlc2VudC4gIFNvLCBJIGFsc28g
YWdyZWUgdGhhdCBteSBwcm9wb3NlZCBhZGRpdGlvbmFsIGNvZGUgZm9yIApoeXBoZW5zIChjaGFu
Z2UgIzMpIG91Z2h0IHRvIGJlIG1hZGUgY29uZGl0aW9uYWwgb24gYSBuZXcgb3B0aW9uIGluIHRo
ZSAKKi5kYXQgZmlsZS4gIFRoZSBvcHRpb24gd291bGQgb25seSBiZSByZWxldmFudCB3aGVuIHRo
ZSAqLmRhdCBhbHNvIGNvbnRhaW5zIAoic3BlY2lhbCAtICIgd2l0aCBhbiBhc3RlcmlzayBpbiB0
aGUgbWVkaWFsIHBvc2l0aW9uLiAgUG9zc2libGUgZm9ybWF0IGZvciAKdGhlIG9wdGlvbjogImNv
bXBvdW5kcyAtIiA/CgpBbHRob3VnaCBteSBvcHRpb24gaXMgbm90IGRpcmVjdGVkIGF0IEZyZW5j
aCwgaXQgaXMgd29ydGggbG9va2luZyBhdCB0aGUgCmltcGxpY2F0aW9ucyBvZiBhIGNob2ljZSBm
b3IgRnJlbmNoLiAgQ29tcGFyZWQgd2l0aCB0aGUgcHJlc2VudCBiZWhhdmlvdXIsIApteSBvcHRp
b24gd2lsbCB3cm9uZ2x5IGFwcHJvdmUgZWcuICJiZWF1LXNvZXVyIiBvciAiYXZlei1ub3VzIiwg
YW5kIHByb2JhYmx5IAptb3N0IHVzZXJzIHdpbGwgYWdyZWUgd2l0aCBDYXJsbyB0aGF0IHN1Y2gg
d3JvbmcgYXBwcm92YWxzIGFyZSBtb3JlIG9mIGEgCm51aXNhbmNlIHRoYW4gYW55IGFtb3VudCBv
ZiB3cm9uZyByZWZ1c2Fscy4gIEJ1dCBvbiB0aGUgb3RoZXIgc2lkZSBvZiB0aGUgCmNvaW4sIEZy
ZW5jaCBoYXMgYSBudW1iZXIgb2YgdHlwZXMgb2YgZmFpcmx5IHByb2R1Y3RpdmUgaHlwaGVuYXRl
ZCBjb21wb3VuZHMgCnN1Y2ggYXMgImRpdC1pbCIgb3IgInRlbXBzLWzDoCI7IG9yIHdvcmRzIHdo
aWNoIGluY2x1ZGUgcHJlZml4ZXMgbGlrZSAKImRlbWktIiwgIm1pLSIsICJhdS0iLCAiYXZhbnQt
IiwgImFycmnDqHJlLSIgb3Igc3VmZml4ZXMgbGlrZSAiLWNpIiBvciAiLWzDoCI7IApvciBudW1l
cmFsczsgb3IgcHJvcGVyIG5hbWVzIOKAlCBhbmQgdGhlIGRpY3Rpb25hcnkgY2FuIG9ubHkgaG9w
ZSB0byBjb250YWluIGEgCnNlbGVjdGlvbiBvZiBhbnkgb2YgdGhlc2UgY29tcG91bmRzIChlZy4g
aXQgaW5jbHVkZXMgImF2ZXotdm91cyI7IGFjdHVhbGx5LCAKdGhlIDAuNTAgRnJlbmNoIGRpY3Rp
b25hcnkgaGFzIDYyOSw1NzAgd29yZHMsIGluIHdoaWNoIHRoZXJlIGFyZSA3LDAwMSAKaHlwaGVu
cyksIGFuZCB0aG9zZSBjb21wb3VuZHMgdGhhdCBpdCBkb2VzIG5vdCBjb250YWluIGFyZSByZWZ1
c2VkLCByaWdodGx5IApvciB3cm9uZ2x5LiAgTXkgb3B0aW9uIHdpbGwgZWxpbWluYXRlIHRoZXNl
IHJlZnVzYWxzLCBhbmQgd2lsbCBhbGxvdyByZW1vdmFsIApmcm9tIHRoZSBkaWN0aW9uYXJ5IG9m
IHRoZSBzZWxlY3Rpb24gb2YgY29tcG91bmRzIHdoaWNoIGhhcyBnb3QgaW50byBpdC4gCkZ1cnRo
ZXIsIGlmIHRoZSBzYW1lIGRlY29tcG91bmRpbmcgYmVoYXZpb3VyIHdlcmUgZXh0ZW5kZWQgdG8g
dGhlIAphcG9zdHJvcGhlLCBhbmQgYycsIGQnLCBqJywgbCcsIG0nLCBuJywgcXUnLCBzJywgdCcg
d2VyZSBhZGRlZCBhcyBwcmVmaXhlcyAKdG8gdGhlIGRpY3Rpb25hcnksIGFnYWluIHRoZSByZXN1
bHQgd291bGQgYmUgYSBtdWNoIHNtYWxsZXIgRnJlbmNoIApkaWN0aW9uYXJ5LCB3aXRoIGZld2Vy
IHdyb25nIHJlZnVzYWxzLCBidXQgbW9yZSB3cm9uZyBhcHByb3ZhbHMgKGVnLiAKbCdkYW1lIOKA
lCBhbHRob3VnaCBvbmUgY291bGQgc2F5IHN1Y2ggYW4gZXJyb3IgaXMgbW9yZSB0byBkbyB3aXRo
IGdyYW1tYXIgCnRoYW4gd2l0aCBzcGVsbGluZykuCgpDaWFyw6FuIMOTIER1aWJow61uCgotLS0t
LSBPcmlnaW5hbCBNZXNzYWdlIC0tLS0tIApGcm9tOiAiQ2FybG8gVHJhdmVyc28iIDx0cmF2ZXJz
b0Bwb3Nzby5kbS51bmlwaS5pdD4KVG86IDxjaWFyYW5Ab2R1aWJoaW4uZnJlZXNlcnZlLmNvLnVr
PgpDYzogPGFzcGVsbC1kZXZlbEBnbnUub3JnPgpTZW50OiBUaHVyc2RheSwgSnVuZSAyNywgMjAx
MyA1OjU5IFBNClN1YmplY3Q6IFJlOiBbYXNwZWxsLWRldmVsXSBUb2tlbml6YXRpb24gb2Ygd29y
ZHMgY29udGFpbmluZyBoeXBoZW5zCgoKPj4+Pj4+ICJjaWFyYW4iID09ID0/aXNvLTg4NTktMT9C
P1EybGhjdUZ1SU5NZ1JIVnBZbWp0Ymc9PT89ICA8aXNvLTg4NTktMT4gCj4+Pj4+PiB3cml0ZXM6
Cj4KPiAgICBjaWFyYW4+IExhbmd1YWdlcyBtYXkgaGF2ZSB3b3JkcyBjb250YWluaW5nIGFuIGlu
dGVybmFsIGh5cGhlbiwKPiAgICBjaWFyYW4+IGJ1dCB3aXRoIHRoZSBjb21wb25lbnRzIG5vdCBi
ZWluZyB0aGVtc2VsdmVzIHdvcmRzIG9mCj4gICAgY2lhcmFuPiB0aGUgbGFuZ3VhZ2UgKGEgcG9z
c2libGUgRW5nbGlzaCBleGFtcGxlIGlzCj4gICAgY2lhcmFuPiBob3RjaC1wb3RjaCkuICBJbiBz
dWNoIGxhbmd1YWdlcyBpdCBpcyB3ZWxsIHRvIGFsbG93IGEKPiAgICBjaWFyYW4+IHdvcmQtaW50
ZXJuYWwgaHlwaGVuIGluICouZGF0IGFuZCBwdXQgc3VjaCAiY29tcG91bmRzIiBpbgo+ICAgIGNp
YXJhbj4gdGhlIGRpY3Rpb25hcnkuICBObyBuZXcgY29kZSBpcyByZXF1aXJlZCBmb3IgdGhpcy4K
PiAgICBjaWFyYW4+IEhvd2V2ZXIsIHdpdGggdGhlIGNoYW5nZSBpbiBzdGF0dXMgb2YgdGhlIGh5
cGhlbiwgYWxsCj4gICAgY2lhcmFuPiBoeXBoZW5hdGVkIGNvbXBvdW5kcyBub3QgZXhwbGljaXRs
eSBpbmNsdWRlZCBpbiB0aGUKPiAgICBjaWFyYW4+IGRpY3Rpb25hcnkgd2lsbCBub3cgYmUgcmVq
ZWN0ZWQsIGV2ZW4gaWYgdGhlaXIKPiAgICBjaWFyYW4+IGNvbXBvbmVudHMgYXJlIGFsbCBpbiB0
aGUgZGljdGlvbmFyeS4gIFRvIGF2b2lkIHRoaXMsIG5ldwo+ICAgIGNpYXJhbj4gY29kZSBpcyBu
ZWVkZWQsIGZvciBsYW5ndWFnZXMgc3VwcG9ydGluZyBpbnRlcm5hbAo+ICAgIGNpYXJhbj4gaHlw
aGVuLCB0byBleGFtaW5lIGEgcmVqZWN0ZWQgd29yZCwgYW5kIGlmIGl0IGNvbnRhaW5zCj4gICAg
Y2lhcmFuPiBhbiBpbnRlcm5hbCBoeXBoZW4sIHRvIGNoZWNrIHRoZSBjb21wb25lbnRzIHNlcGFy
YXRlbHkuCj4gICAgY2lhcmFuPiBJZiBhbGwgdGhlIGNvbXBvbmVudHMgYXJlIGFjY2VwdGVkLCBz
byBpcyB0aGUgY29tcG91bmQuCj4gICAgY2lhcmFuPiBUaGUgaHlwaGVuIGl0c2VsZiB3aWxsIG5v
dCBiZSBpbmNsdWRlZCBpbiB0aGUgc2VwYXJhdGUKPiAgICBjaWFyYW4+IGNvbXBvbmVudHMgb24g
ZWl0aGVyIHNpZGUgb2YgaXQuCj4KPiBJIGRvbid0IHRoaW5rIHRoYXQgdGhpcyBpcyBhIGdvb2Qg
aWRlYSBmb3IgZXhhbXBsZSBmb3IgRnJlbmNoICh0aGUKPiBsYW5ndWFnZSB0aGF0IEkga25vdyB0
aGF0IGFsbG93cyB3b3JkLWludGVybmFsIGh5cGhlbiBpbiAqLmRhdCkuCj4KPiBDb21wb3Npbmcg
d29yZHMgd2l0aCBoeXBoZW5zIGlzIHN1YmplY3QgdG8gc2V2ZXJhbCBydWxlcywgc28gbGlzdGlu
Zwo+IHRoZSBhbGxvd2VkIHBhaXJzIGlzIGJldHRlci4gRm9yIGV4YW1wbGUsIGJlYXUsIGJlbGxl
LCBmcsOocmUsIHNvZXVyCj4gYXJlIGFsbG93ZWQgd29yZHMsIGFzIHdlbGwgYXMgYmVhdS1mcsOo
cmUsIGJlbGxlLXNvZXVyIChicm90aGVyLQo+IChzaXN0ZXIpLWluLWxhdykuIEJ1dCBiZWF1LXNv
ZXVyIGlzIHdyb25nIChhcyBpcyAiYmVhdSBzb2V1ciIgd2l0aG91dAo+IGh5cGhlbiBidHcpIGJl
Y2F1c2UgeW91IG5lZWQgdG8gbWF0Y2ggZ2VucmVzLCBtYXNjdWxpbmUgd2l0aCBtYXNjdWxpbmUK
PiBldGMuIEFuZCBtb3N0IHdvcmQgcGFpcnMsIGV2ZW4gaWYgdGhlIGdlbnJlIGFuZCBjYXNlIGFy
ZSBPSywgYXJlCj4gd3JvbmcuIFRoZSBjb3JyZWN0IG9uZXMgc2hvdWxkIGJlIGluY2x1ZGVkIGlu
IHRoZSBkaWN0aW9uYXJ5LCB0aGUKPiBvdGhlciBvbmVzIGNvbnNpZGVyZWQgd3JvbmcuCj4KPiBJ
biBHZXJtYW4sIGZvciBleGFtcGxlLCB5b3UgY29tcG9zZSB3b3JkcyB3aXRob3V0IGh5cGhlbnMu
IFlvdSB3b3VsZAo+IG5ldmVyIGFsbG93IGEgd29yZCBqdXN0IGJlY2F1c2UgaXQgaXMgb2J0YWlu
ZWQgY29tcG9zaW5nIHR3byB3b3Jkcy4KPgo+IE15IGZlZWxpbmcgbWlnaHQgZGVwZW5kIGZyb20g
bXkgdXNhZ2Ugb2YgYSBzcGVsbC1jaGVja2VyLCB0aGF0IGlzIHRvCj4gY2hlY2sgT0NSIHJlc3Vs
dHMgb2YgdmludGFnZSB0ZXh0cywgaGVuY2UgSSBwcmVmZXIgZmFsc2UgcG9zaXRpdmVzCj4gKG1h
cmsgYXMgZXJyb25lb3VzIHdvcmRzIHRoYXQgYXJlIE9LKSB0byBmYWxzZSBuZWdhdGl2ZXMsIGFu
ZCBPQ1IKPiBxdWl0ZSBlYXNpbHkgaW5zZXJ0cyBhIGh5cGhlbiBpbiBwbGFjZSBvZiBhIHNwZWNr
bGUgYmV0d2VlbiB3b3Jkcywgc28KPiB0aGlzIG1pZ2h0IGxlYWQgdG8gYWNjZXB0IGEgZmV3IHdy
b25nIHdvcmRzIHdpdGhvdXQgYWNjZXB0aW5nIGEKPiBzdWJzdGFudGlhbCBudW1iZXIgb2Ygd29y
ZHMgdGhhdCBjdXJyZW50bHkgd291bGQgYmUgT0sgYnV0IGZsYWdnZWQuCj4KPiBTbyBpZiB5b3Ug
cmVhbGx5IHdhbnQgdG8gYWNjZXB0IHdvcmRzIHRoYXQgYXJlIGNvbXBvc2VkIGZyZWVseSwgdGhl
Cj4gZmVhdHVyZSBzaG91bGQgYmUgbGFuZ3VhZ2Utc3BlY2lmaWMgKEVuZ2xpc2ggSSBiZWxpZXZl
IG1pZ2h0IGJlIHRoZQo+IG9ubHkgb25lKSBoZW5jZSBpbmNsdWRlZCBpbiAqLmRhdCBmaWxlcy4g
Rm9yIEZyZW5jaCBpdCBpcyBPSyBhcyBpcywKPiBkb24ndCBmaXggd2hhdCBpc24ndCBicm9rZW4u
Cj4KPgo+IENhcmxvIFRyYXZlcnNvIAoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fCkFzcGVsbC1kZXZlbCBtYWlsaW5nIGxpc3QKQXNwZWxsLWRldmVsQGdu
dS5vcmcKaHR0cHM6Ly9saXN0cy5nbnUub3JnL21haWxtYW4vbGlzdGluZm8vYXNwZWxsLWRldmVs
Cg==