Stemming Romanian with s&t with comma (vs cedilla)

--===============0829760453738230420==
Content-Type: multipart/alternative; boundary="000000000000ac122505f5caa191"

--000000000000ac122505f5caa191
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi All,

I work on the Search Platform Team at the Wikimedia Foundation and I've
been looking at Romanian Wikipedia data and working with the Romanian
Elasticsearch analysis chain, which uses Lucene analyzers, which use the
Snowball stemmer.

Historically poor software, OS, and font support made s&t with cedilla a
practical choice for many years, but s&t with commas are generally well
supported now, and seem to have been since at least 2010. See more at
Wikipedia:

https://en.wikipedia.org/wiki/Romanian_alphabet#Comma-below_(%C8%99_and_%C8=
%9B)_versus_cedilla_(%C5%9F_and_%C5%A3)

I recently discovered that the Lucene stopword list uses the cedilla forms,
and the following discussion on GitHub revealed that the Snowball stemmer
uses the cedilla forms, too. At least on Romanian Wikipedia, the correct
comma forms are much more common, though both are present.

For example, =C5=9Fi ("and", with cedilla) occurs in 4,976 documents[1],
while =C8=99i ("and" with comma) occurs in 288,999 documents[2]. I haven't =
yet
finished my analysis, but I expect conflating =C8=99 and =C5=9F will have a=
 big
impact on stemming on Wikipedia. While I can patch our implementation, and
I think the Lucene folks will patch theirs, it'd be great if the underlying
Snowball stemmer could handle modern Romanian text correctly, too.

I'd make a pull request on GitHub, but I don't speak Snowball well enough
to be sure how to do it. But in general it looks like if variables s and t
on lines 26 and 27 here:

https://github.com/snowballstem/snowball/blob/master/algorithms/romanian.sb=
l#L26-L27

could be expanded to include {U+0219} (s with comma) and {U+021B} (t with
comma), respectively, that might do it.

Thanks for all the work that's gone into the Snowball stemmers over the
years!

=E2=80=94Trey

[1]
https://ro.wikipedia.org/w/index.php?search=3D%22%C5%9Fi%22&title=3DSpecial=
:C%C4%83utare&profile=3Ddefault&fulltext=3D1
[2]
https://ro.wikipedia.org/w/index.php?search=3D%22%C8%99i%22&title=3DSpecial=
:C%C4%83utare&profile=3Ddefault&fulltext=3D1

Trey Jones
Staff Computational Linguist, Search Platform
Wikimedia Foundation
UTC=E2=80=935 / EST

--000000000000ac122505f5caa191
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi All,</div><div><br></div><div>I work on the Search=
 Platform Team at the Wikimedia Foundation and I&#39;ve been looking at Rom=
anian Wikipedia data and working with the Romanian Elasticsearch analysis c=
hain, which uses Lucene analyzers, which use the Snowball stemmer.</div><di=
v><br></div><div>Historically poor software, OS, and font support made s&am=
p;t with cedilla a practical choice for many years, but s&amp;t with commas=
 are generally well supported now, and seem to have been since at least 201=
0. See more at Wikipedia:<br></div><div><br></div><div><div><a href=3D"http=
s://en.wikipedia.org/wiki/Romanian_alphabet#Comma-below_(%C8%99_and_%C8%9B)=
_versus_cedilla_(%C5%9F_and_%C5%A3)">https://en.wikipedia.org/wiki/Romanian=
_alphabet#Comma-below_(%C8%99_and_%C8%9B)_versus_cedilla_(%C5%9F_and_%C5%A3=
)</a><br></div><div><br></div></div><div>I recently discovered that the Luc=
ene stopword list uses the cedilla forms, and the following discussion on G=
itHub revealed that the Snowball stemmer uses the cedilla forms, too. At le=
ast on Romanian Wikipedia, the correct comma forms are much more common, th=
ough both are present.</div><div><br></div><div>For example,=C2=A0=C5=9Fi (=
&quot;and&quot;, with cedilla) occurs in 4,976 documents[1], while=C2=A0=C8=
=99i (&quot;and&quot; with comma) occurs in 288,999 documents[2]. I haven&#=
39;t yet finished my analysis, but I expect conflating =C8=99 and =C5=9F wi=
ll have a big impact on stemming on Wikipedia. While I can patch our implem=
entation, and I think the Lucene folks will patch theirs, it&#39;d be great=
 if the underlying Snowball stemmer could handle modern Romanian text corre=
ctly, too.</div><div><br></div><div>I&#39;d make a pull request on GitHub, =
but I don&#39;t speak Snowball well enough to be sure how to do it. But in =
general it looks like if variables s and t on lines 26 and 27 here:</div><d=
iv><br></div><div><a href=3D"https://github.com/snowballstem/snowball/blob/=
master/algorithms/romanian.sbl#L26-L27">https://github.com/snowballstem/sno=
wball/blob/master/algorithms/romanian.sbl#L26-L27</a><br></div><div><br></d=
iv><div>could be expanded to include {U+0219} (s with comma) and {U+021B} (=
t with comma), respectively, that might do it.</div><div><br></div><div>Tha=
nks for all the work that&#39;s gone into the Snowball stemmers over the ye=
ars!</div><div><br></div><div>=E2=80=94Trey</div><div><br></div><div>[1]=C2=
=A0<a href=3D"https://ro.wikipedia.org/w/index.php?search=3D%22%C5%9Fi%22&a=
mp;title=3DSpecial:C%C4%83utare&amp;profile=3Ddefault&amp;fulltext=3D1">htt=
ps://ro.wikipedia.org/w/index.php?search=3D%22%C5%9Fi%22&amp;title=3DSpecia=
l:C%C4%83utare&amp;profile=3Ddefault&amp;fulltext=3D1</a></div><div>[2]=C2=
=A0<a href=3D"https://ro.wikipedia.org/w/index.php?search=3D%22%C8%99i%22&a=
mp;title=3DSpecial:C%C4%83utare&amp;profile=3Ddefault&amp;fulltext=3D1">htt=
ps://ro.wikipedia.org/w/index.php?search=3D%22%C8%99i%22&amp;title=3DSpecia=
l:C%C4%83utare&amp;profile=3Ddefault&amp;fulltext=3D1</a></div><br clear=3D=
"all"><div><div dir=3D"ltr" class=3D"gmail_signature" data-smartmail=3D"gma=
il_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr">=
<div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div=
 dir=3D"ltr"><div><div style=3D"font-size:small"><font color=3D"#999999"><s=
pan style=3D"font-size:12.8px">Trey Jones</span><br></font></div><span styl=
e=3D"font-size:12.8px"><font color=3D"#999999">Staff Computational Linguist=
, Search Platform<br>Wikimedia Foundation</font></span><br></div><div><span=
 style=3D"font-size:12.8px"><font color=3D"#999999">UTC=E2=80=935 / EST</fo=
nt></span></div></div></div></div></div></div></div></div></div></div></div=
></div></div></div></div></div></div>

--000000000000ac122505f5caa191--


--===============0829760453738230420==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Snowball-discuss mailing list
[email protected]
https://lists.tartarus.org/mailman/listinfo/snowball-discuss

--===============0829760453738230420==--