Re: Stemming Romanian with s&t with comma (vs cedilla)

--===============3510698555313884280==
Content-Type: multipart/alternative; boundary="000000000000f2497405f679a05e"

--000000000000f2497405f679a05e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks for the reply, Martin.

Since this issue started downstream of Snowball, there have been other
discussions not on this list. Those discussions led to Robert Muir
submitting pull requests to snowball (#177) and snowball-data (#23) on
GitHub. He's opted to map to the comma forms, which I agree with; obviously
it could be wired the other way around if necessary.

Thanks for the pointer to Italian! That was the model that probably would
have helped me write my own patch, but Robert already had a handle on it so
I'm happy to let him take care of it.

=E2=80=94Trey
Trey Jones
Staff Computational Linguist, Search Platform
Wikimedia Foundation
UTC=E2=80=935 / EST


On Wed, Mar 8, 2023 at 7:14=E2=80=AFPM Martin Porter <martin.f.porter@gmail=
.com>
wrote:

> Trey,
>
> Thank you for pointing this out. (Sorry for the late reply.) When I
> wrote the Romanian stemmer in 2007, the data I used certainly had s&t
> with cedilla, and I was not aware of the s&t with comma form.
>
> Some normalisation of the text strings given to the stemmers is
> expected, and most especially the mapping of upper case letters to
> lower case. But sometimes a bit of normalisation is done in the
> stemmer. I think that is what should happen here.
>
> Have a look at the 'prelude' routine in the Italian stemmer, and
> compare with the 'prelude' routine in Romanian. In Italian, the rules
> for the direction (acute or grave) of accents over vowels were quite
> relaxed, and varied from one publishing house to another. Perhaps the
> rules are tighter now, but there was no harm in mapping the acutes to
> graves, which is done in the first repeat loop. We put the same idea
> in the Romanian stemmer.
>
> The question is, do we map cedilla to comma-under, or comma-under to
> cedilla? Since comma-under is becoming standard, it would seem sensibe
> to map cedilla to comma-under, but even so, I would suggest doing it
> the other way round, so that users who have had to rely on the cedilla
> representation will not notice any change. The stemmed forms will have
> the cedilla form of s&t, but these should be hidden from view anyway.
>
> Olly Betts manages github Snowball, and we must see what he thinks.
>
> I could write the necessary patch, but it would take me some time to
> test it out.
>
> Martin
>

--000000000000f2497405f679a05e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks for the reply, Martin.<div><br></div><div>Since thi=
s issue started downstream of Snowball, there have been other discussions n=
ot on this list. Those discussions led to Robert Muir submitting pull reque=
sts to snowball (#177) and snowball-data (#23) on GitHub. He&#39;s opted to=
 map=C2=A0to the comma forms, which I agree with; obviously it could be wir=
ed the other way around if necessary.</div><div><br></div><div>Thanks for t=
he pointer to Italian! That was the model that probably would have helped m=
e write my own patch, but Robert already had a handle on it so I&#39;m happ=
y to let him take care of it.</div><div><br></div><div>=E2=80=94Trey</div><=
div><div><div dir=3D"ltr" class=3D"gmail_signature" data-smartmail=3D"gmail=
_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><d=
iv><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div d=
ir=3D"ltr"><div><div style=3D"font-size:small"><font color=3D"#999999"><spa=
n style=3D"font-size:12.8px">Trey Jones</span><br></font></div><span style=
=3D"font-size:12.8px"><font color=3D"#999999">Staff Computational Linguist,=
 Search Platform<br>Wikimedia Foundation</font></span><br></div><div><span =
style=3D"font-size:12.8px"><font color=3D"#999999">UTC=E2=80=935 / EST</fon=
t></span></div></div></div></div></div></div></div></div></div></div></div>=
</div></div></div></div></div><br></div></div><br><div class=3D"gmail_quote=
"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Mar 8, 2023 at 7:14=E2=80=
=AFPM Martin Porter &lt;<a href=3D"mailto:[email protected]">martin=
[email protected]</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quot=
e" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)=
;padding-left:1ex">Trey,<br>
<br>
Thank you for pointing this out. (Sorry for the late reply.) When I<br>
wrote the Romanian stemmer in 2007, the data I used certainly had s&amp;t<b=
r>
with cedilla, and I was not aware of the s&amp;t with comma form.<br>
<br>
Some normalisation of the text strings given to the stemmers is<br>
expected, and most especially the mapping of upper case letters to<br>
lower case. But sometimes a bit of normalisation is done in the<br>
stemmer. I think that is what should happen here.<br>
<br>
Have a look at the &#39;prelude&#39; routine in the Italian stemmer, and<br=
>
compare with the &#39;prelude&#39; routine in Romanian. In Italian, the rul=
es<br>
for the direction (acute or grave) of accents over vowels were quite<br>
relaxed, and varied from one publishing house to another. Perhaps the<br>
rules are tighter now, but there was no harm in mapping the acutes to<br>
graves, which is done in the first repeat loop. We put the same idea<br>
in the Romanian stemmer.<br>
<br>
The question is, do we map cedilla to comma-under, or comma-under to<br>
cedilla? Since comma-under is becoming standard, it would seem sensibe<br>
to map cedilla to comma-under, but even so, I would suggest doing it<br>
the other way round, so that users who have had to rely on the cedilla<br>
representation will not notice any change. The stemmed forms will have<br>
the cedilla form of s&amp;t, but these should be hidden from view anyway.<b=
r>
<br>
Olly Betts manages github Snowball, and we must see what he thinks.<br>
<br>
I could write the necessary patch, but it would take me some time to<br>
test it out.<br>
<br>
Martin<br>
</blockquote></div>

--000000000000f2497405f679a05e--


--===============3510698555313884280==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Snowball-discuss mailing list
[email protected]
https://lists.tartarus.org/mailman/listinfo/snowball-discuss

--===============3510698555313884280==--