Re: Removing été from French stop words
Philippe Ouellet <[email protected]> Thu, 16 Apr 2020 06:46:41 -0400
| Newsgroups | gmane.comp.search.snowball |
|---|---|
| Message-ID | <[email protected]> |
--===============8849154819750874283== Content-Type: multipart/alternative; boundary="Apple-Mail=_2669EFFA-63D6-483A-85DC-DA03CD73FDA5" --Apple-Mail=_2669EFFA-63D6-483A-85DC-DA03CD73FDA5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 What is the git repository url? I can=E2=80=99t find the file your are = talking about in https://github.com/snowballstem/snowball = <https://github.com/snowballstem/snowball> -- Philippe Ouellet Web Developer https://camellia-sinensis.com > On Apr 15, 2020, at 23:25, Olly Betts <[email protected]> wrote: >=20 > On Wed, Apr 15, 2020 at 08:14:55PM -0400, Philippe Ouellet wrote: >> Could you give me the link to the file in the repo? I have no idea = where that is. >=20 > algorithms/french/stop.txt >=20 > "git grep" is good for such situations: >=20 > $ git grep '=C3=A9t=C3=A9' > algorithms/french/stop.txt:=C3=A9t=C3=A9 > algorithms/french/stop.txt:=C3=A9t=C3=A9e > algorithms/french/stop.txt:=C3=A9t=C3=A9es > algorithms/french/stop.txt:=C3=A9t=C3=A9s >=20 >> I only did notice =C3=A9t=C3=A9, because one of our product name = contains that >> word, but you are right about aura and avions.=20 >>=20 >> Do we need to remove avions from the stop word, if it get changed to >> its singular form during analysis? >=20 > Yes - the stop word lists as shipped are intended for stopping before > stemming. >=20 > You could run each entry through the stemmer to get a post-stemming > stopword list, though it will tend to have more issues with stopping > words that are wanted because each entry will now remove all words = which > stem to the same thing as a stopword - e.g. "avions" stems to "avion" > and so both "avion" and "avions" would get treated as stopwords if you > stop after stemming. >=20 >> I am having second thought: removing =C3=A9t=C3=A9 could have a great = impact on >> the search result, someone searching for =E2=80=9Csummer=E2=80=9D = would result in >> finding all results containing the past tense form of =E2=80=9Cto = be=E2=80=9D: the >> impact is huge. >=20 > That's still better than not being able to search for "summer" at all. > There are similar issues in English and we explicitly don't include > words like "can", "may", "will", "must", etc in > algorithms/english/stop.txt (as noted in the comments). >=20 > For such words a search for just the word itself will tend to have > somewhat poor results (a French document that's actually talking about > "summer" will probably mention "=C3=A9t=C3=A9" more on average and so = tend to rank > higher, but there are likely to be some documents that aren't about > "summer" at all ranked above some that are.) >=20 > Searches for more than one word are likely to fare better. Especially > so if the ranking favours cases where the search terms appear close > together. >=20 >> Is there a way to make =E2=80=9Ca =C3=A9t=C3=A9=E2=80=9D the stop = word instead? >=20 > Not in the scope of a simple list of stop words. I guess you could > try part of speech tagging to try to differentiate the cases where > the stop word and the homonym are different parts of speech (which > they typically seem to be). >=20 > Cheers, > Olly --Apple-Mail=_2669EFFA-63D6-483A-85DC-DA03CD73FDA5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">What = is the git repository url? I can=E2=80=99t find the file your are = talking about in <a href=3D"https://github.com/snowballstem/snowball"= class=3D"">https://github.com/snowballstem/snowball</a><div = class=3D""><br class=3D""><div class=3D""> <div dir=3D"auto" style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0); letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: = break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" = class=3D""><div>--<br class=3D"">Philippe Ouellet<br class=3D"">Web = Developer</div><div><a href=3D"https://camellia-sinensis.com" = class=3D"">https://camellia-sinensis.com</a></div></div> </div> <div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Apr 15, 2020, at 23:25, Olly Betts <<a = href=3D"mailto:[email protected]" class=3D"">[email protected]</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = class=3D"">On Wed, Apr 15, 2020 at 08:14:55PM -0400, Philippe Ouellet = wrote:<br class=3D""><blockquote type=3D"cite" class=3D"">Could you give = me the link to the file in the repo? I have no idea where that is.<br = class=3D""></blockquote><br class=3D"">algorithms/french/stop.txt<br = class=3D""><br class=3D"">"git grep" is good for such situations:<br = class=3D""><br class=3D"">$ git grep '=C3=A9t=C3=A9'<br = class=3D"">algorithms/french/stop.txt:=C3=A9t=C3=A9<br = class=3D"">algorithms/french/stop.txt:=C3=A9t=C3=A9e<br = class=3D"">algorithms/french/stop.txt:=C3=A9t=C3=A9es<br = class=3D"">algorithms/french/stop.txt:=C3=A9t=C3=A9s<br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">I only did notice =C3=A9t=C3= =A9, because one of our product name contains that<br class=3D"">word, = but you are right about aura and avions. <br class=3D""><br class=3D"">Do = we need to remove avions from the stop word, if it get changed to<br = class=3D"">its singular form during analysis?<br = class=3D""></blockquote><br class=3D"">Yes - the stop word lists as = shipped are intended for stopping before<br class=3D"">stemming.<br = class=3D""><br class=3D"">You could run each entry through the stemmer = to get a post-stemming<br class=3D"">stopword list, though it will tend = to have more issues with stopping<br class=3D"">words that are wanted = because each entry will now remove all words which<br class=3D"">stem to = the same thing as a stopword - e.g. "avions" stems to "avion"<br = class=3D"">and so both "avion" and "avions" would get treated as = stopwords if you<br class=3D"">stop after stemming.<br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">I am having second = thought: removing =C3=A9t=C3=A9 could have a great impact on<br = class=3D"">the search result, someone searching for =E2=80=9Csummer=E2=80=9D= would result in<br class=3D"">finding all results containing the past = tense form of =E2=80=9Cto be=E2=80=9D: the<br class=3D"">impact is = huge.<br class=3D""></blockquote><br class=3D"">That's still better than = not being able to search for "summer" at all.<br class=3D"">There are = similar issues in English and we explicitly don't include<br = class=3D"">words like "can", "may", "will", "must", etc in<br = class=3D"">algorithms/english/stop.txt (as noted in the comments).<br = class=3D""><br class=3D"">For such words a search for just the word = itself will tend to have<br class=3D"">somewhat poor results (a French = document that's actually talking about<br class=3D"">"summer" will = probably mention "=C3=A9t=C3=A9" more on average and so tend to rank<br = class=3D"">higher, but there are likely to be some documents that aren't = about<br class=3D"">"summer" at all ranked above some that are.)<br = class=3D""><br class=3D"">Searches for more than one word are likely to = fare better. Especially<br class=3D"">so if the ranking favours = cases where the search terms appear close<br class=3D"">together.<br = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D"">Is there = a way to make =E2=80=9Ca =C3=A9t=C3=A9=E2=80=9D the stop word = instead?<br class=3D""></blockquote><br class=3D"">Not in the scope of a = simple list of stop words. I guess you could<br class=3D"">try = part of speech tagging to try to differentiate the cases where<br = class=3D"">the stop word and the homonym are different parts of speech = (which<br class=3D"">they typically seem to be).<br class=3D""><br = class=3D"">Cheers,<br class=3D""> Olly<br = class=3D""></div></div></blockquote></div><br = class=3D""></div></body></html>= --Apple-Mail=_2669EFFA-63D6-483A-85DC-DA03CD73FDA5-- --===============8849154819750874283== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Snowball-discuss mailing list [email protected] https://lists.tartarus.org/mailman/listinfo/snowball-discuss --===============8849154819750874283==--