Possible bug using FLAG_WORD_BREAKS with fullwidth Unicode codepoints
"Robert Stepanek" <[email protected]> Thu, 04 Jan 2024 17:50:22 +0100
| Newsgroups | gmane.comp.search.xapian.devel |
|---|---|
| Message-ID | <[email protected]> |
--2b917c80487a4685bbb1efd94a9cca1e
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable
I think I found a bug in Xapian 1.5 when using FLAG_WORD_BREAKS for inpu=
t that contains characters in Unicode Halfwidth and Fullwidth Forms (htt=
ps://unicode.org/charts/PDF/UFF00.pdf).
Since I am undecided yet if and how to fix this in Xapian I haven't come=
up with a pull request. Because trac currently is offline, I could not =
file a bug. I hope it's OK to post my analysis here first, I'll be happy=
to follow up reporting that bug proper later (should we conclude that i=
t actually is a bug).
Imagine indexing the following Japanese text "=E4=B8=89=E8=8F=B1=EF=BC=B5=
=EF=BC=A6=EF=BC=AA=E3=83=95=E3=82=A1=E3=82=AF=E3=82=BF=E3=83=BC=E6=A0=AA=
=E5=BC=8F=E4=BC=9A=E7=A4=BE" which in English denotes the "Mitsubishi UF=
J Factors Limited" bank.
Using word segmentation in Xapian 1.5 this causes the following terms to=
get indexed:
=E3=83=95=E3=82=A1=E3=82=AF=E3=82=BF=E3=83=BC
=E4=B8=89=E8=8F=B1
=E6=A0=AA=E5=BC=8F=E4=BC=9A=E7=A4=BE
=EF=BC=B5=EF=BC=A6=EF=BC=AA
Note that last term, which starts with FULLWIDTH LATIN CAPITAL LETTER U'=
(U+FF35). Xapian's Unicode library correctly assigns this the UPPERCASE=
_LETTER category and indexes this verbatim.
However, querying for =EF=BC=B5=EF=BC=A6=EF=BC=AA produces the query Que=
ry(=EF=BD=95=EF=BD=86=EF=BD=8A@1). That is, it queries for the lowercase=
form which seems to be the result of unconditional lower-casing at http=
s://github.com/xapian/xapian/blob/master/xapian-core/queryparser/querypa=
rser.lemony#L1459. As a result, the query returns no result.
I have written code that demonstrates this at https://gist.github.com/rs=
to/168a61536793e10a0a07c3920977e5eb
Now, I think that much of this issue can be prevented by normalizing bot=
h indexed text and queries before passing them over the Xapian, but this=
requires to rewrite indexes so isn't necessarily a quick fix. As a work=
around, I chose to detect such queries and query for both the lower-case=
d and original uppercase forms in our systems.
Still, I do think it is a bug for Xapian not to return a result when que=
rying for a term that's verbatim in the original input and the database.=
Should you agree I will be happy to discuss how to fix this and might c=
ome up with a pull request once we agreed on a solution.
--2b917c80487a4685bbb1efd94a9cca1e
Content-Type: text/html;charset=utf-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html><html><head><title></title><style type=3D"text/css">p.Mso=
Normal,p.MsoNoSpacing{margin:0}
p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div>I think I =
found a bug in Xapian 1.5 when using FLAG_WORD_BREAKS for input that con=
tains characters in Unicode Halfwidth and Fullwidth Forms (<a href=3D"ht=
tps://unicode.org/charts/PDF/UFF00.pdf">https://unicode.org/charts/PDF/U=
FF00.pdf</a>).<br></div><div><br></div><div>Since I am undecided yet if =
and how to fix this in Xapian I haven't come up with a pull request. Bec=
ause trac currently is offline, I could not file a bug. I hope it's OK t=
o post my analysis here first, I'll be happy to follow up reporting that=
bug proper later (should we conclude that it actually is a bug).<br></d=
iv><div><br></div><div>Imagine indexing the following Japanese text "=E4=
=B8=89=E8=8F=B1=EF=BC=B5=EF=BC=A6=EF=BC=AA=E3=83=95=E3=82=A1=E3=82=AF=E3=
=82=BF=E3=83=BC=E6=A0=AA=E5=BC=8F=E4=BC=9A=E7=A4=BE" which in English de=
notes the "<span class=3D"js-company-name">Mitsubishi UFJ Factors Limite=
d" bank.</span><br></div><div><br></div><div>Using word segmentation in =
Xapian 1.5 this causes the following terms to get indexed:<br></div><div=
>=E3=83=95=E3=82=A1=E3=82=AF=E3=82=BF=E3=83=BC<br></div><div>=E4=B8=89=E8=
=8F=B1<br></div><div>=E6=A0=AA=E5=BC=8F=E4=BC=9A=E7=A4=BE<br></div><div>=
=EF=BC=B5=EF=BC=A6=EF=BC=AA<br></div><div><br></div><div>Note that last =
term, which starts with FULLWIDTH LATIN CAPITAL LETTER U' (U+FF35). Xapi=
an's Unicode library correctly assigns this the UPPERCASE_LETTER categor=
y and indexes this verbatim.<br></div><div><br></div><div>However, query=
ing for =EF=BC=B5=EF=BC=A6=EF=BC=AA produces the query Query(=EF=BD=95=EF=
=BD=86=EF=BD=8A@1). That is, it queries for the lowercase form which see=
ms to be the result of unconditional lower-casing at <a href=3D"https://=
github.com/xapian/xapian/blob/master/xapian-core/queryparser/queryparser=
.lemony#L1459">https://github.com/xapian/xapian/blob/master/xapian-core/=
queryparser/queryparser.lemony#L1459</a>. As a result, the query returns=
no result.<br></div><div><br></div><div>I have written code that demons=
trates this at <a href=3D"https://gist.github.com/rsto/168a61536793e10a0=
a07c3920977e5eb">https://gist.github.com/rsto/168a61536793e10a0a07c39209=
77e5eb</a><br></div><div><br></div><div>Now, I think that much of this i=
ssue can be prevented by normalizing both indexed text and queries befor=
e passing them over the Xapian, but this requires to rewrite indexes so =
isn't necessarily a quick fix. As a workaround, I chose to detect such q=
ueries and query for both the lower-cased and original uppercase forms i=
n our systems.<br></div><div><br></div><div>Still, I do think it is a bu=
g for Xapian not to return a result when querying for a term that's verb=
atim in the original input and the database. Should you agree I will be =
happy to discuss how to fix this and might come up with a pull request o=
nce we agreed on a solution.<br></div></body></html>
--2b917c80487a4685bbb1efd94a9cca1e--