Re: Possible bug using FLAG_WORD_BREAKS with fullwidth Unicode codepoints
"Robert Stepanek" <[email protected]> Mon, 08 Jan 2024 14:01:46 +0100
| Newsgroups | gmane.comp.search.xapian.devel |
|---|---|
| Message-ID | <[email protected]> |
--1b0a25f83b6e4f27acffe0886ca4de83
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable
On Sun, Jan 7, 2024, at 7:45 PM, Olly Betts wrote:
> I've restarted trac.
I now created a pull request: https://github.com/xapian/xapian/pull/329 =
Should I create a trac issue, too?
> Assuming the latter is valid, just removing this block (or removing the
> parts of it which are Lu or Ll) should fix the problem as then
> tokenisation will switch mode - I tried this and it fixes your case at
> least:
Removing the whole block will cause word-breaker to not correctly handle=
halfwidth Katakana, such as "=EF=BD=BC=EF=BD=B0=EF=BD=BB=EF=BD=B2=EF=BE=
=84=EF=BE=9E=EF=BE=97=EF=BD=B2=EF=BE=85=EF=BD=B0" which it would treat a=
s a single term, whereas it should be two: =EF=BD=BC=EF=BD=B0=EF=BD=BB=EF=
=BD=B2=EF=BE=84=EF=BE=9Eand =EF=BE=97=EF=BD=B2=EF=BE=85=EF=BD=B0).
My pull request causes word-breaker to only handle halfwidth Katakana an=
d Hangul codepoints as unbroken script and treats Latin characters, numb=
ers, symbols and punctuation as broken script. There's a couple of unit =
tests that check for this.
diff --git a/xapian-core/queryparser/word-breaker.cc b/xapian-core/query=
parser/word-breaker.cc
index 8108523ccd53..6122dcdccc97 100644
--- a/xapian-core/queryparser/word-breaker.cc
+++ b/xapian-core/queryparser/word-breaker.cc
@@ -102,8 +102,10 @@ is_unbroken_script(unsigned p)
0xF900 - 1, 0xFAFF,
// FE30..FE4F; CJK Compatibility Forms
0xFE30 - 1, 0xFE4F,
- // FF00..FFEF; Halfwidth and Fullwidth Forms
- 0xFF00 - 1, 0xFFEF,
+ // FF00..FF60: Fullwidth Numbers, Latin Characters, Punctuation
+ // FF61..FF64: Halfwidth Punctuation
+ 0xFF65 - 1, 0xFFDC, // Halfwidth Katakana and Hangul
+ // FFE0..FFEF; Fullwidth and Halfwidth Symbols
The fullwidth "=EF=BD=88=EF=BD=85=EF=BD=8C=EF=BD=8C=EF=BD=8F =EF=BC=8C=EF=
=BD=97=EF=BD=8F=EF=BD=92=EF=BD=8C=EF=BD=84" tests suggests to me that ei=
ther Xapian should allow for Unicode normalization, or application devel=
opers must take care of this before indexing.
--1b0a25f83b6e4f27acffe0886ca4de83
Content-Type: text/html;charset=utf-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html><html><head><title></title><style type=3D"text/css">p.Mso=
Normal,p.MsoNoSpacing{margin:0}</style></head><body><div>On Sun, Jan 7, =
2024, at 7:45 PM, Olly Betts wrote:<br></div><blockquote type=3D"cite" i=
d=3D"qt" style=3D""><div>I've restarted trac.<br></div></blockquote><div=
><br></div><div>I now created a pull request: <a href=3D"https://github.=
com/xapian/xapian/pull/329">https://github.com/xapian/xapian/pull/329</a=
> Should I create a trac issue, too?<br></div><div><br></div><blockquote=
type=3D"cite" id=3D"qt" style=3D""><div>Assuming the latter is valid, j=
ust removing this block (or removing the<br></div><div>parts of it which=
are Lu or Ll) should fix the problem as then<br></div><div>tokenisation=
will switch mode - I tried this and it fixes your case at<br></div><div=
>least:<br></div></blockquote><div><br></div><div>Removing the whole blo=
ck will cause word-breaker to not correctly handle halfwidth Katakana, s=
uch as "=EF=BD=BC=EF=BD=B0=EF=BD=BB=EF=BD=B2=EF=BE=84=EF=BE=9E=EF=BE=97=EF=
=BD=B2=EF=BE=85=EF=BD=B0" which it would treat as a single term, whereas=
it should be two: =EF=BD=BC=EF=BD=B0=EF=BD=BB=EF=BD=B2=EF=BE=84=EF=BE=9E=
and =EF=BE=97=EF=BD=B2=EF=BE=85=EF=BD=B0).<br></div><div><br></div=
><div>My pull request causes word-breaker to only handle halfwidth Katak=
ana and Hangul codepoints as unbroken script and treats Latin characters=
, numbers, symbols and punctuation as broken script. There's a couple of=
unit tests that check for this.<br></div><div><br></div><div>diff --git=
a/xapian-core/queryparser/word-breaker.cc b/xapian-core/queryparser/wor=
d-breaker.cc<br></div><div>index 8108523ccd53..6122dcdccc97 100644<br></=
div><div>--- a/xapian-core/queryparser/word-breaker.cc<br></div><div>+++=
b/xapian-core/queryparser/word-breaker.cc<br></div><div>@@ -102,8 +102,=
10 @@ is_unbroken_script(unsigned p)<br></div><div> &nb=
sp; 0xF900 - 1, 0xFAFF,<br></div><div> &nbs=
p; // FE30..FE4F; CJK Compatibility Forms<br></d=
iv><div> 0xFE30 - 1, 0xFE4F,<b=
r></div><div>- // FF00..FFEF; Halfwi=
dth and Fullwidth Forms<br></div><div>- &nb=
sp; 0xFF00 - 1, 0xFFEF,<br></div><div>+ &nb=
sp; // FF00..FF60: Fullwidth Numbers, Latin Characters, Punctuation<br><=
/div><div>+ // FF61..FF64: Halfwidth=
Punctuation<br></div><div>+ 0xFF65 =
- 1, 0xFFDC, // Halfwidth Katakana and Hangul<br></div><div>+  =
; // FFE0..FFEF; Fullwidth and Halfwidth Symbols=
<br></div><div><br></div><div>The fullwidth "=EF=BD=88=EF=BD=85=EF=BD=8C=
=EF=BD=8C=EF=BD=8F =EF=BC=8C=EF=BD=97=EF=BD=8F=EF=BD=92=EF=BD=8C=EF=BD=84=
" tests suggests to me that either Xapian should allow for Unicode norma=
lization, or application developers must take care of this before indexi=
ng.<br></div></body></html>
--1b0a25f83b6e4f27acffe0886ca4de83--