Re: Possible bug using FLAG_WORD_BREAKS with fullwidth Unicode codepoints
"Robert Stepanek" <[email protected]> Wed, 10 Jan 2024 09:02:03 +0100
| Newsgroups | gmane.comp.search.xapian.devel |
|---|---|
| Message-ID | <[email protected]> |
--7df9dee51c584c74b88274a8e2136fdd
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable
On Tue, Jan 9, 2024, at 3:28 AM, Olly Betts wrote:
> Thanks, that looks good - now merged.
Thanks!
> Did you already check the other ranges for cased letters? I can but if
> you have already there's not much point.
I did not. If you find time, that'd be great. Otherwise I can make room =
for it in the next days.
> > The fullwidth "=EF=BD=88=EF=BD=85=EF=BD=8C=EF=BD=8C=EF=BD=8F =EF=BC=8C=
=EF=BD=97=EF=BD=8F=EF=BD=92=EF=BD=8C=EF=BD=84" tests suggests to me that
> > either Xapian should allow for Unicode normalization, or application
> > developers must take care of this before indexing.
>=20
> We currently leave it to the API user to normalise Unicode
> representation, though maybe we should provide support for doing so.
Thinking some more about this, I think it's sane to leave this out of Xa=
pian. Unless there is also some bookkeeping added within Xapian to tell =
which normalisation was applied to terms, which can get complex for sub-=
databases or mixed normalisations within one database.
--7df9dee51c584c74b88274a8e2136fdd
Content-Type: text/html;charset=utf-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html><html><head><title></title><style type=3D"text/css">p.Mso=
Normal,p.MsoNoSpacing{margin:0}</style></head><body><div>On Tue, Jan 9, =
2024, at 3:28 AM, Olly Betts wrote:<br></div><blockquote type=3D"cite" i=
d=3D"qt" style=3D""><div>Thanks, that looks good - now merged.<br></div>=
</blockquote><div><br></div><div>Thanks!<br></div><div><br></div><blockq=
uote type=3D"cite" id=3D"qt" style=3D""><div>Did you already check the o=
ther ranges for cased letters? I can but if<br></div><div>you have=
already there's not much point.<br></div></blockquote><div><br></div><d=
iv>I did not. If you find time, that'd be great. Otherwise I can make ro=
om for it in the next days.<br></div><div><br></div><blockquote type=3D"=
cite" id=3D"qt" style=3D""><div>> The fullwidth "=EF=BD=88=EF=BD=85=EF=
=BD=8C=EF=BD=8C=EF=BD=8F =EF=BC=8C=EF=BD=97=EF=BD=8F=EF=BD=92=EF=BD=8C=EF=
=BD=84" tests suggests to me that<br></div><div>> either Xapian shoul=
d allow for Unicode normalization, or application<br></div><div>> dev=
elopers must take care of this before indexing.<br></div><div><br></div>=
<div>We currently leave it to the API user to normalise Unicode<br></div=
><div>representation, though maybe we should provide support for doing s=
o.<br></div></blockquote><div><br></div><div>Thinking some more about th=
is, I think it's sane to leave this out of Xapian. Unless there is also =
some bookkeeping added within Xapian to tell which normalisation was app=
lied to terms, which can get complex for sub-databases or mixed normalis=
ations within one database.<br></div><div><br></div></body></html>
--7df9dee51c584c74b88274a8e2136fdd--