Re: full-text search : How to improve ?

Hi Ralf:

In message <[email protected]>,
Ralf Schlatterbeck writes:
>On Mon, Dec 20, 2021 at 01:11:18PM -0500, John P. Rouillard wrote:
>> >How can the full-text search be improved to distinguish for example between
>> > tvmRel41Acg16A and 
>> > tvmRel41Acg19A ?
>> 
>> Hmm, that's an interesting question. What do you mean by "improved".
>> Although your "words" aren't really words, they are a run of \w (word)
>> characters. Which according to my brief look at the code should be
>> included as items in the index.
>> 
>> What happens when you do a full search for tvmRel41Acg16A? Does it
>> show issues with the text in the messages?
>
>On first glance this also looks short enough to be below any hard-coded
>word length limits. There is a hardcoded limit in indexer_common.py:
>self.maxlength = 25
>
>But I'm not so sure if the word-delimiting code would not break up at
>digit boundaries, John, what do you think? It's a long time I've looked
>at the indexer code...

I took a fast look before responding. I agree with the word length,
his examples are 14 characters. I had to go looking for the max length
of 25.

For the rdbms backend indexer_rdbms.py::add_text is the method to look
at I think.

    wordlist = [u2s(w)
      for w in re.findall(r'(?u)\b\w{%d,%d}\b'
             % (self.minlength, self.maxlength), text)]

I am reading the regexp as a run of delimited word (\w = [A-Za-z0-9_])
characters between min and max length (2,25). IIUC (?u) should set
it to unicode match so \w may have additional word characters, but
unless locale changes things ascii alphanumeric plus _ should be
captured. So at least with the native indexer the entire examples
should be treated as a word.

The xapian indexer code uses the same regexp for tokenizing except
without unicode. I am not sure what whoosh.add_document does.
 
--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.