Re: Synonyms and searching

Hi,

Another way to do this is using Word Delimiter Filter and use "catenate" 
options. Be aware that you need special text tokenization (not use 
standard tokenizer, but instead WhitespaceTokenizer). This approach is 
common for product numbers.

To no break you "normal" analysis, it is often a good idea to have a 
separate field with the "alternative" analysis, which also applies other 
things like stemming and search in both fields using a Dismax Query, 
ranking the "normal" standard analyzer based query field higher.

Uwe

Am 05.03.2025 um 20:19 schrieb Mikhail Khludnev:
> Hello Trevor.
>
> Maintaining such a synonym map is too much of a burden.
> One idea: sticks words together with "" separator with
> https://lucene.apache.org/core/8_0_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilter.html
> Another idea, the opposite breaks user's words via dictionary
> https://lucene.apache.org/core/6_6_1/analyzers-common/org/apache/lucene/analysis/compound/DictionaryCompoundWordTokenFilter.html
> However, it's actually a suggester's duty
> https://lucene.apache.org/core//8_0_0/suggest/org/apache/lucene/search/spell/WordBreakSpellChecker.html
> however it's aside of the main search flow.
>
>
> On Wed, Mar 5, 2025 at 5:28 PM Trevor Nicholls <[email protected]>
> wrote:
>
>> I don't know if I have completely the wrong idea or not, hopefully somebody
>> can point out where I have got this wrong
>>
>>
>>
>> I am indexing technical documentation; the content contains strings like
>> "http_proxy_server". When building the index my analyzer breaks this into
>> the tokens "http", "proxy" and "server". It generatees the same tokens for
>> "http.proxy.server"; constructions like this are also common in the
>> documents.
>>
>>
>>
>> At the moment the application is using Lucene 8.6.3.
>>
>>
>>
>> If the document contains "http_proxy_server" the user can search for "http
>> proxy server", "http.proxy.server" or "http proxy server" and all will
>> match.
>>
>>
>>
>> However, I am trying to construct the index and the search so that if the
>> user searches for e.g. "http proxyserver" they also find a match. I thought
>> it would be sufficient to add an entry to the synonym map specifying that
>> "http proxy" and "httpproxy" are synonyms, and likewise "proxy server" and
>> "proxyserver". (When adding multiple-word phrases the spaces are replaced
>> by
>> SynonymMap.WORD_SEPARATOR).
>>
>>
>>
>> The analyzer incorporates the synonym map when building the index, but not
>> when searching - the synonyms (both words and phrases) should already be in
>> the index so a user's search pattern should not need to be extended by
>> them.
>>
>>
>>
>> Unfortunately this doesn't appear to be working as I expected. If a user
>> searches for "httpproxy" or "proxyserver" nothing is matched.
>>
>> When I print the tokens in the stream emitted by the analyser, I can see
>> all
>> the word for word synonyms output (e.g. if the content contains "license",
>> the emerging tokens include both "licence" and "license"), but the phrase
>> substitutions are not. "http", "proxy" and "server " are there, but none of
>> the conjunctions appear.
>>
>>
>>
>> I don't think synonym replacement should be occurring at search time, if
>> only for performance reasons, but what have I missed in how this should
>> work? Am I chasing the impossible dream?
>>
>>
>>
>> cheers
>>
>> T
>>
>>
>>
>>
>>
>>
-- 
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: [email protected]