Re: Proposing a 10.3.1 bug fix release

Thank you Ben, fix has been already pushed to the branch.

On Wed, Oct 1, 2025 at 3:42 PM Benjamin Trent <[email protected]> wrote:

> I volunteer as RM!
>
> On Wed, Oct 1, 2025 at 8:26 AM Adrien Grand <[email protected]> wrote:
>
>> I agree that this is a performance bug, though the fact that it only
>> affects queries sorted by score where a term query runs on a field that
>> doesn't index term frequencies suggests to me that it's more a problem for
>> synthetic benchmarks than for real-world workloads. Most of the time, if
>> you sort by score, then your `TermQuery`s would typically either have
>> frequencies indexed (e.g. lexical search, learned sparse retrieval), or be
>> applied as filters (e.g. filter on category for e-commerce) rather than as
>> scoring clauses. (I'm not trying to say it's not a bug, it is a bug and I'm
>> supportive of fixing it, it's an easy/safe fix.)
>>
>> Lucene's nightly benchmarks on the Wikipedia dataset only use term
>> queries on fields that index term frequencies, so they couldn't have caught
>> this. These two slowdowns are different. They happened when we switched the
>> evaluation of conjunctive queries to a "score-first" approach, ie.
>> computing scores on the n-th clause before checking if the [{N+1}-th clause
>> matches (as opposed to only computing scores after verifying that all
>> clauses match, which the code refers to as a "doc-first" approach).
>> Empirically, score-first worked better on queries that match many
>> documents, and doc-first worked better on queries that match few documents.
>> Because of the filter that only matches 5% of docs, FilteredAndHighHigh
>> doesn't match many docs.
>>  - At annotation IN, conjunctive queries switched from a doc-first
>> approach to a score-first approach.
>>  - At annotation IO, conjunctive queries were refactored to evaluate
>> blocks of documents where the min competitive score isn't set with a
>> doc-first approach, and other blocks of documents with a score-first
>> approach. This helped speed up queries that don't match many docs as per
>> the above observation that doc-first approach works better for "sparse"
>> queries.
>>  - At annotation IR, we removed the alignment of this heuristic with
>> blocks so that conjunctive queries moved to score-first immediately after a
>> minimum competitive score was set, instead of waiting til the end of the
>> current block before doing it. Hence the slowdown on sparse queries since
>> we evaluated fewer docs with a doc-first approach and more docs with a
>> score-first approach.
>>  - Annotation IS was a kernel upgrade that reverted the slowdown that we
>> had observed after the previous kernel upgrade at annotation II.
>>
>> You can tell from the benchmark results that it would be better to switch
>> to score-first a bit later, we just haven't found the perfect heuristic yet.
>>
>> However, nightly benchmarks on the NYC taxi dataset did see this
>> performance problem, since they run a disjunctive query on two term queries
>> that run on fields that don't index term frequencies.
>> https://benchmarks.mikemccandless.com/sparseResults.html. As noted
>> elsewhere it wouldn't have told us about the bug on the 10.3 branch though
>> since nightly benchmarks only run against the main branch and the main
>> branch doesn't have this bug at the moment.
>>
>>
>>
>>
>> On Wed, Oct 1, 2025 at 1:11 PM Benjamin Trent <[email protected]>
>> wrote:
>>
>>> Michael,
>>>
>>> Lucene Benchmarks actually caught this. Then other optimizations made
>>> the regression go away (with the original cause not being directly fixed).
>>> Then later it was directly fixed, and never actually backported.
>>>
>>> Ignacio has indeed added a new test on main for this particular bug
>>> (see: https://github.com/apache/lucene/pull/15266)
>>>
>>> Here is the benchmark in Lucene that caught the issue, it was partially
>>> addressed, but regressed again on 'IR', and "fixed" accidentally by other
>>> optimizations.
>>>
>>> https://benchmarks.mikemccandless.com/FilteredAndHighHigh.html
>>>
>>> [image: image.png]
>>>
>>> On Wed, Oct 1, 2025 at 6:43 AM Michael Sokolov <[email protected]>
>>> wrote:
>>>
>>>> It sounds like this is needed. Do we also have any ideas about how we
>>>> could have avoided this? Should we have some added test coverage, maybe a
>>>> luceneutil enhancement? Did we already have a test that would have caught
>>>> but we did not notice?
>>>>
>>>> On Wed, Oct 1, 2025, 4:03 AM Ignacio Vera <[email protected]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> In Elasticsearch, we have detected some of our benchmarks to degrade
>>>>> badly after upgrading to Lucene 10.3.0. We have nailed down the issue to a
>>>>> bug <https://github.com/apache/lucene/pull/15263> in the posting
>>>>> formats where the impacts returned when there are no frequencies is wrong,
>>>>> leading to a miscalculation of scores.
>>>>>
>>>>> In order to fix this regression, I would like to propose a bug fix
>>>>> release for 10.3 line. I will be pushing the bug fix today so we can start
>>>>> the release process as soon as this is done as far as there are no
>>>>> concerns. I will try to find a release manager or if I cannot I can do it
>>>>> myself.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Ignacio
>>>>>
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>
>>
>> --
>> Adrien
>>
>