Re: Proposing a 10.3.1 bug fix release

Thank you all for pointing out the test history, and the new unit test.

On Wed, Oct 1, 2025 at 10:10 AM Ignacio Vera <[email protected]> wrote:

> Thank you Ben, fix has been already pushed to the branch.
>
> On Wed, Oct 1, 2025 at 3:42 PM Benjamin Trent <[email protected]>
> wrote:
>
>> I volunteer as RM!
>>
>> On Wed, Oct 1, 2025 at 8:26 AM Adrien Grand <[email protected]> wrote:
>>
>>> I agree that this is a performance bug, though the fact that it only
>>> affects queries sorted by score where a term query runs on a field that
>>> doesn't index term frequencies suggests to me that it's more a problem for
>>> synthetic benchmarks than for real-world workloads. Most of the time, if
>>> you sort by score, then your `TermQuery`s would typically either have
>>> frequencies indexed (e.g. lexical search, learned sparse retrieval), or be
>>> applied as filters (e.g. filter on category for e-commerce) rather than as
>>> scoring clauses. (I'm not trying to say it's not a bug, it is a bug and I'm
>>> supportive of fixing it, it's an easy/safe fix.)
>>>
>>> Lucene's nightly benchmarks on the Wikipedia dataset only use term
>>> queries on fields that index term frequencies, so they couldn't have caught
>>> this. These two slowdowns are different. They happened when we switched the
>>> evaluation of conjunctive queries to a "score-first" approach, ie.
>>> computing scores on the n-th clause before checking if the [{N+1}-th clause
>>> matches (as opposed to only computing scores after verifying that all
>>> clauses match, which the code refers to as a "doc-first" approach).
>>> Empirically, score-first worked better on queries that match many
>>> documents, and doc-first worked better on queries that match few documents.
>>> Because of the filter that only matches 5% of docs, FilteredAndHighHigh
>>> doesn't match many docs.
>>>  - At annotation IN, conjunctive queries switched from a doc-first
>>> approach to a score-first approach.
>>>  - At annotation IO, conjunctive queries were refactored to evaluate
>>> blocks of documents where the min competitive score isn't set with a
>>> doc-first approach, and other blocks of documents with a score-first
>>> approach. This helped speed up queries that don't match many docs as per
>>> the above observation that doc-first approach works better for "sparse"
>>> queries.
>>>  - At annotation IR, we removed the alignment of this heuristic with
>>> blocks so that conjunctive queries moved to score-first immediately after a
>>> minimum competitive score was set, instead of waiting til the end of the
>>> current block before doing it. Hence the slowdown on sparse queries since
>>> we evaluated fewer docs with a doc-first approach and more docs with a
>>> score-first approach.
>>>  - Annotation IS was a kernel upgrade that reverted the slowdown that we
>>> had observed after the previous kernel upgrade at annotation II.
>>>
>>> You can tell from the benchmark results that it would be better to
>>> switch to score-first a bit later, we just haven't found the perfect
>>> heuristic yet.
>>>
>>> However, nightly benchmarks on the NYC taxi dataset did see this
>>> performance problem, since they run a disjunctive query on two term queries
>>> that run on fields that don't index term frequencies.
>>> https://benchmarks.mikemccandless.com/sparseResults.html. As noted
>>> elsewhere it wouldn't have told us about the bug on the 10.3 branch though
>>> since nightly benchmarks only run against the main branch and the main
>>> branch doesn't have this bug at the moment.
>>>
>>>
>>>
>>>
>>> On Wed, Oct 1, 2025 at 1:11 PM Benjamin Trent <[email protected]>
>>> wrote:
>>>
>>>> Michael,
>>>>
>>>> Lucene Benchmarks actually caught this. Then other optimizations made
>>>> the regression go away (with the original cause not being directly fixed).
>>>> Then later it was directly fixed, and never actually backported.
>>>>
>>>> Ignacio has indeed added a new test on main for this particular bug
>>>> (see: https://github.com/apache/lucene/pull/15266)
>>>>
>>>> Here is the benchmark in Lucene that caught the issue, it was partially
>>>> addressed, but regressed again on 'IR', and "fixed" accidentally by other
>>>> optimizations.
>>>>
>>>> https://benchmarks.mikemccandless.com/FilteredAndHighHigh.html
>>>>
>>>> [image: image.png]
>>>>
>>>> On Wed, Oct 1, 2025 at 6:43 AM Michael Sokolov <[email protected]>
>>>> wrote:
>>>>
>>>>> It sounds like this is needed. Do we also have any ideas about how we
>>>>> could have avoided this? Should we have some added test coverage, maybe a
>>>>> luceneutil enhancement? Did we already have a test that would have caught
>>>>> but we did not notice?
>>>>>
>>>>> On Wed, Oct 1, 2025, 4:03 AM Ignacio Vera <[email protected]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In Elasticsearch, we have detected some of our benchmarks to degrade
>>>>>> badly after upgrading to Lucene 10.3.0. We have nailed down the issue to a
>>>>>> bug <https://github.com/apache/lucene/pull/15263> in the posting
>>>>>> formats where the impacts returned when there are no frequencies is wrong,
>>>>>> leading to a miscalculation of scores.
>>>>>>
>>>>>> In order to fix this regression, I would like to propose a bug fix
>>>>>> release for 10.3 line. I will be pushing the bug fix today so we can start
>>>>>> the release process as soon as this is done as far as there are no
>>>>>> concerns. I will try to find a release manager or if I cannot I can do it
>>>>>> myself.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Ignacio
>>>>>>
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>
>>>
>>> --
>>> Adrien
>>>
>>