Re: Proposing a 10.3.1 bug fix release

I am starting the release, right now the only bug fix is
https://github.com/apache/lucene/pull/15263

Please let me know if we want to add anything else. Otherwise I will likely
have a candidate ready for voting tomorrow.

On Wed, Oct 1, 2025 at 12:22 PM Michael Sokolov <[email protected]> wrote:

> Thank you all for pointing out the test history, and the new unit test.
>
> On Wed, Oct 1, 2025 at 10:10 AM Ignacio Vera <[email protected]> wrote:
>
>> Thank you Ben, fix has been already pushed to the branch.
>>
>> On Wed, Oct 1, 2025 at 3:42 PM Benjamin Trent <[email protected]>
>> wrote:
>>
>>> I volunteer as RM!
>>>
>>> On Wed, Oct 1, 2025 at 8:26 AM Adrien Grand <[email protected]> wrote:
>>>
>>>> I agree that this is a performance bug, though the fact that it only
>>>> affects queries sorted by score where a term query runs on a field that
>>>> doesn't index term frequencies suggests to me that it's more a problem for
>>>> synthetic benchmarks than for real-world workloads. Most of the time, if
>>>> you sort by score, then your `TermQuery`s would typically either have
>>>> frequencies indexed (e.g. lexical search, learned sparse retrieval), or be
>>>> applied as filters (e.g. filter on category for e-commerce) rather than as
>>>> scoring clauses. (I'm not trying to say it's not a bug, it is a bug and I'm
>>>> supportive of fixing it, it's an easy/safe fix.)
>>>>
>>>> Lucene's nightly benchmarks on the Wikipedia dataset only use term
>>>> queries on fields that index term frequencies, so they couldn't have caught
>>>> this. These two slowdowns are different. They happened when we switched the
>>>> evaluation of conjunctive queries to a "score-first" approach, ie.
>>>> computing scores on the n-th clause before checking if the [{N+1}-th clause
>>>> matches (as opposed to only computing scores after verifying that all
>>>> clauses match, which the code refers to as a "doc-first" approach).
>>>> Empirically, score-first worked better on queries that match many
>>>> documents, and doc-first worked better on queries that match few documents.
>>>> Because of the filter that only matches 5% of docs, FilteredAndHighHigh
>>>> doesn't match many docs.
>>>>  - At annotation IN, conjunctive queries switched from a doc-first
>>>> approach to a score-first approach.
>>>>  - At annotation IO, conjunctive queries were refactored to evaluate
>>>> blocks of documents where the min competitive score isn't set with a
>>>> doc-first approach, and other blocks of documents with a score-first
>>>> approach. This helped speed up queries that don't match many docs as per
>>>> the above observation that doc-first approach works better for "sparse"
>>>> queries.
>>>>  - At annotation IR, we removed the alignment of this heuristic with
>>>> blocks so that conjunctive queries moved to score-first immediately after a
>>>> minimum competitive score was set, instead of waiting til the end of the
>>>> current block before doing it. Hence the slowdown on sparse queries since
>>>> we evaluated fewer docs with a doc-first approach and more docs with a
>>>> score-first approach.
>>>>  - Annotation IS was a kernel upgrade that reverted the slowdown that
>>>> we had observed after the previous kernel upgrade at annotation II.
>>>>
>>>> You can tell from the benchmark results that it would be better to
>>>> switch to score-first a bit later, we just haven't found the perfect
>>>> heuristic yet.
>>>>
>>>> However, nightly benchmarks on the NYC taxi dataset did see this
>>>> performance problem, since they run a disjunctive query on two term queries
>>>> that run on fields that don't index term frequencies.
>>>> https://benchmarks.mikemccandless.com/sparseResults.html. As noted
>>>> elsewhere it wouldn't have told us about the bug on the 10.3 branch though
>>>> since nightly benchmarks only run against the main branch and the main
>>>> branch doesn't have this bug at the moment.
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Oct 1, 2025 at 1:11 PM Benjamin Trent <[email protected]>
>>>> wrote:
>>>>
>>>>> Michael,
>>>>>
>>>>> Lucene Benchmarks actually caught this. Then other optimizations made
>>>>> the regression go away (with the original cause not being directly fixed).
>>>>> Then later it was directly fixed, and never actually backported.
>>>>>
>>>>> Ignacio has indeed added a new test on main for this particular bug
>>>>> (see: https://github.com/apache/lucene/pull/15266)
>>>>>
>>>>> Here is the benchmark in Lucene that caught the issue, it was
>>>>> partially addressed, but regressed again on 'IR', and "fixed" accidentally
>>>>> by other optimizations.
>>>>>
>>>>> https://benchmarks.mikemccandless.com/FilteredAndHighHigh.html
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> On Wed, Oct 1, 2025 at 6:43 AM Michael Sokolov <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> It sounds like this is needed. Do we also have any ideas about how we
>>>>>> could have avoided this? Should we have some added test coverage, maybe a
>>>>>> luceneutil enhancement? Did we already have a test that would have caught
>>>>>> but we did not notice?
>>>>>>
>>>>>> On Wed, Oct 1, 2025, 4:03 AM Ignacio Vera <[email protected]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> In Elasticsearch, we have detected some of our benchmarks to degrade
>>>>>>> badly after upgrading to Lucene 10.3.0. We have nailed down the issue to a
>>>>>>> bug <https://github.com/apache/lucene/pull/15263> in the posting
>>>>>>> formats where the impacts returned when there are no frequencies is wrong,
>>>>>>> leading to a miscalculation of scores.
>>>>>>>
>>>>>>> In order to fix this regression, I would like to propose a bug fix
>>>>>>> release for 10.3 line. I will be pushing the bug fix today so we can start
>>>>>>> the release process as soon as this is done as far as there are no
>>>>>>> concerns. I will try to find a release manager or if I cannot I can do it
>>>>>>> myself.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Ignacio
>>>>>>>
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Adrien
>>>>
>>>