Re: Proposing a 10.3.1 bug fix release

I volunteer as RM!

On Wed, Oct 1, 2025 at 8:26 AM Adrien Grand <[email protected]> wrote:

> I agree that this is a performance bug, though the fact that it only
> affects queries sorted by score where a term query runs on a field that
> doesn't index term frequencies suggests to me that it's more a problem for
> synthetic benchmarks than for real-world workloads. Most of the time, if
> you sort by score, then your `TermQuery`s would typically either have
> frequencies indexed (e.g. lexical search, learned sparse retrieval), or be
> applied as filters (e.g. filter on category for e-commerce) rather than as
> scoring clauses. (I'm not trying to say it's not a bug, it is a bug and I'm
> supportive of fixing it, it's an easy/safe fix.)
>
> Lucene's nightly benchmarks on the Wikipedia dataset only use term queries
> on fields that index term frequencies, so they couldn't have caught this.
> These two slowdowns are different. They happened when we switched the
> evaluation of conjunctive queries to a "score-first" approach, ie.
> computing scores on the n-th clause before checking if the [{N+1}-th clause
> matches (as opposed to only computing scores after verifying that all
> clauses match, which the code refers to as a "doc-first" approach).
> Empirically, score-first worked better on queries that match many
> documents, and doc-first worked better on queries that match few documents.
> Because of the filter that only matches 5% of docs, FilteredAndHighHigh
> doesn't match many docs.
>  - At annotation IN, conjunctive queries switched from a doc-first
> approach to a score-first approach.
>  - At annotation IO, conjunctive queries were refactored to evaluate
> blocks of documents where the min competitive score isn't set with a
> doc-first approach, and other blocks of documents with a score-first
> approach. This helped speed up queries that don't match many docs as per
> the above observation that doc-first approach works better for "sparse"
> queries.
>  - At annotation IR, we removed the alignment of this heuristic with
> blocks so that conjunctive queries moved to score-first immediately after a
> minimum competitive score was set, instead of waiting til the end of the
> current block before doing it. Hence the slowdown on sparse queries since
> we evaluated fewer docs with a doc-first approach and more docs with a
> score-first approach.
>  - Annotation IS was a kernel upgrade that reverted the slowdown that we
> had observed after the previous kernel upgrade at annotation II.
>
> You can tell from the benchmark results that it would be better to switch
> to score-first a bit later, we just haven't found the perfect heuristic yet.
>
> However, nightly benchmarks on the NYC taxi dataset did see this
> performance problem, since they run a disjunctive query on two term queries
> that run on fields that don't index term frequencies.
> https://benchmarks.mikemccandless.com/sparseResults.html. As noted
> elsewhere it wouldn't have told us about the bug on the 10.3 branch though
> since nightly benchmarks only run against the main branch and the main
> branch doesn't have this bug at the moment.
>
>
>
>
> On Wed, Oct 1, 2025 at 1:11 PM Benjamin Trent <[email protected]>
> wrote:
>
>> Michael,
>>
>> Lucene Benchmarks actually caught this. Then other optimizations made the
>> regression go away (with the original cause not being directly fixed). Then
>> later it was directly fixed, and never actually backported.
>>
>> Ignacio has indeed added a new test on main for this particular bug (see:
>> https://github.com/apache/lucene/pull/15266)
>>
>> Here is the benchmark in Lucene that caught the issue, it was partially
>> addressed, but regressed again on 'IR', and "fixed" accidentally by other
>> optimizations.
>>
>> https://benchmarks.mikemccandless.com/FilteredAndHighHigh.html
>>
>> [image: image.png]
>>
>> On Wed, Oct 1, 2025 at 6:43 AM Michael Sokolov <[email protected]>
>> wrote:
>>
>>> It sounds like this is needed. Do we also have any ideas about how we
>>> could have avoided this? Should we have some added test coverage, maybe a
>>> luceneutil enhancement? Did we already have a test that would have caught
>>> but we did not notice?
>>>
>>> On Wed, Oct 1, 2025, 4:03 AM Ignacio Vera <[email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> In Elasticsearch, we have detected some of our benchmarks to degrade
>>>> badly after upgrading to Lucene 10.3.0. We have nailed down the issue to a
>>>> bug <https://github.com/apache/lucene/pull/15263> in the posting
>>>> formats where the impacts returned when there are no frequencies is wrong,
>>>> leading to a miscalculation of scores.
>>>>
>>>> In order to fix this regression, I would like to propose a bug fix
>>>> release for 10.3 line. I will be pushing the bug fix today so we can start
>>>> the release process as soon as this is done as far as there are no
>>>> concerns. I will try to find a release manager or if I cannot I can do it
>>>> myself.
>>>>
>>>> Thanks,
>>>>
>>>> Ignacio
>>>>
>>>>
>>>> [image: image.png]
>>>>
>>>
>
> --
> Adrien
>