Re: Proposing a 10.3.1 bug fix release

I agree that this is a performance bug, though the fact that it only
affects queries sorted by score where a term query runs on a field that
doesn't index term frequencies suggests to me that it's more a problem for
synthetic benchmarks than for real-world workloads. Most of the time, if
you sort by score, then your `TermQuery`s would typically either have
frequencies indexed (e.g. lexical search, learned sparse retrieval), or be
applied as filters (e.g. filter on category for e-commerce) rather than as
scoring clauses. (I'm not trying to say it's not a bug, it is a bug and I'm
supportive of fixing it, it's an easy/safe fix.)

Lucene's nightly benchmarks on the Wikipedia dataset only use term queries
on fields that index term frequencies, so they couldn't have caught this.
These two slowdowns are different. They happened when we switched the
evaluation of conjunctive queries to a "score-first" approach, ie.
computing scores on the n-th clause before checking if the [{N+1}-th clause
matches (as opposed to only computing scores after verifying that all
clauses match, which the code refers to as a "doc-first" approach).
Empirically, score-first worked better on queries that match many
documents, and doc-first worked better on queries that match few documents.
Because of the filter that only matches 5% of docs, FilteredAndHighHigh
doesn't match many docs.
 - At annotation IN, conjunctive queries switched from a doc-first approach
to a score-first approach.
 - At annotation IO, conjunctive queries were refactored to evaluate blocks
of documents where the min competitive score isn't set with a doc-first
approach, and other blocks of documents with a score-first approach. This
helped speed up queries that don't match many docs as per the above
observation that doc-first approach works better for "sparse" queries.
 - At annotation IR, we removed the alignment of this heuristic with blocks
so that conjunctive queries moved to score-first immediately after a
minimum competitive score was set, instead of waiting til the end of the
current block before doing it. Hence the slowdown on sparse queries since
we evaluated fewer docs with a doc-first approach and more docs with a
score-first approach.
 - Annotation IS was a kernel upgrade that reverted the slowdown that we
had observed after the previous kernel upgrade at annotation II.

You can tell from the benchmark results that it would be better to switch
to score-first a bit later, we just haven't found the perfect heuristic yet.

However, nightly benchmarks on the NYC taxi dataset did see this
performance problem, since they run a disjunctive query on two term queries
that run on fields that don't index term frequencies.
https://benchmarks.mikemccandless.com/sparseResults.html. As noted
elsewhere it wouldn't have told us about the bug on the 10.3 branch though
since nightly benchmarks only run against the main branch and the main
branch doesn't have this bug at the moment.




On Wed, Oct 1, 2025 at 1:11 PM Benjamin Trent <[email protected]> wrote:

> Michael,
>
> Lucene Benchmarks actually caught this. Then other optimizations made the
> regression go away (with the original cause not being directly fixed). Then
> later it was directly fixed, and never actually backported.
>
> Ignacio has indeed added a new test on main for this particular bug (see:
> https://github.com/apache/lucene/pull/15266)
>
> Here is the benchmark in Lucene that caught the issue, it was partially
> addressed, but regressed again on 'IR', and "fixed" accidentally by other
> optimizations.
>
> https://benchmarks.mikemccandless.com/FilteredAndHighHigh.html
>
> [image: image.png]
>
> On Wed, Oct 1, 2025 at 6:43 AM Michael Sokolov <[email protected]> wrote:
>
>> It sounds like this is needed. Do we also have any ideas about how we
>> could have avoided this? Should we have some added test coverage, maybe a
>> luceneutil enhancement? Did we already have a test that would have caught
>> but we did not notice?
>>
>> On Wed, Oct 1, 2025, 4:03 AM Ignacio Vera <[email protected]> wrote:
>>
>>> Hi,
>>>
>>> In Elasticsearch, we have detected some of our benchmarks to degrade
>>> badly after upgrading to Lucene 10.3.0. We have nailed down the issue to a
>>> bug <https://github.com/apache/lucene/pull/15263> in the posting
>>> formats where the impacts returned when there are no frequencies is wrong,
>>> leading to a miscalculation of scores.
>>>
>>> In order to fix this regression, I would like to propose a bug fix
>>> release for 10.3 line. I will be pushing the bug fix today so we can start
>>> the release process as soon as this is done as far as there are no
>>> concerns. I will try to find a release manager or if I cannot I can do it
>>> myself.
>>>
>>> Thanks,
>>>
>>> Ignacio
>>>
>>>
>>> [image: image.png]
>>>
>>

-- 
Adrien