Re: [JENKINS-EA] Lucene-main-Windows (64bit/hotspot/jdk-27-ea+23) - Build # 19864 - Unstable!

It's a new flavor of a recurrent problem; it's hard to test
probabilistic data structures since every now and then you get a wonky
case. I think we can make it more reliable by reducing the number of
requested vectors relative to the size of the test index. It's
currently:

int k =3D random().nextInt(size / 10 + 1) + 1; // size is maxdoc

if we make it divide by 50 instead of 10 that should help.

While I was looking I also happened to notice:

  @Override
  protected boolean mergeIsStable() {
    // suppress this test from base class: merges for knn graphs are
not stable due to connected
    // components
    // logic
    return false;
  }

But I think we removed the "connected components logic" so perhaps we
can add back the stable merges checking?

On Fri, Jun 19, 2026 at 4:58=E2=80=AFAM Dawid Weiss <[email protected]>=
 wrote:
>
>
> This does reproduce for:
>
> gradlew :lucene:backward-codecs:test --tests "org.apache.lucene.backward_=
codecs.lucene99.TestLucene99HnswScalarQuantizedVectorsFormat.testRandomWith=
UpdatesAndGraph" -Ptests.asserts=3Dtrue -Ptests.file.encoding=3DISO-8859-1 =
-Ptests.gui=3Dtrue -Ptests.haltonfailure=3Dfalse -Ptests.jvmargs=3D -Ptests=
.jvms=3D5 -Ptests.seed=3D23B608638D19F83 -Ptests.vectorsize=3D256
>
> I've ran claude/opus on this is here's what it says:
>
> This is not a correctness bug in the codec =E2=80=94 it's an over-strict =
assertion colliding with approximate, filtered HNSW search:
>
>   1. The test (testRandomWithUpdatesAndGraph) re-adds documents under ran=
dom ids drawn from [0, numDoc), so many docs get overwritten =E2=86=92 lots=
 of deleted docs in the segment.
>   2. The search runs with AcceptDocs.fromLiveDocs(liveDocs, =E2=80=A6). I=
n Lucene99HnswVectorsReader.search, since the graph is large enough relativ=
e to k, it takes the HNSW path (doHnsw
>   =3D=3D true), not the exhaustive fallback that would guarantee k.
>   3. In HnswGraphSearcher.search, the live-doc ratio matters (KnnSearchSt=
rategy.Hnsw.useFilteredSearch): when fewer than 60% of docs pass the filter
>   (DEFAULT_FILTERED_SEARCH_THRESHOLD =3D 60), it switches to FilteredHnsw=
GraphSearcher (ACORN-1 style). The Lucene99 graph reports a known maxConn, =
so this path is eligible. With
>   this seed's heavy deletions, the live ratio falls below 60% and the fil=
tered searcher is used.
>   4. FilteredHnswGraphSearcher.searchLevel does bounded 1=E2=80=932-hop e=
xploration (maxExplorationMultiplier, minToScore, a capped visited bitset o=
f ~log(graphSize)*k / filterRatio). Only
>   accepted (live) ordinals become candidates; deleted nodes are only used=
 for a single extra exploration hop. When the candidate queue drains before=
 k live neighbors are
>   collected, the loop at line 134 (while (candidates.size() > 0 =E2=80=A6=
)) exits =E2=80=94 legitimately returning fewer than k.
>
> I am not familiar with how hnsw searches run so I'd like somebody with mo=
re insight to take a look - even if the above analysis is right and it only=
 requires test assertion relaxation.
>
> Dawid
>
>
> On Fri, Jun 19, 2026 at 10:47=E2=80=AFAM Policeman Jenkins Server via bui=
lds <[email protected]> wrote:
>>
>> Build: https://jenkins.thetaphi.de/job/Lucene-main-Windows/19864/
>> Java: 64bit/hotspot/jdk-27-ea+23 -XX:+UseCompressedOops -XX:+UseParallel=
GC
>>
>> 3 tests failed.
>> FAILED:  org.apache.lucene.backward_codecs.lucene102.TestLucene102HnswBi=
naryQuantizedVectorsFormat.testRandomWithUpdatesAndGraph
>>
>> Error Message:
>> java.lang.AssertionError: expected:<5> but was:<2>
>>
>> Stack Trace:
>> java.lang.AssertionError: expected:<5> but was:<2>
>>         at __randomizedtesting.SeedInfo.seed([23B608638D19F83:7CE6A59B8F=
BA37E0]:0)
>>         at [email protected]/org.apache.l=
ucene.tests.index.BaseKnnVectorsFormatTestCase.testRandomWithUpdatesAndGrap=
h(BaseKnnVectorsFormatTestCase.java:1696)
>>         at java.base/java.lang.reflect.Method.invoke(Method.java:583)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner.invoke(RandomizedRunner.java:1780)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$8.evaluate(RandomizedRunner.java:959)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$9.evaluate(RandomizedRunner.java:995)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1009)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeard=
ownChained.java:48)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTest=
Name.java:45)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAf=
terMaxFailures.java:60)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47=
)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner.runSingleTest(RandomizedRunner.java:968)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$5.evaluate(RandomizedRunner.java:853)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$6.evaluate(RandomizedRunner.java:904)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$7.evaluate(RandomizedRunner.java:915)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.j=
ava:38)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOver=
ridesOnMethodsRule.java:40)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOver=
ridesOnMethodsRule.java:40)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRe=
quired.java:31)
>>         at [email protected]/org.apache.l=
ucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule=
.java:43)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47=
)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAf=
terMaxFailures.java:60)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuit=
es.java:47)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:8=
50)
>>         at java.base/java.lang.Thread.run(Thread.java:1527)
>>
>>
>> FAILED:  org.apache.lucene.backward_codecs.lucene95.TestLucene95HnswVect=
orsFormat.testRandomWithUpdatesAndGraph
>>
>> Error Message:
>> java.lang.AssertionError: expected:<5> but was:<2>
>>
>> Stack Trace:
>> java.lang.AssertionError: expected:<5> but was:<2>
>>         at __randomizedtesting.SeedInfo.seed([23B608638D19F83:7CE6A59B8F=
BA37E0]:0)
>>         at [email protected]/org.apache.l=
ucene.tests.index.BaseKnnVectorsFormatTestCase.testRandomWithUpdatesAndGrap=
h(BaseKnnVectorsFormatTestCase.java:1696)
>>         at java.base/java.lang.reflect.Method.invoke(Method.java:583)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner.invoke(RandomizedRunner.java:1780)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$8.evaluate(RandomizedRunner.java:959)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$9.evaluate(RandomizedRunner.java:995)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1009)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeard=
ownChained.java:48)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTest=
Name.java:45)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAf=
terMaxFailures.java:60)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47=
)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner.runSingleTest(RandomizedRunner.java:968)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$5.evaluate(RandomizedRunner.java:853)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$6.evaluate(RandomizedRunner.java:904)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$7.evaluate(RandomizedRunner.java:915)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.j=
ava:38)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOver=
ridesOnMethodsRule.java:40)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOver=
ridesOnMethodsRule.java:40)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRe=
quired.java:31)
>>         at [email protected]/org.apache.l=
ucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule=
.java:43)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47=
)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAf=
terMaxFailures.java:60)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuit=
es.java:47)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:8=
50)
>>         at java.base/java.lang.Thread.run(Thread.java:1527)
>>
>>
>> FAILED:  org.apache.lucene.backward_codecs.lucene99.TestLucene99HnswScal=
arQuantizedVectorsFormat.testRandomWithUpdatesAndGraph
>>
>> Error Message:
>> java.lang.AssertionError: expected:<5> but was:<2>
>>
>> Stack Trace:
>> java.lang.AssertionError: expected:<5> but was:<2>
>>         at __randomizedtesting.SeedInfo.seed([23B608638D19F83:7CE6A59B8F=
BA37E0]:0)
>>         at [email protected]/org.apache.l=
ucene.tests.index.BaseKnnVectorsFormatTestCase.testRandomWithUpdatesAndGrap=
h(BaseKnnVectorsFormatTestCase.java:1696)
>>         at java.base/java.lang.reflect.Method.invoke(Method.java:583)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner.invoke(RandomizedRunner.java:1780)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$8.evaluate(RandomizedRunner.java:959)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$9.evaluate(RandomizedRunner.java:995)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1009)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeard=
ownChained.java:48)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTest=
Name.java:45)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAf=
terMaxFailures.java:60)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47=
)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner.runSingleTest(RandomizedRunner.java:968)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$5.evaluate(RandomizedRunner.java:853)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$6.evaluate(RandomizedRunner.java:904)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.RandomizedRunner$7.evaluate(RandomizedRunner.java:915)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.j=
ava:38)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOver=
ridesOnMethodsRule.java:40)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOver=
ridesOnMethodsRule.java:40)
>>         at [email protected]/org.apache.l=
ucene.tests.util.CallbacksToRuleAdapter$1.evaluate(CallbacksToRuleAdapter.j=
ava:35)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRe=
quired.java:31)
>>         at [email protected]/org.apache.l=
ucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule=
.java:43)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47=
)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAf=
terMaxFailures.java:60)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/org.apache.l=
ucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuit=
es.java:47)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
>>         at [email protected]/com.carrotsearch.randomizedtes=
ting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:8=
50)
>>         at java.base/java.lang.Thread.run(Thread.java:1527)
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]