Re: Inconsistent randomisation behaviour

In general, you should try to pass all the parameters that the repro line
mentions as these may affect the result. I understand this is problematic
if you want to re-launch from intellij, in which case I'd suggest two
options -

1) run the test in intellij via gradle, setting all the options that the
repro line mentioned; this should work most of the time,
2) add @Repeat on the test you wish to trigger to fail and run that from
intellij, then try to pick the seed that failed and add @Seed on the test,
once
you've identified it as the problem.

Hope this helps,

On Thu, Sep 25, 2025 at 11:35 AM Alan Woodward <[email protected]> wrote:

> Aha, yes, I need to set -Ptests.nightly=true as it changes the behaviour
> of `rarely()`.  Thanks, this drove me absolutely nuts for about three hours
> yesterday.
>
> > On 24 Sep 2025, at 16:23, Michael Sokolov <[email protected]> wrote:
> >
> > did you also set the other test parameters like  -Ptests.nightly=true ?
> >
> > On Wed, Sep 24, 2025 at 7:01 AM Alan Woodward <[email protected]>
> wrote:
> >>
> >> Hi all,
> >>
> >> I’m seeing some strange behaviour trying to reproduce a test failure in
> IntelliJ, compared to what happens when I run the test on the command line
> via gradle.
> >>
> >> The following failed in the elasticsearch CI this morning, and
> reproduces when I run it on my laptop:
> >>
> >> ./gradlew :lucene:test-framework:test --tests
> "org.apache.lucene.tests.store.TestMockDirectoryWrapper.testIsLoadedOnSlice"
> -Ptests.jvms=12 -Ptests.jvmargs= -Ptests.seed=7AA64BB2BA5EE192
> -Ptests.useSecurityManager=true -Ptests.nightly=true -Ptests.gui=true
> -Ptests.file.encoding=US-ASCII -Ptests.vectorsize=128
> -Ptests.forceintegervectors=true
> >>
> >> However, if I take that seed and fix it on TestMockDirectoryWrapper,
> the same test passes when run inside IntelliJ.  Adding some debug println
> statements indicates that the seed is indeed the same, but the random code
> paths taken seem to be different.  I made the following changes to the code:
> >>
> >> ---
> >>
> >> $ git diff
> >> diff --git
> a/lucene/test-framework/src/java/org/apache/lucene/tests/store/BaseDirectoryTestCase.java
> b/lucene/test-framework/src/java/org/apache/lucene/tests/store/BaseDirectoryTestCase.java
> >> index 27bfa3c72f1..e522168b43c 100644
> >> ---
> a/lucene/test-framework/src/java/org/apache/lucene/tests/store/BaseDirectoryTestCase.java
> >> +++
> b/lucene/test-framework/src/java/org/apache/lucene/tests/store/BaseDirectoryTestCase.java
> >> @@ -22,6 +22,7 @@ import static org.hamcrest.Matchers.containsString;
> >> import static org.hamcrest.Matchers.hasItemInArray;
> >> import static org.hamcrest.Matchers.not;
> >>
> >> +import com.carrotsearch.randomizedtesting.RandomizedContext;
> >> import com.carrotsearch.randomizedtesting.RandomizedTest;
> >> import com.carrotsearch.randomizedtesting.generators.RandomBytes;
> >> import com.carrotsearch.randomizedtesting.generators.RandomNumbers;
> >> @@ -1647,6 +1648,10 @@ public abstract class BaseDirectoryTestCase
> extends LuceneTestCase {
> >>         }
> >>         var loaded = in.isLoaded();
> >>
> >> +
> System.out.println(RandomizedContext.current().getRunnerSeedAsString());
> >> +        System.out.println(dir.toString());
> >> +        System.out.println(dir.getClass().getName());
> >> +
> >>         if (Constants.WINDOWS) {
> >>           // On Windows, we temporarily don't care until this is fixed:
> #14050
> >>         } else if (FilterDirectory.unwrap(dir) instanceof MMapDirectory
> >> diff --git
> a/lucene/test-framework/src/test/org/apache/lucene/tests/store/TestMockDirectoryWrapper.java
> b/lucene/test-framework/src/test/org/apache/lucene/tests/store/TestMockDirectoryWrapper.java
> >> index b5ca29dcbb0..d06e32b4ba1 100644
> >> ---
> a/lucene/test-framework/src/test/org/apache/lucene/tests/store/TestMockDirectoryWrapper.java
> >> +++
> b/lucene/test-framework/src/test/org/apache/lucene/tests/store/TestMockDirectoryWrapper.java
> >> @@ -21,6 +21,7 @@ import java.io.FileNotFoundException;
> >> import java.io.IOException;
> >> import java.nio.file.NoSuchFileException;
> >> import java.nio.file.Path;
> >> +import com.carrotsearch.randomizedtesting.annotations.Seed;
> >> import org.apache.lucene.document.Document;
> >> import org.apache.lucene.store.ByteArrayDataInput;
> >> import org.apache.lucene.store.ByteBuffersDirectory;
> >> @@ -33,6 +34,7 @@ import
> org.apache.lucene.tests.index.RandomIndexWriter;
> >>
> >> // See: https://issues.apache.org/jira/browse/SOLR-12028 Tests cannot
> remove files on Windows
> >> // machines occasionally
> >> +@Seed("7AA64BB2BA5EE192")
> >> public class TestMockDirectoryWrapper extends BaseDirectoryTestCase {
> >>
> >>   @Override
> >>
> >> —
> >>
> >> Running from gradle I get the following output:
> >>
> >> 1> 7AA64BB2BA5EE192
> >> 1> MockDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/Users/romseygeek/projects/lucene/lucene/test-framework/build/tests-tmp/lucene.tests.store.TestMockDirectoryWrapper_7AA64BB2BA5EE192-008/index-MMapDirectory-001
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bb5ff68;
> maxCacheMB=0.9546899795532227 maxMergeSizeMB=0.9689407348632812))
> >> 1> org.apache.lucene.tests.store.MockDirectoryWrapper
> >>
> >> And running via IntelliJ:
> >>
> >> 7AA64BB2BA5EE192
> >> MockDirectoryWrapper(ByteBuffersDirectory@7e97aee2
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@dfeade5)
> >> org.apache.lucene.tests.store.MockDirectoryWrapper
> >>
> >> Same seed, different Directory implementations.  This happens with both
> the IntelliJ and gradle test runners, and I’ve checked that both situations
> use the same JDK.  I am pretty stumped at the moment.  Any ideas, anyone?
> >>
> >> - Alan
> >>
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [email protected]
> >> For additional commands, e-mail: [email protected]
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>