Re: Inconsistent randomisation behaviour

Aha, yes, I need to set -Ptests.nightly=true as it changes the behaviour of `rarely()`.  Thanks, this drove me absolutely nuts for about three hours yesterday.

> On 24 Sep 2025, at 16:23, Michael Sokolov <[email protected]> wrote:
> 
> did you also set the other test parameters like  -Ptests.nightly=true ?
> 
> On Wed, Sep 24, 2025 at 7:01 AM Alan Woodward <[email protected]> wrote:
>> 
>> Hi all,
>> 
>> I’m seeing some strange behaviour trying to reproduce a test failure in IntelliJ, compared to what happens when I run the test on the command line via gradle.
>> 
>> The following failed in the elasticsearch CI this morning, and reproduces when I run it on my laptop:
>> 
>> ./gradlew :lucene:test-framework:test --tests "org.apache.lucene.tests.store.TestMockDirectoryWrapper.testIsLoadedOnSlice" -Ptests.jvms=12 -Ptests.jvmargs= -Ptests.seed=7AA64BB2BA5EE192 -Ptests.useSecurityManager=true -Ptests.nightly=true -Ptests.gui=true -Ptests.file.encoding=US-ASCII -Ptests.vectorsize=128 -Ptests.forceintegervectors=true
>> 
>> However, if I take that seed and fix it on TestMockDirectoryWrapper, the same test passes when run inside IntelliJ.  Adding some debug println statements indicates that the seed is indeed the same, but the random code paths taken seem to be different.  I made the following changes to the code:
>> 
>> ---
>> 
>> $ git diff
>> diff --git a/lucene/test-framework/src/java/org/apache/lucene/tests/store/BaseDirectoryTestCase.java b/lucene/test-framework/src/java/org/apache/lucene/tests/store/BaseDirectoryTestCase.java
>> index 27bfa3c72f1..e522168b43c 100644
>> --- a/lucene/test-framework/src/java/org/apache/lucene/tests/store/BaseDirectoryTestCase.java
>> +++ b/lucene/test-framework/src/java/org/apache/lucene/tests/store/BaseDirectoryTestCase.java
>> @@ -22,6 +22,7 @@ import static org.hamcrest.Matchers.containsString;
>> import static org.hamcrest.Matchers.hasItemInArray;
>> import static org.hamcrest.Matchers.not;
>> 
>> +import com.carrotsearch.randomizedtesting.RandomizedContext;
>> import com.carrotsearch.randomizedtesting.RandomizedTest;
>> import com.carrotsearch.randomizedtesting.generators.RandomBytes;
>> import com.carrotsearch.randomizedtesting.generators.RandomNumbers;
>> @@ -1647,6 +1648,10 @@ public abstract class BaseDirectoryTestCase extends LuceneTestCase {
>>         }
>>         var loaded = in.isLoaded();
>> 
>> +        System.out.println(RandomizedContext.current().getRunnerSeedAsString());
>> +        System.out.println(dir.toString());
>> +        System.out.println(dir.getClass().getName());
>> +
>>         if (Constants.WINDOWS) {
>>           // On Windows, we temporarily don't care until this is fixed: #14050
>>         } else if (FilterDirectory.unwrap(dir) instanceof MMapDirectory
>> diff --git a/lucene/test-framework/src/test/org/apache/lucene/tests/store/TestMockDirectoryWrapper.java b/lucene/test-framework/src/test/org/apache/lucene/tests/store/TestMockDirectoryWrapper.java
>> index b5ca29dcbb0..d06e32b4ba1 100644
>> --- a/lucene/test-framework/src/test/org/apache/lucene/tests/store/TestMockDirectoryWrapper.java
>> +++ b/lucene/test-framework/src/test/org/apache/lucene/tests/store/TestMockDirectoryWrapper.java
>> @@ -21,6 +21,7 @@ import java.io.FileNotFoundException;
>> import java.io.IOException;
>> import java.nio.file.NoSuchFileException;
>> import java.nio.file.Path;
>> +import com.carrotsearch.randomizedtesting.annotations.Seed;
>> import org.apache.lucene.document.Document;
>> import org.apache.lucene.store.ByteArrayDataInput;
>> import org.apache.lucene.store.ByteBuffersDirectory;
>> @@ -33,6 +34,7 @@ import org.apache.lucene.tests.index.RandomIndexWriter;
>> 
>> // See: https://issues.apache.org/jira/browse/SOLR-12028 Tests cannot remove files on Windows
>> // machines occasionally
>> +@Seed("7AA64BB2BA5EE192")
>> public class TestMockDirectoryWrapper extends BaseDirectoryTestCase {
>> 
>>   @Override
>> 
>> —
>> 
>> Running from gradle I get the following output:
>> 
>> 1> 7AA64BB2BA5EE192
>> 1> MockDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/Users/romseygeek/projects/lucene/lucene/test-framework/build/tests-tmp/lucene.tests.store.TestMockDirectoryWrapper_7AA64BB2BA5EE192-008/index-MMapDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bb5ff68; maxCacheMB=0.9546899795532227 maxMergeSizeMB=0.9689407348632812))
>> 1> org.apache.lucene.tests.store.MockDirectoryWrapper
>> 
>> And running via IntelliJ:
>> 
>> 7AA64BB2BA5EE192
>> MockDirectoryWrapper(ByteBuffersDirectory@7e97aee2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@dfeade5)
>> org.apache.lucene.tests.store.MockDirectoryWrapper
>> 
>> Same seed, different Directory implementations.  This happens with both the IntelliJ and gradle test runners, and I’ve checked that both situations use the same JDK.  I am pretty stumped at the moment.  Any ideas, anyone?
>> 
>> - Alan
>> 
>> 
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>