Re: [JENKINS] Lucene-nightly-main-Linux (64bit/hotspot/jdk-24) - Build # 134 - Unstable!

Dawid Weiss <[email protected]>
Newsgroups gmane.comp.jakarta.lucene.devel
Message-ID <CAM21Rt9+QxCe2KGSpeOpTXjxz5-qYO+cj=Sao=sL4zL-pMXGVQ@mail.gmail.com> 
For what it's worth, thread stack traces at the moment of the timeout don't
include any "startup threads", which would support my finding above.

  2> WARNING: Suite execution timed out:
org.apache.lucene.replicator.nrt.TestStressNRTReplication
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=3 WAITING on
com.carrotsearch.randomizedtesting.RandomizedRunner$2@404e220d
  2> at java.base@24/java.lang.Object.wait0(Native Method)
  2> - waiting on
com.carrotsearch.randomizedtesting.RandomizedRunner$2@404e220d
  2> at java.base@24/java.lang.Object.wait(Object.java:389)
  2> at java.base@24/java.lang.Thread.join(Thread.java:1860)
  2> at java.base@24/java.lang.Thread.join(Thread.java:1936)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112)
  2> at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
  2> at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
  2> at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:54)
  2> at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:53)
  2> at java.base@24
/java.lang.invoke.LambdaForm$DMH/0x000074427c004000.invokeInterface(LambdaForm$DMH)
  2> at java.base@24
/java.lang.invoke.LambdaForm$MH/0x000074427c012800.invoke(LambdaForm$MH)
  2> at java.base@24
/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  2> at java.base@24
/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:155)
  2> at java.base@24
/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  2> at java.base@24/java.lang.reflect.Method.invoke(Method.java:565)
  2> at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:92)
  2> at jdk.proxy1/jdk.proxy1.$Proxy4.processTestClass(Unknown Source)
  2> at
org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:183)
  2> at
org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:132)
  2> at
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)
  2> at
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)
  2> at
org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:122)
  2> at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:72)
  2> at
app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> at
app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2>
  2> "Reference Handler" ID=10 RUNNABLE
  2> at java.base@24/java.lang.ref.Reference.waitForReferencePendingList(Native
Method)
  2> at java.base@24
/java.lang.ref.Reference.processPendingReferences(Reference.java:246)
  2> at java.base@24
/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208)
  2>
  2> "Finalizer" ID=11 WAITING on java.lang.ref.ReferenceQueue$Lock@26e1965d
  2> at java.base@24/java.lang.Object.wait0(Native Method)
  2> - waiting on java.lang.ref.ReferenceQueue$Lock@26e1965d
  2> at java.base@24/java.lang.Object.wait(Object.java:389)
  2> at java.base@24/java.lang.Object.wait(Object.java:351)
  2> at java.base@24
/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:138)
  2> at java.base@24
/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:229)
  2> at java.base@24
/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:165)
  2>
  2> "Signal Dispatcher" ID=12 RUNNABLE
  2>
  2> "Common-Cleaner" ID=27 TIMED_WAITING on
java.lang.ref.ReferenceQueue$Lock@20cefd64
  2> at java.base@24/java.lang.Object.wait0(Native Method)
  2> - timed waiting on java.lang.ref.ReferenceQueue$Lock@20cefd64
  2> at java.base@24/java.lang.Object.wait(Object.java:389)
  2> at java.base@24
/java.lang.ref.ReferenceQueue.remove0(ReferenceQueue.java:124)
  2> at java.base@24
/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:215)
  2> at java.base@24/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)
  2> at java.base@24/java.lang.Thread.runWith(Thread.java:1460)
  2> at java.base@24/java.lang.Thread.run(Thread.java:1447)
  2> at java.base@24
/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:148)
  2>
  2> "Notification Thread" ID=28 RUNNABLE
  2>
  2> "/127.0.0.1:42964 to /127.0.0.1:41317 workers" ID=30 WAITING on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@777173b5
  2> at java.base@24/jdk.internal.misc.Unsafe.park(Native Method)
  2> - waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@777173b5
  2> at java.base@24
/java.util.concurrent.locks.LockSupport.park(LockSupport.java:369)
  2> at java.base@24
/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
  2> at java.base@24
/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3945)
  2> at java.base@24
/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3891)
  2> at java.base@24
/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1751)
  2> at
org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> at
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> at
org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
  2> at java.base@24
/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
  2> at java.base@24
/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
  2> at java.base@24/java.lang.Thread.runWith(Thread.java:1460)
  2> at java.base@24/java.lang.Thread.run(Thread.java:1447)
  2> Locked synchronizers:
  2> - java.util.concurrent.ThreadPoolExecutor$Worker@45018215
  2>
  2> "/127.0.0.1:42964 to /127.0.0.1:41317 workers Thread 2" ID=31 WAITING
on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@221afbca
  2> at java.base@24/jdk.internal.misc.Unsafe.park(Native Method)
  2> - waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@221afbca
  2> at java.base@24
/java.util.concurrent.locks.LockSupport.park(LockSupport.java:369)
  2> at java.base@24
/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
  2> at java.base@24
/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3945)
  2> at java.base@24
/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3891)
  2> at java.base@24
/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1751)
  2> at
org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> at
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> at
org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
  2> at java.base@24
/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
  2> at java.base@24
/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
  2> at java.base@24/java.lang.Thread.runWith(Thread.java:1460)
  2> at java.base@24/java.lang.Thread.run(Thread.java:1447)
  2> Locked synchronizers:
  2> - java.util.concurrent.ThreadPoolExecutor$Worker@659a969b
  2>
  2> "/127.0.0.1:42964 to /127.0.0.1:41317 workers Thread 3" ID=32 RUNNABLE
(in native code)
  2> at java.base@24/sun.nio.ch.EPoll.wait(Native Method)
  2> at java.base@24
/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:117)
  2> at java.base@24
/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:130)
  2> - locked sun.nio.ch.Util$2@863c2aa
  2> - locked sun.nio.ch.EPollSelectorImpl@23811783
  2> at java.base@24/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:147)
  2> at
org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:187)
  2> at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> at
org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> at
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> at
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> at
org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:83)
  2> at
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> at
org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
  2> at java.base@24
/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
  2> at java.base@24
/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
  2> at java.base@24/java.lang.Thread.runWith(Thread.java:1460)
  2> at java.base@24/java.lang.Thread.run(Thread.java:1447)
  2> Locked synchronizers:
  2> - java.util.concurrent.ThreadPoolExecutor$Worker@49b0b76
  2>
  2> "SUITE-TestStressNRTReplication-seed#[29461C4B60E8F8BD]" ID=33 RUNNABLE
  2> at java.management@24/sun.management.ThreadImpl.dumpThreads0(Native
Method)
  2> at java.management@24
/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:505)
  2> at java.management@24
/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:493)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> - locked java.lang.Object@56306c92
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2>
  2> "main" ID=34 WAITING on
org.apache.lucene.replicator.nrt.TestStressNRTReplication$RestartThread@7a57a0c5
  2> at java.base@24/java.lang.Object.wait0(Native Method)
  2> - waiting on
org.apache.lucene.replicator.nrt.TestStressNRTReplication$RestartThread@7a57a0c5
  2> at java.base@24/java.lang.Object.wait(Object.java:389)
  2> at java.base@24/java.lang.Thread.join(Thread.java:1860)
  2> at java.base@24/java.lang.Thread.join(Thread.java:1936)
  2> at
app//org.apache.lucene.replicator.nrt.TestStressNRTReplication.test(TestStressNRTReplication.java:350)
  2> at java.base@24
/java.lang.invoke.LambdaForm$DMH/0x000074427c0b0400.invokeVirtual(LambdaForm$DMH)
  2> at java.base@24
/java.lang.invoke.LambdaForm$MH/0x000074427c084800.invoke(LambdaForm$MH)
  2> at java.base@24
/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
  2> at java.base@24
/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
  2> at java.base@24
/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  2> at java.base@24/java.lang.reflect.Method.invoke(Method.java:565)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> at app/[email protected]
/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> at app/[email protected]/org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> at app/[email protected]
/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> at app/[email protected]
/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> at app/[email protected]
/org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> at app/[email protected]/org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> at app/[email protected]
/com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> at app/[email protected]/com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda/0x000074427c1027f0.run(Unknown
Source)
  2> at java.base@24/java.lang.Thread.runWith(Thread.java:1460)
  2> at java.base@24/java.lang.Thread.run(Thread.java:1447)
  2>
  2> "restarter" ID=39 TIMED_WAITING
  2> at java.base@24/java.lang.Thread.sleepNanos0(Native Method)
  2> at java.base@24/java.lang.Thread.sleepNanos(Thread.java:482)
  2> at java.base@24/java.lang.Thread.sleep(Thread.java:513)
  2> at
app//org.apache.lucene.replicator.nrt.TestStressNRTReplication$RestartThread.run(TestStressNRTReplication.java:1014)

Dawid

On Tue, May 13, 2025 at 9:36 PM Dawid Weiss <[email protected]> wrote:

>
> This test evidently hung. Another problem with it is that it logs 60 mb of
> text... Heavy.
>
> It hangs in the 'restarter' thread on this condition:
>
>         while (startupThreads.size() > 0) {
>           Thread.sleep(10);
>         }
>
> the main thread just joins the restarted and never ends. Looking at the
> logs or the code... I don't understand much of it
> (but I can make the log available... all 60mb of it).
>
> This looks wrong though - startupThreads.add(t) is populated after the
> sub-thread has been started; this means
> that it is possible (although unlikely) that t removes itself from the
> startupThreads list before it's added to it...
>
> [image: image.png]
>
> I'm not sure it's the root of the problem in this failure though. Anybody
> who knows this replication code better would like to
> speak up?
>
> Dawid
>
> On Tue, May 13, 2025 at 6:40 PM Policeman Jenkins Server <
> [email protected]> wrote:
>
>> Build: https://jenkins.thetaphi.de/job/Lucene-nightly-main-Linux/134/
>> Java: 64bit/hotspot/jdk-24 -XX:-UseCompressedOops -XX:+UseSerialGC
>>
>> 2 tests failed.
>> FAILED:
>> org.apache.lucene.replicator.nrt.TestStressNRTReplication.classMethod
>>
>> Error Message:
>> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>>
>> Stack Trace:
>> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>>         at __randomizedtesting.SeedInfo.seed([29461C4B60E8F8BD]:0)
>>
>>
>> FAILED:  org.apache.lucene.replicator.nrt.TestStressNRTReplication.test
>>
>> Error Message:
>> java.lang.Exception: Test abandoned because suite timeout was reached.
>>
>> Stack Trace:
>> java.lang.Exception: Test abandoned because suite timeout was reached.
>>         at __randomizedtesting.SeedInfo.seed([29461C4B60E8F8BD]:0)
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>
>
image.png (image/png, 112.1 KB) - not displayed
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.