bug#75440: dejagnu does not like java.io.Console of OpenJDK >= 22

This is a multi-part message in MIME format.
--===============2503732182790611452==
Content-Type: multipart/alternative;
 boundary="------------OEXIPA2If4is5aXu9WAOVcSY"
Content-Language: en-US

This is a multi-part message in MIME format.
--------------OEXIPA2If4is5aXu9WAOVcSY
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 1/8/25 10:44, Chris Nospam via Bug-dejagnu via wrote:
> Dear maintainers,
>
> starting with OpenJDK Java 22 java programs using java.io.Console cannot be testet with dejagnu any more. With JDK 21 it works like a charm.

This is a JVM bug or misfeature.  We have recently had similar problems 
with MinGWGCC tested under Wine. In that case, it was a consequence of 
Wine's new support for Windows quasi-ptys.

> I belive it has to do with excape sequences that are not filtered out correctly or similar. I am using dejagnu 1.6.3-1, debain package under Ubuntu 24.04.

DejaGnu does not and cannot filter escape sequences.  Expect matches the 
patterns it is given.  You can write patterns in your testsuite that 
accommodate or even verify escape sequences from the program under test.

> For an easy reproduction of the bug I have attached a simple example simple.zip. Just unpack it and execute exectest.sh. Be sure to update JAVA_HOME in this script to a jdk 22 or 23 installation.
>
> In testsuite/simple.log some escape sequences are shown were no one are expected:
> ^[[?1h^[=^[[?2004hhello
> ^[[?1l^[>^[[?1000l^[[?2004lhello

The JVM IO facilities are inserting escape sequenceswithout your program 
calling for them. This is a JVM bug or misfeature that Expect (on which 
DejaGnu is built) exposes.

> The attached simple echo example works after simply commenting the lines 13-14 (using Console) and uncommenting lines 16-17 (using BufferedReader) in testsuite/simple.sh.
> Alternatively it also works by passing -Djdk.console=java.base to the jvm (1st line in exectest.sh), to trigger the old implementation in JDK.
>
> #!/usr/bin/env -S java --source 23
> // #!/usr/local/java/jdk-23.0.1/bin/java --source 23 -Djdk.console=java.base
>
> import java.io.BufferedReader;
> import java.io.IOException;
> import java.io.InputStreamReader;
>
> public class ConsoleTest {
>
>      public static void main(String[] args) throws IOException {
>          System.out.print("prompt> ");
>
>          java.io.Console c = System.console();
>          String input = c.readLine();
>
>          // BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
>          // String input = in.readLine();
>
>          System.out.println(input);
>          System.out.println("prompt> ");
>      }
>
> }

First, think about what your program actually does compared to what your 
testsuite code expects it to do.  Examine the message produced at the 
end of the test run in simple.log.  In fact, that message is produced 
*after* echo.exp has completed.  Where does that message come from?

(You will benefit far more from the lesson if you find it for yourself 
than if I tell you.  There is only one place in your testsuite that can 
emit that message.)

Second, the inability to send text to the program under test is not a 
failure:  it is an error and probably an UNRESOLVED test.  A block that 
is executed as a result of catching a Tcl error probably should be 
invoking "error" or possibly "unresolved", not "fail" or even "pass" 
(unless you are testing an error condition).

Third, Tcl interpolates backslash escapes in double-quoted strings.  You 
will need to double (some of) the backslashes and/or review re_syntax(n) 
for Tcl regexp syntax.

Fourth, think about simple_test could actually fail.  That depends on 
what you are trying to do, but any "pass" should have a corresponding 
"fail" somewhere.

(Most of the above refers to Expect code in your 
testsuite/config/default.exp file.)

> It has something to do with the change of the Java default Console provider to JLine in Java 22.
> https://bugs.openjdk.org/browse/JDK-8309141is is not special with DejaGnu; we have recently had similar issues with
>   Wine, encountered while testing mingw32 builds on *nix hosts
> However, as Java Console works everywhere else (windows/linux console/bash/EclipseIDE internal "console"/IntelliJ internal "console") as far I can see, it seems to be something special with dejagnu.

No, the problem is with the JVM and/or your programs and scripts.  It 
appears to work everywhere else because those escape sequences either 
perform terminal control functions or are ignored.  You can write 
regexps that will optionally match and discard the escape sequences.  
Expect exposes this issue because it is strict about matching exactly 
what you tell it to match.


-- Jacob


--------------OEXIPA2If4is5aXu9WAOVcSY
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 1/8/25 10:44, Chris Nospam via
      Bug-dejagnu via wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:trinity-883ac4d3-67e0-4d8c-b926-1fcc387c6cf0-1736354694938@trinity-msg-rest-gmx-gmx-live-bbc95d94c-jjgc7">
      <pre wrap="" class="moz-quote-pre">Dear maintainers,

starting with OpenJDK Java 22 java programs using java.io.Console cannot be testet with dejagnu any more. With JDK 21 it works like a charm.</pre>
    </blockquote>
    <p>This is a JVM bug or misfeature.  We have recently had similar
      problems with MinGW<span style="white-space: pre-wrap"> GCC tested under Wine.  In that case, it was a consequence of Wine's new support for Windows quasi-ptys.
</span></p>
    <blockquote type="cite"
cite="mid:trinity-883ac4d3-67e0-4d8c-b926-1fcc387c6cf0-1736354694938@trinity-msg-rest-gmx-gmx-live-bbc95d94c-jjgc7">
      <pre wrap="" class="moz-quote-pre">I belive it has to do with excape sequences that are not filtered out correctly or similar. I am using dejagnu 1.6.3-1, debain package under Ubuntu 24.04.</pre>
    </blockquote>
    <p>DejaGnu does not and cannot filter escape sequences.  Expect
      matches the patterns it is given.  You can write patterns in your
      testsuite that accommodate or even verify escape sequences <span
      style="white-space: pre-wrap">from the program under test.
</span></p>
    <blockquote type="cite"
cite="mid:trinity-883ac4d3-67e0-4d8c-b926-1fcc387c6cf0-1736354694938@trinity-msg-rest-gmx-gmx-live-bbc95d94c-jjgc7">
      <pre wrap="" class="moz-quote-pre">For an easy reproduction of the bug I have attached a simple example simple.zip. Just unpack it and execute exectest.sh. Be sure to update JAVA_HOME in this script to a jdk 22 or 23 installation.

In testsuite/simple.log some escape sequences are shown were no one are expected:
^[[?1h^[=^[[?2004hhello
^[[?1l^[&gt;^[[?1000l^[[?2004lhello</pre>
    </blockquote>
    <p>The JVM IO facilities are inserting escape sequences<span
      style="white-space: pre-wrap"> without your program calling for them.  This is a JVM bug or misfeature that Expect (on which DejaGnu is built) exposes.
</span></p>
    <blockquote type="cite"
cite="mid:trinity-883ac4d3-67e0-4d8c-b926-1fcc387c6cf0-1736354694938@trinity-msg-rest-gmx-gmx-live-bbc95d94c-jjgc7">
      <pre wrap="" class="moz-quote-pre">The attached simple echo example works after simply commenting the lines 13-14 (using Console) and uncommenting lines 16-17 (using BufferedReader) in testsuite/simple.sh.
Alternatively it also works by passing -Djdk.console=java.base to the jvm (1st line in exectest.sh), to trigger the old implementation in JDK.

#!/usr/bin/env -S java --source 23
// #!/usr/local/java/jdk-23.0.1/bin/java --source 23 -Djdk.console=java.base

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

public class ConsoleTest {

    public static void main(String[] args) throws IOException {
        System.out.print("prompt&gt; ");

        java.io.Console c = System.console();
        String input = c.readLine();

        // BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        // String input = in.readLine();

        System.out.println(input);
        System.out.println("prompt&gt; ");
    }

}
</pre>
    </blockquote>
    <p>First, think about what your program actually does compared to
      what your testsuite code expects it to do.  Examine the message
      produced at the end of the test run in simple.log.  In fact, that
      message is produced *after* echo.exp has completed.  Where does
      that message come from?</p>
    <p>(You will benefit far more from the lesson if you find it for
      yourself than if I tell you.  There is only one place in your
      testsuite that can emit that message.)<br>
    </p>
    <p>Second, the inability to send text to the program under test is
      not a failure:  it is an error and probably an UNRESOLVED test.  A
      block that is executed as a result of catching a Tcl error
      probably should be invoking "error" or possibly "unresolved", not
      "fail" or even "pass" (unless you are testing an error condition).</p>
    <p>Third, Tcl interpolates backslash escapes in double-quoted
      strings.  You will need to double (some of) the backslashes and/or
      review re_syntax(n) for Tcl regexp syntax.</p>
    <p>Fourth, think about simple_test could actually fail.  That
      depends on what you are trying to do, but any "pass" should have a
      corresponding "fail" somewhere.</p>
    <p>(Most of the above refers to Expect code in your
      testsuite/config/default.exp file.)<br>
    </p>
    <blockquote type="cite"
cite="mid:trinity-883ac4d3-67e0-4d8c-b926-1fcc387c6cf0-1736354694938@trinity-msg-rest-gmx-gmx-live-bbc95d94c-jjgc7">
      <pre wrap="" class="moz-quote-pre">It has something to do with the change of the Java default Console provider to JLine in Java 22.
<a class="moz-txt-link-freetext" href="https://bugs.openjdk.org/browse/JDK-8309141is">https://bugs.openjdk.org/browse/JDK-8309141is</a> is not special with DejaGnu; we have recently had similar issues with
 Wine, encountered while testing mingw32 builds on *nix hosts
However, as Java Console works everywhere else (windows/linux console/bash/EclipseIDE internal "console"/IntelliJ internal "console") as far I can see, it seems to be something special with dejagnu.</pre>
    </blockquote>
    <p>No, the problem is with the JVM and/or your programs and
      scripts.  It appears to work everywhere else because those escape
      sequences either perform terminal control functions or are
      ignored.  You can write regexps that will optionally match and
      discard the escape sequences.  Expect exposes this issue because
      it is strict about matching exactly what you tell it to match.<br>
    </p>
    <p><br>
    </p>
    <p>-- Jacob<br>
    </p>
    <p><br>
    </p>
  </body>
</html>

--------------OEXIPA2If4is5aXu9WAOVcSY--




--===============2503732182790611452==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KQnVnLWRlamFn
bnUgbWFpbGluZyBsaXN0CkJ1Zy1kZWphZ251QGdudS5vcmcKaHR0cHM6Ly9saXN0cy5nbnUub3Jn
L21haWxtYW4vbGlzdGluZm8vYnVnLWRlamFnbnUK

--===============2503732182790611452==--