bug#79077: host_execute and non-zero exit status
Jacob Bachmeyer <[email protected]> Thu, 24 Jul 2025 21:13:53 -0500
| Newsgroups | gmane.comp.sysutils.dejagnu.bugs |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --===============8541391370087511990== Content-Type: multipart/alternative; boundary="------------ojfszxGOoYe5QkouL0herLWs" Content-Language: en-US This is a multi-part message in MIME format. --------------ojfszxGOoYe5QkouL0herLWs Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 7/24/25 00:57, Marc Nieper-Wißkirchen wrote: > Hi Jacob, > > Thank you for your quick response! You are welcome. Is Nieper-Wisskirchen a proper ASCII transliteration of your name for a thanks in the ChangeLog? > Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer<[email protected]>: >> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote: >>> Hi, >>> >>> The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to >>> check the exit status of the executed test programs. A test program >>> that simply aborts (e.g. using the C function of the same name) won't >>> cause any testsuite failures. This seems brittle and like a >>> misfeature. >>> >>> What is the supposed way to deal with this? >> The host_execute procedure is intended for running unit test programs >> that speak a special DejaGnu protocol and ignoring exit codes from unit >> test programs is intended. The DejaGnu unit testing protocol does not >> depend on the exit code of the unit test program at all, because that >> exit code might not be available in all environments. > I am sorry if I wasn't clear enough in my previous email. I didn't > mean that the exit code should be part of the actual protocol, only > that running the testcase should be considered unsuccessful if the > program exits with a failure code (in a POSIX system; in some other > environment, there may be other indications for failure of execution). But that would make the exit code part of the protocol. DejaGnu generally supports running tests on "remote" target boards and a target connected over a serial line is unlikely to return an exit code. The only way to be sure that a dependency on the exit code will not creep in is to ignore the exit code. Admittedly, DejaGnu does not yet properly support running unit tests on remote targets. >> Instead, DejaGnu expects an explicit "END" token from the unit test >> program to indicate that the program has reached its intended >> completion. A warning is produced if this token is not observed; >> perhaps a future version of DejaGnu should insert an UNRESOLVED result >> like we currently do when a Tcl test script aborts? > I think inserting an UNRESOLVED result would be more appropriate. > Otherwise, there would be no formal failure result if a unit test > breaks due to, say, a segfault. The explicit "END" token is a relatively recent addition (added after the last release). I was reluctant to outright require it (in case there are any testsuites out there with independent unit test protocol implementations) but I now agree that a warning from the test framework is not enough when a unit test bombs out early. An initial solution has been pushed to Savannah on the PR79077 branch. DejaGnu can now be run directly from a Git checkout, you should be able to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make check" command line. > libgccjit uses a heavily patched version of host_execute ([1]) that > also handles tests run under Valgrind; perhaps some of the ideas from > that version can be incorporated into DejaGnu proper. That heavily patched version has problems that have since been fixed upstream. I would encourage you to try getting rid of it. You should be able to move the Valgrind support into a wrapper around host_execute, since host_execute has accepted arguments since 1.6.3; you can test precisely by matching [info body host_execute] against "The arguments are". Note that the name of the executable needs to be absolute; the DejaGnu utility proc "which" will search PATH for its argument. A first draft (untested) off the top of my head: proc wrap_host_execute {args} { global env set run_under_valgrind [info exists env(RUN_UNDER_VALGRIND)] set exec_args $args set executable [lindex $args 0] set arguments [lrange $args 1 end] if { $run_under_valgrind } { set valgrind_logfile "${executable}.valgrind.txt" set valgrind_params [which "valgrind"] lappend valgrind_params "--leak-check=full" lappend valgrind_params "--log-file=${valgrind_logfile}" set exec_args [eval [list linsert $exec_args 0] $valgrind_params] } set reported_error [eval [list host_execute] $exec_args] if { $run_under_valgrind } { upvar 2 name name parse_valgrind_logfile $name $valgrind_logfile xfail } return $reported_error } You should probably instead use a VALGRIND global and have one of the initialization files default it to [which "valgrind"] if RUN_UNDER_VALGRIND is set in the environment. I would recommend taking VALGRIND from the first of: (1) value given on DejaGnu command line ([info exists VALGRIND]) (2) environment variable VALGRIND ($::env(VALGRIND) in Tcl) (3) default by searching PATH ([which "valgrind"]) A first draft (untested) for that initialization code off the top of my head: if { ![info exists VALGRIND] && [info exists ::env(RUN_UNDER_VALGRIND)] } { if { [info exists ::env(VALGRIND)] } { set VALGRIND $::env(VALGRIND) } else { set VALGRIND [which "valgrind"] } } The code does nothing if VALGRIND is already set because the framework will set it as a Tcl global if it is given on the command line. A framework-provided value can also originate from multi-pass testing and vary between passes, perhaps to compare results from different Valgrind versions. > [1]https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89 -- Jacob --------------ojfszxGOoYe5QkouL0herLWs Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <div class="moz-cite-prefix">On 7/24/25 00:57, Marc Nieper-Wißkirchen wrote:<br> </div> <blockquote type="cite" cite="mid:CAEYrNrQSdbEoNVPPcVysYjO7bxb9r7D4h9r2LW=GNO6_jxSgtQ@mail.gmail.com"> <pre wrap="" class="moz-quote-pre">Hi Jacob, Thank you for your quick response!</pre> </blockquote> <p>You are welcome.</p> <p>Is Nieper-Wisskirchen a proper ASCII transliteration of your name for a thanks in the ChangeLog?<span style="white-space: pre-wrap"> </span></p> <blockquote type="cite" cite="mid:CAEYrNrQSdbEoNVPPcVysYjO7bxb9r7D4h9r2LW=GNO6_jxSgtQ@mail.gmail.com"> <pre wrap="" class="moz-quote-pre">Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer <a class="moz-txt-link-rfc2396E" href="mailto:[email protected]"><[email protected]></a>: </pre> <blockquote type="cite"> <pre wrap="" class="moz-quote-pre"> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote: </pre> <blockquote type="cite"> <pre wrap="" class="moz-quote-pre">Hi, The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to check the exit status of the executed test programs. A test program that simply aborts (e.g. using the C function of the same name) won't cause any testsuite failures. This seems brittle and like a misfeature. What is the supposed way to deal with this? </pre> </blockquote> <pre wrap="" class="moz-quote-pre"> The host_execute procedure is intended for running unit test programs that speak a special DejaGnu protocol and ignoring exit codes from unit test programs is intended. The DejaGnu unit testing protocol does not depend on the exit code of the unit test program at all, because that exit code might not be available in all environments. </pre> </blockquote> <pre wrap="" class="moz-quote-pre"> I am sorry if I wasn't clear enough in my previous email. I didn't mean that the exit code should be part of the actual protocol, only that running the testcase should be considered unsuccessful if the program exits with a failure code (in a POSIX system; in some other environment, there may be other indications for failure of execution).</pre> </blockquote> <p>But that would make the exit code part of the protocol. DejaGnu generally supports running tests on "remote" target boards and a target connected over a serial line is unlikely to return an exit code. The only way to be sure that a dependency on the exit code will not creep in is to ignore the exit code.</p> <p>Admittedly, DejaGnu does not yet properly support running unit tests on remote targets.<br> </p> <blockquote type="cite" cite="mid:CAEYrNrQSdbEoNVPPcVysYjO7bxb9r7D4h9r2LW=GNO6_jxSgtQ@mail.gmail.com"> <blockquote type="cite"> <pre wrap="" class="moz-quote-pre">Instead, DejaGnu expects an explicit "END" token from the unit test program to indicate that the program has reached its intended completion. A warning is produced if this token is not observed; perhaps a future version of DejaGnu should insert an UNRESOLVED result like we currently do when a Tcl test script aborts? </pre> </blockquote> <pre wrap="" class="moz-quote-pre"> I think inserting an UNRESOLVED result would be more appropriate. Otherwise, there would be no formal failure result if a unit test breaks due to, say, a segfault.</pre> </blockquote> <p>The explicit "END" token is a relatively recent addition (added after the last release). I was reluctant to outright require it (in case there are any testsuites out there with independent unit test protocol implementations) but I now agree that a warning from the test framework is not enough when a unit test bombs out early.<span style="white-space: pre-wrap"> </span></p> <p><span style="white-space: pre-wrap">An initial solution has been pushed to Savannah on the PR79077 branch. DejaGnu can now be run directly from a Git checkout, you should be able to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make check" command line. </span></p> <blockquote type="cite" cite="mid:CAEYrNrQSdbEoNVPPcVysYjO7bxb9r7D4h9r2LW=GNO6_jxSgtQ@mail.gmail.com"> <pre wrap="" class="moz-quote-pre">libgccjit uses a heavily patched version of host_execute ([1]) that also handles tests run under Valgrind; perhaps some of the ideas from that version can be incorporated into DejaGnu proper.</pre> </blockquote> <p>That heavily patched version has problems that have since been fixed upstream. I would encourage you to try getting rid of it.</p> <p>You should be able to move the Valgrind support into a wrapper around host_execute, since host_execute has accepted arguments since 1.6.3; you can test precisely by matching [info body host_execute] against "The arguments are". Note that the name of the executable needs to be absolute; the DejaGnu utility proc "which" will search PATH for its argument.</p> <p>A first draft (untested) off the top of my head:</p> <pre>proc wrap_host_execute {args} { global env set run_under_valgrind [info exists env(RUN_UNDER_VALGRIND)] set exec_args $args set executable [lindex $args 0] set arguments [lrange $args 1 end] if { $run_under_valgrind } { set valgrind_logfile "${executable}.valgrind.txt" set valgrind_params [which "valgrind"] lappend valgrind_params "--leak-check=full" lappend valgrind_params "--log-file=${valgrind_logfile}" set exec_args [eval [list linsert $exec_args 0] $valgrind_params] } set reported_error [eval [list host_execute] $exec_args] if { $run_under_valgrind } { upvar 2 name name parse_valgrind_logfile $name $valgrind_logfile xfail } return $reported_error } </pre> <p>You should probably instead use a VALGRIND global and have one of the initialization files default it to [which "valgrind"] if RUN_UNDER_VALGRIND is set in the environment. I would recommend taking VALGRIND from the first of:</p> <p> (1) value given on DejaGnu command line ([info exists VALGRIND])<br> (2) environment variable VALGRIND ($::env(VALGRIND) in Tcl)<br> (3) default by searching PATH ([which "valgrind"])</p> <p>A first draft (untested) for that initialization code off the top of my head:</p> <pre>if { ![info exists VALGRIND] && [info exists ::env(RUN_UNDER_VALGRIND)] } { if { [info exists ::env(VALGRIND)] } { set VALGRIND $::env(VALGRIND) } else { set VALGRIND [which "valgrind"] } } </pre> <p>The code does nothing if VALGRIND is already set because the framework will set it as a Tcl global if it is given on the command line. A framework-provided value can also originate from multi-pass testing and vary between passes, perhaps to compare results from different Valgrind versions.<br> </p> <p><span style="white-space: pre-wrap"> </span></p> <blockquote type="cite" cite="mid:CAEYrNrQSdbEoNVPPcVysYjO7bxb9r7D4h9r2LW=GNO6_jxSgtQ@mail.gmail.com"> <pre wrap="" class="moz-quote-pre">[1] <a class="moz-txt-link-freetext" href="https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89">https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89</a></pre> </blockquote> <br> <pre wrap="" class="moz-quote-pre"><span style="white-space: pre-wrap">-- Jacob</span></pre> </body> </html> --------------ojfszxGOoYe5QkouL0herLWs-- --===============8541391370087511990== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KQnVnLWRlamFn bnUgbWFpbGluZyBsaXN0CkJ1Zy1kZWphZ251QGdudS5vcmcKaHR0cHM6Ly9saXN0cy5nbnUub3Jn L21haWxtYW4vbGlzdGluZm8vYnVnLWRlamFnbnUK --===============8541391370087511990==--