Re: How to debug, among other questions

This is a multi-part message in MIME format.
--------------27JtSMVVoYbzxTR9nvYIiA11
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 12/30/24 14:17, Eduardo Costa wrote:
> Hi!,
>
> I'm new to dejagnu and would like to know, as I see the `debug.exp'
> file under `/usr/share/dejagnu', what is the 'tcl debugger' mentioned
> in this file that we can use to debug the scripts? Is it referring to
> `tkcon', `tcl-pro-debug', or other I don't know about?

There is some kind of ancient debugging support in Expect that I do not 
yet fully understand that `debugger.exp' extends.

I have so far always used "printf debugging" with DejaGnu, but I do have 
updating lib/debugger.exp on my TODO list, as it uses an old API for 
watchpoints that was deprecated in Tcl 8.4 and removed in Tcl 8.5.(Tcl 
still has watchpoints, just with a slightly different interface now.)

> Also, if I want to test a C program, I understand I can use `expect'
> functions (maybe together with utility functions declared in, say,
> `dg.exp') to compile and examine it's output, but what about testing
> it at API level (the internal functions it's made of)?

The `dg.exp' module is intended for testing compilers and interpreters.  
(DejaGnu was originally written to test the GNU toolchain:  GCC, GDB, 
GNU binutils, etc.)  That module is useful for testing programs that 
take input and produce output or additional programs.

> I've had some success including `dejagnu.h' in a test file, moving all
> my code into a convenience library I now link this test file against,
> and using `dejagnu.h' functions from within such test-file to somewhat
> get to see the expected output when I run `make check'.

This is how unit testing with DejaGnu is supposed to work:  you write 
unit test programs that emit DejaGnu's unit test protocol and use the 
"host_execute" procedure in lib/dejagnu.exp in your testsuite code to 
run the unit test programs and collect the results.

The `dejagnu.h' header (the entirety of "libdejagnu" currently) provides 
an interface for C and C++ programs to produce DejaGnu unit test 
protocol.  Note that `dejagnu.h' is GPL, but the protocol is simple and 
documented if you cannot use the GPL for your program and thus need to 
write your own support module.

The GNU Coding Standards recommend providing library interfaces when 
practical. 
(<URL:https://www.gnu.org/prep/standards/standards.html#Standards-for-Graphical-Interfaces>) 
Organizing your code as a frontend to a library also allows you to 
easily add other frontends in the future.

You have already taken the first steps on this path: consider that you 
now have one "regular use" frontend and one "unit test" frontend. You 
can use the Expect facilities in DejaGnu for testing a CLI "regular use" 
frontend (atop the backend; this is known as "system testing") in 
addition to using the "unit test" frontend to directly test the backend 
(known as (of course) "unit testing"). Well-engineered software has both 
kinds of tests.

> I did setup a `Makefile.am' under `testsuite' to compile this test
> program and called `host-execute' on it, but I'm now guessing the
> "normal" workflow is to compile whatever the test files via
> tcl/expect/dejagnu, maybe use `dg.exp' functions to check the result,
> and also `framework.exp' functions (i.e: `setup_xfail',
> `check-conditional-xfail', etc) to have it automatically responding to
> the output? Is this more or less how it works?

Nope!  Look at DejaGnu's own Makefile.am for the unit tests for 
`dejagnu.h' itself (search for "check_PROGRAMS") for a worked example.  
(It is slightly complicated because DejaGnu does not use recursive make, 
but should give you the basic idea.)

The `dg.exp' functions are only intended for compiling test programs 
when testing compilers or other toolchain components. Normal unit test 
programs should be built using make, as dependencies of the "check" target.

There is a sound software engineering basis for this:  tests should 
have, as much as possible, the same environment as the "serious" code to 
ensure that bugs cannot be missed in testing due to a different environment.

> Note that for now I'm not interested in cross-compilation or remote
> debugging, but simply using dejagnu to test my program as thoroughly
> as possible, so I'd really appreciate any comments or insights,
> because nor I've had much previous experience with tcl/expect, neither
> I still fully understand dejagnu's architecture. Also, a comment or
> two regarding `config/gdb-comm.exp' and `config/gdb-stub.exp' would be
> really appreciated, if only to figure out how these are supposed to be
> used.

Those appear to be interface modules for using GDB to load and run 
programs on embedded targets.  For native unit tests, you will want the 
"unix" target, which simply runs the test programs on the same host as 
DejaGnu itself.

> If it would be the case that some small c project is known to exist
> doing just what I'm trying to, a pointer to it would be really
> helpful. I know of gcc, coreutils, and gdb, gnu poke (mostly tests
> interactively), and `ctlseqs' found in sourceware's git repository
> (doesn't test the API but program's output), but the formers are way
> too optimized for a newcomer to understand anyways. Even I get back to
> these every now and then as I learn new things about
> dejagnu/tcl/expect on their respective manuals/books, I suspect I'm
> far from making full sense of it.

The best sources in my experience have been the "n" section manpages for 
the commands in Tcl and the expect(1) manpage for Expect itself.  The 
DejaGnu reference manual is still incomplete; improving it remains on my 
TODO list.

DejaGnu's own testsuite, while not always ideal examples, is also a good 
place to look.

For `dejagnu.h', the C and C++ programs in testsuite/libdejagnu give 
simple cases of using the API, although they do not actually perform any 
tests, instead simply emitting the results requested on their command lines.

However, the `harness.exp' file in that same directory is not a good 
example for running unit tests.  Instead, look at the tests for the 
framework side of the unit test protocol to find a better example (in 
testsuite/runtest.main/nested/testsuite/stat.test/unit-sub.exp) of using 
host_execute with a fairly complex command.


-- Jacob


--------------27JtSMVVoYbzxTR9nvYIiA11
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 12/30/24 14:17, Eduardo Costa wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAKTLjWm8kxFTB6sYEB6ujm=4zLF79RFkO_osYDmRw5McOUvs0w@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">Hi!,

I'm new to dejagnu and would like to know, as I see the `debug.exp'
file under `/usr/share/dejagnu', what is the 'tcl debugger' mentioned
in this file that we can use to debug the scripts? Is it referring to
`tkcon', `tcl-pro-debug', or other I don't know about?</pre>
    </blockquote>
    <p>There is some kind of ancient debugging support in Expect that I
      do not yet fully understand that `debugger.exp' extends.</p>
    <p>I have so far always used "printf debugging" with DejaGnu, but I
      do have updating lib/debugger.exp on my TODO list, as it uses an
      old API for watchpoints that was deprecated in Tcl 8.4 and removed
      in Tcl 8.5.<span style="white-space: pre-wrap">  (Tcl still has watchpoints, just with a slightly different interface now.)
</span></p>
    <blockquote type="cite"
cite="mid:CAKTLjWm8kxFTB6sYEB6ujm=4zLF79RFkO_osYDmRw5McOUvs0w@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">Also, if I want to test a C program, I understand I can use `expect'
functions (maybe together with utility functions declared in, say,
`dg.exp') to compile and examine it's output, but what about testing
it at API level (the internal functions it's made of)?</pre>
    </blockquote>
    <p>The `dg.exp' module is intended for testing compilers and
      interpreters.  (DejaGnu was originally written to test the GNU
      toolchain:  GCC, GDB, GNU binutils, etc.)  That module is useful
      for testing programs that take input and produce output or
      additional programs.<span style="white-space: pre-wrap">
</span></p>
    <blockquote type="cite"
cite="mid:CAKTLjWm8kxFTB6sYEB6ujm=4zLF79RFkO_osYDmRw5McOUvs0w@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">I've had some success including `dejagnu.h' in a test file, moving all
my code into a convenience library I now link this test file against,
and using `dejagnu.h' functions from within such test-file to somewhat
get to see the expected output when I run `make check'.</pre>
    </blockquote>
    <p>This is how unit testing with DejaGnu is supposed to work:  you
      write unit test programs that emit DejaGnu's unit test protocol
      and use the "host_execute" procedure in lib/dejagnu.exp in your
      testsuite code to run the unit test programs and collect the
      results.</p>
    <p>The `dejagnu.h' header (the entirety of "libdejagnu" currently)
      provides an interface for C and C++ programs to produce DejaGnu
      unit test protocol.  Note that `dejagnu.h' is GPL, but the
      protocol is simple and documented if you cannot use the GPL for
      your program and thus need to write your own support module.<br>
    </p>
    <p>The GNU Coding Standards recommend providing library interfaces
      when practical. 
(<a class="moz-txt-link-rfc1738" href="https://www.gnu.org/prep/standards/standards.html#Standards-for-Graphical-Interfaces">&lt;URL:https://www.gnu.org/prep/standards/standards.html#Standards-for-Graphical-Interfaces&gt;</a>) 
      Organizing your code as a frontend to a library also allows you to
      easily add other frontends in the future.</p>
    <p><span style="white-space: pre-wrap">You have already taken the first steps on this path:  consider that you now have one "regular use" frontend and one "unit test" frontend.  You can use the Expect facilities in DejaGnu for testing a CLI "regular use" frontend (atop the backend; this is known as "system testing") in addition to using the "unit test" frontend to directly test the backend (known as (of course) "unit testing").  Well-engineered software has both kinds of tests.
</span></p>
    <blockquote type="cite"
cite="mid:CAKTLjWm8kxFTB6sYEB6ujm=4zLF79RFkO_osYDmRw5McOUvs0w@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">I did setup a `Makefile.am' under `testsuite' to compile this test
program and called `host-execute' on it, but I'm now guessing the
"normal" workflow is to compile whatever the test files via
tcl/expect/dejagnu, maybe use `dg.exp' functions to check the result,
and also `framework.exp' functions (i.e: `setup_xfail',
`check-conditional-xfail', etc) to have it automatically responding to
the output? Is this more or less how it works?</pre>
    </blockquote>
    <p>Nope!  Look at DejaGnu's own Makefile.am for the unit tests for
      `dejagnu.h' itself (search for "check_PROGRAMS") for a worked
      example.  (It is slightly complicated because DejaGnu does not use
      recursive make, but should give you the basic idea.)</p>
    <p>The `dg.exp' functions are only intended for compiling test
      programs when testing compilers or other toolchain components. 
      Normal unit test programs should be built using make, as
      dependencies of the "check" target.</p>
    <p>There is a sound software engineering basis for this:  tests
      should have, as much as possible, the same environment as the
      "serious" code to ensure that bugs cannot be missed in testing due
      to a different environment.<span style="white-space: pre-wrap">
</span></p>
    <blockquote type="cite"
cite="mid:CAKTLjWm8kxFTB6sYEB6ujm=4zLF79RFkO_osYDmRw5McOUvs0w@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">Note that for now I'm not interested in cross-compilation or remote
debugging, but simply using dejagnu to test my program as thoroughly
as possible, so I'd really appreciate any comments or insights,
because nor I've had much previous experience with tcl/expect, neither
I still fully understand dejagnu's architecture. Also, a comment or
two regarding `config/gdb-comm.exp' and `config/gdb-stub.exp' would be
really appreciated, if only to figure out how these are supposed to be
used.</pre>
    </blockquote>
    <p>Those appear to be interface modules for using GDB to load and
      run programs on embedded targets.  For native unit tests, you will
      want the "unix" target, which simply runs the test programs on the
      same host as DejaGnu itself.</p>
    <p><span style="white-space: pre-wrap">
</span></p>
    <blockquote type="cite"
cite="mid:CAKTLjWm8kxFTB6sYEB6ujm=4zLF79RFkO_osYDmRw5McOUvs0w@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">If it would be the case that some small c project is known to exist
doing just what I'm trying to, a pointer to it would be really
helpful. I know of gcc, coreutils, and gdb, gnu poke (mostly tests
interactively), and `ctlseqs' found in sourceware's git repository
(doesn't test the API but program's output), but the formers are way
too optimized for a newcomer to understand anyways. Even I get back to
these every now and then as I learn new things about
dejagnu/tcl/expect on their respective manuals/books, I suspect I'm
far from making full sense of it.</pre>
    </blockquote>
    <p>The best sources in my experience have been the "n" section
      manpages for the commands in Tcl and the expect(1) manpage for
      Expect itself.  The DejaGnu reference manual is still incomplete;
      improving it remains on my TODO list.</p>
    <p>DejaGnu's own testsuite, while not always ideal examples, is also
      a good place to look.</p>
    <p>For `dejagnu.h', the C and C++ programs in testsuite/libdejagnu
      give simple cases of using the API, although they do not actually
      perform any tests, instead simply emitting the results requested
      on their command lines.</p>
    <p>However, the `harness.exp' file in that same directory is not a
      good example for running unit tests.  Instead, look at the tests
      for the framework side of the unit test protocol to find a better
      example (in testsuite/runtest.main/nested/testsuite/stat.test/unit-sub.exp)
      of using host_execute with a fairly complex command.<br>
    </p>
    <p><br>
    </p>
    <p>-- Jacob</p>
    <p><br>
    </p>
  </body>
</html>

--------------27JtSMVVoYbzxTR9nvYIiA11--