Re: Running the test suite independently of the build

Hi,

On Fri, May 15, 2026 at 08:24:24AM -0600, Todd C. Miller wrote:
>On Fri, 15 May 2026 14:09:10 +0200, Marc Haber wrote:
>> make check runs the test suite after a successful build. And even right
>> after a successful "make", it still compiles things before doing
>> testing. Make installcheck doesn't work at that stage.
>
>The test binaries for things like unit tests (as well as testsudoers)
>are not build by default.
>
>> Does make installcheck depend on actually having done make install
>> before?
>
>Sudo doesn't currently support an installcheck target.

Ah, INSTALL.configure is boilerplate docs from GNU autotools. I mistook 
that as sudo configuration. Sorry for that.

>> Debian's autopkgtests work by building the actual package, installing it
>> the normal way like an end user would, and then running the tests
>> against the installed package, calling /usr/bin/sudo etc. autopkgtest
>> can run without having the package built before, and it even _usually_
>> runs from a source tree that was not build before (since our tooling
>> makes a copy of the source tree and builds there).
>
>Many (most?) of the tests are not testing the installed binaries.
>There is a combination of tests for specific functions as well as
>tests for behavior using "testsudoers".  The only tests that could
>run against the installed version of sudo would be the ones that
>use visudo.

If there are tests that test sudo's interpretation of sudoers by 
checking whether sudo will allow certain operations when faced certain 
configuration files, I would love to have a possibility to run them 
independencly of the build.

>How does Debian normally deal with packages that include unit tests?

We run those tests during build and fail the build when the tests fail. 
I THINK that Debian's build process also does that for sudo, although I 
never experienced that.

Our autopkgtests are in addition executed outside of package build, for 
example during library migrations, when reverse dependencies are updated 
and during general QA efforts. When our autopkgtests don't run the 
upstream test suite, failures that would be detected by the upstream 
test suite but which we don't have added repeated tests will go 
undetected.

Since our autopkgtest infrastructure heavily leverages container and 
virtualization, it is possible to run tests as root, and even to have 
tests that break the host system.

Executing the upstream suite just as part of the autopkgtest is a 
problem, mostly for the same reasons that we have on our table here. 
Upstream test suites are often tied to a build tree, test the binaries
from the build tree and cannot test the binaries that are actually 
installed on the system.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
____________________________________________________________
sudo-workers mailing list <[email protected]>
For list information, options, or to unsubscribe, visit:
https://www.sudo.ws/mailman/listinfo/sudo-workers