Re: Poor condition of test execution in interactive sessions

Hi Po Lu,

Po Lu <[email protected]> writes:

> I am of opinion that ERT tests must not prompt for user input when
> executed in an interactive Emacs session, but with
> `ert-run-tests-batch'.  This has proved to be essential in executing
> tests on Android, where the only available Emacs sessions are
> interactive.  Moreover, they should not require assets stored elsewhere
> than their ERT resources directories.
>
> Many of our tests which do prompt for user input interactively fail to
> satisfy these criteria.  Some of them have already been corrected, but
> this issue continues to interfere with automatic regression testing of
> Android installations, e.g. eshell-*-tests.el require
> eshell-tests-helpers.el, which is not stored in a resource directory,
> and erc-match-tests.el prompts for user input.
>
> Let us agree here and now not to introduce any more interactive tests,
> while I correct those which are already extant.
>
> Thanks.

Is this spoofed `noninteractive' binding in `ats-run-test'

  (let ((noninteractive t))
    (ert-run-tests-batch ',selector)))

a permanent thing? If not, we'd likely need to change all of ERC's

  (when noninteractive
    (erc-tests-common-kill-buffers))

to something like

  (unless erc-tests-common-defer-cleanup-p
    (erc-tests-common-kill-buffers))

Also, I see that `ats-execute-tests-batch' currently hard codes SELECTOR
to nil. If you plan on running :expensive tests someday, additional
changes will be needed.

Thanks,
J.P.