Re: [PATCH] tests: improve readability
Philippe Altherr <[email protected]>
| Newsgroups | gmane.comp.shells.zsh.devel |
|---|---|
| Message-ID | <CAGdYcht84XMPPfUBbrvubmLkV5B32eXt+PL8FztDxsiPSU=qDg@mail.gmail.com> |
>
> mikael and i were talking about how
> *the test output is not as readableas we'd like*.
I came to the same conclusion and developed the following sed command to
improve the test output readability.
sed -E \
-e "s/^([.]\/.*[.]ztst: skipped .*)/\x1b[1m\x1b[36m&\x1b[0m/" \
-e "s/^([.]\/.*[.]ztst: .*)/\x1b[1m&\x1b[0m/" \
-e "s/^(Was testing: )(.*)/\x1b[1m\1\x1b[31m\2\x1b[0m/" \
-e "s/^(Test [.]\/.*[.]ztst failed:
)(.*)/\x1b[1m\1\x1b[31m\2\x1b[0m/" \
-e "s/^([-][^-].*)/\x1b[31m&\x1b[0m/" \
-e "s/^([+][^+].*)/\x1b[32m&\x1b[0m/" \
-e "s/^(Test case skipped: .*)/\x1b[36m&\x1b[0m/" \
-e "s/^([*]+)/\x1b[1m&\x1b[0m/" \
-e "s/^([0-9]+ successful test scripts?, 0 failures, 0
skipped)/\x1b[1m\x1b[32m&\x1b[0m/" \
-e "s/^([0-9]+ successful test scripts?, 0 failures), (.+
skipped)/\x1b[1m\x1b[32m\1\x1b[30m, \x1b[36m\2\x1b[0m/" \
-e "s/^([0-9]+ successful test scripts?), (.+ failures?), (0
skipped)/\x1b[1m\1, \x1b[31m\2\x1b[30m, \3\x1b[0m/" \
-e "s/^([0-9]+ successful test scripts?), (.+ failures?), (.+
skipped)/\x1b[1m\1, \x1b[31m\2\x1b[30m, \x1b[36m\3\x1b[0m/"
One important difference with your patch is that it puts more weight on
highlighting failed test cases within test files than just highlinging the
failed test files. This is especially important for test files run
with ZTST_continue=1 (which I usually always do) because there are often
multiple test cases that fail and it's important to be able to quickly
identify them.
Here is a side-by-side comparison of the output for A03quoting.ztst after
adding a bogus "X" in the expected output of two test cases and running the
test with ZTST_continue=1.
*Your patch* *My sed command*
A03quoting.ztst: starting.
*--- expected*
*+++ actual*
@@ -1,7 +1,7 @@
16#42
-16#53X
+16#53
16#5C
-16#4DX
+16#4D
16#42
16#53
16#DC
Test A03quoting.ztst failed: output differs from expected as shown above
for:
chars=$(print -r $'BS\\MBS\M-\\')
for (( i = 1; i <= $#chars; i++ )); do
char=$chars[$i]
print $(( [#16] #char ))
done
Was testing: $'-style quote with metafied backslash
A03quoting.ztst: test failed.
*--- expected*
*+++ actual*
@@ -1,6 +1,6 @@
:
16#61
-16#62X
+16#62
16#0
16#63
16#64
Test A03quoting.ztst failed: output differs from expected as shown above
for:
null1="$(print -r a$'b\0c'd)"
null2="$(setopt posixstrings; print -r a$'b\0c'd)"
for string in $null1 $null2; do
print ":"
for (( i = 1; i <= $#string; i++ )); do
char=$string[$i]
print $(( [#16] #char ))
done
done
Was testing: Embedded null characters in $'...' strings.
A03quoting.ztst: test failed.
*./A03quoting.ztst: starting.*
--- /tmp/zsh.ztst.74957/ztst.out 2026-06-05 12:43:51
+++ /tmp/zsh.ztst.74957/ztst.tout 2026-06-05 12:43:51
@@ -1,7 +1,7 @@
16#42
-16#53X
+16#53
16#5C
-16#4DX
+16#4D
16#42
16#53
16#DC
*Test ./A03quoting.ztst failed: **output differs from expected as shown
above for:*
chars=$(print -r $'BS\\MBS\M-\\')
for (( i = 1; i <= $#chars; i++ )); do
char=$chars[$i]
print $(( [#16] #char ))
done
*Was testing: **$'-style quote with metafied backslash*
*./A03quoting.ztst: test failed.*
--- /tmp/zsh.ztst.74957/ztst.out 2026-06-05 12:43:51
+++ /tmp/zsh.ztst.74957/ztst.tout 2026-06-05 12:43:51
@@ -1,6 +1,6 @@
:
16#61
-16#62X
+16#62
16#0
16#63
16#64
*Test ./A03quoting.ztst failed: **output differs from expected as shown
above for:*
null1="$(print -r a$'b\0c'd)"
null2="$(setopt posixstrings; print -r a$'b\0c'd)"
for string in $null1 $null2; do
print ":"
for (( i = 1; i <= $#string; i++ )); do
char=$string[$i]
print $(( [#16] #char ))
done
done
*Was testing: **Embedded null characters in $'...' strings.*
*./A03quoting.ztst: test failed.*
Your patch only puts "A03quoting.ztst: test failed." in red. My sed command
instead puts in red *and bold*, the names of the failing test cases and the
reasons for their failures.
For each test file, the output starts with "<filename>: starting." and ends
with "<filename>: all tests successful.", if all test cases were
successful, and with "<filename>: test failed." otherwise. My sed command
puts these messages in bold (but doesn't color them). This is useful to
identify the start and stop of each test file and distinguish these
messages from other output (some tests are noisy even in case of success).
With the addition of blank lines after each test file, this boldening
becomes less important but I think it would still be useful.
The message "<filename>: test failed." is added after each failed test
case. In my opinion, an improvement would be to output it only once at the
end of the test file. It could include the number of failed test cases and
maybe list their names, like you now list the names of the failed test
files at the very end.
Below are a few more side-by-side comparisons of when
- there are failures and skipped test cases and files
- there are failures but no skipped files
- there are no failures but skipped files
- all tests were successful.
*Your patch* *My sed command* (with manually added blank lines)
[…]
V07pcre.ztst: starting.
V07pcre.ztst: skipped (the zsh/pcre module was disabled by configure (see
config.modules))
V08zpty.ztst: starting.
V08zpty.ztst: all tests successful.
V09datetime.ztst: starting.
Test case skipped: strftime extensions not supported
Test case skipped: strftime extensions not supported
V09datetime.ztst: all tests successful.
[…]
Z04zgetopt.ztst: starting.
Z04zgetopt.ztst: all tests successful.
************************************************
77 successful test scripts, 1 failure, 2 skipped
failing test scripts:
A03quoting.ztst
************************************************
[…]
*./V07pcre.ztst: starting.*
*./V07pcre.ztst: skipped (the zsh/pcre module was disabled by configure
(see config.modules))*
*./V08zpty.ztst: starting.*
*./V08zpty.ztst: all tests successful.*
*./V09datetime.ztst: starting.*
Test case skipped: strftime extensions not supported
Test case skipped: strftime extensions not supported
*./V09datetime.ztst: all tests successful.*
[…]
*./Z04zgetopt.ztst: starting.*
*./Z04zgetopt.ztst: all tests successful.*
****************************************
*72 successful test scripts, **1 failure**, **2 skipped*
****************************************
*Your patch* *My sed command* (with manually added blank lines)
[…]
***********************************************
8 successful test scripts, 1 failure, 0 skipped
failing test scripts:
A03quoting.ztst
***********************************************
[…]
****************************************
*8 successful test scripts, **1 failure**, 0 skipped*
****************************************
*Your patch* *My sed command* (with manually added blank lines)
[…]
*************************************************
13 successful test scripts, 0 failures, 1 skipped
*************************************************
[…]
****************************************
*13 successful test scripts, 0 failures**, **1 skipped*
****************************************
*Your patch* *My sed command* (with manually added blank lines)
[…]
*************************************************
15 successful test scripts, 0 failures, 0 skipped
*************************************************
[…]
****************************************
*13 successful test scripts, 0 failures, 0 skipped*
****************************************
My recommendations/wishes:
- Put at least the main summary line and the star lines in bold
- Use more fine grained coloring for the summary line (see results of my
sed command)
- Put the star lines in red if there are any failures and otherwise in
green (and maybe in cyan if there are only skipped files)
Would it be useful to also list (in cyan) the skipped files?
Philippe