Re: etiquette for new failing test cases, followed by fix

Am 20.09.2024 um 14:44 schrieb Greg Troxel:
> Roland Illig <[email protected]> writes:
>
>> The first commit demonstrates the buggy state by having a test that
>> technically succeeds but has lots of FIXME comments in all the places
>> that are wrong, already stating what is expected after the bugfix.
>>
>> The second commit fixes one of the bugs and updates the tests, removing
>> several of the FIXME comments from the first commit. Further commits fix
>> more of the problems, one by one.
>
> Why do you propose a test that passes with a FIXME, rather than xfail?
> Isn't that what xfail is for?

Yes, you're right. I'm just not used to using xfail, and I never tried
to integrate xfail into the test suites of usr.bin/make or
usr.bin/xlint. And since the FIXME comments usually have a short
half-life, it didn't make much of a difference to me, in practical terms.

>> The benefit of this approach is that each commit shows the changes to
>> the main code together with their effects on the tests, and it provides
>> a direct before-after comparison.
>
> Doesn't xfail do that too?

No. With xfail, the results before the bugfix are only in the test
output, but not under version control. I expected the old test output to
be deleted after a few weeks, until I saw that I can still view the test
results from 2011, which, given the amount of test runs, is impressive.
Still, in general, having the before-after comparison in the code itself
feels more straight-forward to me, to have a single source to rely on.

> I don't follow "to see what the _change_ was".  There will be a bugfix
> commit, with a comment.  How is this hard, and how is it different from
> what we usually do?

The hard thing is the extra step of looking up the old results in a
different source, instead of having all information bundled in the commit.

Plus, your current case involves formatted floating point output, so the
test output using %g may not be as accurate as an equality comparison in
the source code, as the %g conversion specifier rounds the output.

Roland