Re: Debian ISO Testing

Philip Hands <[email protected]> Sun, 30 Jul 2023 15:48:23 +0200
Newsgroups	gmane.linux.debian.devel.cd
Message-ID	<[email protected]>
Martin McCarthy <[email protected]> writes:

> 4) Automation. I read a thread on this mailing list recently talking 
> about openQA and whilst I am not opposed to automatic testing by 
> machines, I would insist that the manual testers are still retained as 
> relying solely on machines for QA can be problematic (machines are not 
> perfect just like humans). Not only that, the testers over on #debian-cd 
> is a community and having full automation would mean that community no 
> longer meets regularly to test stuff...and thus the community disperses 
> and fades away over time. FOSS to me is all about community so I would 
> hope that openQA works alongside the ISO testers rather than replace us. 
> This is slightly off-topic but AI is doing a lot of damage to corporate 
> tech and I hope that the FOSS projects don't fall victim to it too.

I don't think there's any danger of OpenQA doing this (speaking as the
person who set openqa.debian.net up) because the by-hand testing and
openQA are complementary rather than competing.  They're aimed at
differing targets.

The automated tests are mostly about catching regressions, mostly during
development, and I'd hope that as the salsa integration comes together
they'll become something that can provide confidence to people as they
contribute to D-I.

My perception is that the by-hand testing is more about making sure
things are working across as wide a set of use cases on as wide a range
of hardware as possible, and is focussed around release time.

Writing automated tests that can notice things looking "weird" is rather
hard -- OpenQA is very good at noticing that someone changed the font
kerning very slightly (which is where many of our false positives come
from), but if something is broken on a bit of the screen that's not
specified as something to look out for in the test, it'll never notice.

Also, testing on real hardware is not exactly trivial with OpenQA, so
that has not been a priority, but even if we do get to the point where
we can do openQA testing of real hardware, I wouldn't expect that the
spectrum of hardware under test would be anywhere near as wide as
currently gets tested by hand, nor would I expect for it to be useful
(or even possible) to automate all of the tests done by hand.

Of course, if fleshing out the set of tests that OpenQA runs (on VM or
real hardware) renders some of the by-hand tests mostly redundant, that
doesn't seem terrible, especially if we concentrate on automating the
tests that are particularly tedious to do by hand. (suggestions welcome :-) )

Cheers, Phil.
-- 
Philip Hands -- https://hands.com/~phil
signature.asc (application/pgp-signature, 832 B)