Re: Announcement: on-demand openQA testing of dist-git pull requests

On Tue, 2026-07-21 at 16:40 +0200, Cristian Le via devel wrote:
> On 2026/07/21 0:29, Adam Williamson wrote:
> 
> > On Mon, 2026-07-20 at 17:15 -0500, Cristian Le via devel wrote:
> > > And yeah the more I contemplate, the less of a bad idea it seems to me to just wrap it around a tmt test. It is not much different than the current implementation right?
> > I still don't understand the idea here, honestly, or the problem it's
> > trying to solve. Could someone explain it to me in small words? :D
> 
> I hear your offer of small words and counter it with long paragraphs 😅.
> 
> Going back to the original question
> 
> > I don't want to invent some kind of system for marking PRs as related
> > myself
> 
> Best central place to do this would be in packit and if we can bring the 
> PR-based openQA tests on there we would file it under the "figure out 
> how packit can do that bucket". The other issues would be solved 
> automatically as well.

Ah, OK, so it's that bit. I guess my sorta-expectation there was that,
if we're gonna do it, we'll follow how CentOS / RHEL are attempting to
do it, which AIUI is basically "magic tags" - a known format for tag
names which indicates "these PRs are all part of the same group".

> On the how can this be ported to packit, there are 2 possible ways to 
> tackle this

I think there might be three...
> 
>       Wrap OpenQA tests as a tmt test
> 
> This would be the easiest to approach if a bit inefficient. Afaiu this 
> is already similar to how the new PR handler works right? I.e. you 
> submit a job via `OpenQAScheduler._update_schedule`, which eventually 
> submits a job to OpenQA, waits for the job to finish and then reports 
> back. This just needs to be replicated from a CLI or a python API

Well...sort of. It doesn't submit "a job", and it doesn't "wait for the
job to finish". In openQA you schedule an artifact, essentially. You
schedule the *update*, not the job. What _update_schedule does is tell
openQA "this update is ready for testing". openQA then schedules a
whole bunch of jobs (based on our test templates in
https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/src/branch/main/templates-updates.fif.json
).

The scheduler does get the complete list of jobs back, so in theory it
could then literally sit around and wait for them all, but that's not
exactly what happens.

fedora_openqa is a set of multiple fedora-messaging consumers. The
scheduler consumer only schedules jobs, and submits "results" of the
"these jobs are now scheduled" type. It never submits "completed job"
reports. That's done by *other* consumers.

In this case, there is a new consumer - the OpenQADistGitReporter
class,
https://forge.fedoraproject.org/quality/fedora_openqa/src/branch/main/src/fedora_openqa/consumer.py#L412
. It listens for openQA "job complete" messages. It then reads the
message and does a heuristic:

1. Is this job for a dist-git PR? If not, ignore
2. Was this the *last* job for the PR? (There's a field in the message
that tells it this). If not, ignore

If both of those checks 'pass', it goes ahead and "reports the
results". It queries openQA for *all* the results for the PR,
synthesizes an overall status (passed, failed or error), and sets the
PR flag to that status.

The consumers themselves are long-lived, but that lifecycle is owned by
fedora-messaging and I don't really worry about it. As the author of
the consumers, you write code that is *not* long-lived - code that
receives a message and immediately decides on an action. Because the
consume operation consumes one message at a time and is blocking, it
can't sit around indefinitely waiting for something like a test job to
finish. It has to just read the message and more or less immediately
decide what to do. (I also find it's generally best to keep the
consumers stateless, though you *can* make them have state if you
really want to).
> 
>       |Make OpenQA another test executor for packit (same as
>       testing-farm currently is)|
> 
> This is theoretically more efficient, but it depends on how packit can 
> interact with OpenQA. From what I can tell from navigating the 
> packit-service code, this would not be too difficult to implement.

It might be possible, sure. That could work. I think it's a bit hard to
be sure how it'd look because I know exactly how the openQA side works
but I have no idea about packit, and probably vice versa for you :D

However, I think there's also a third option. The current fedora_openqa
implementation actually already "depends" on packit, because it uses
the packit scratch build. The current implementation already has to
deal with the case where someone posts '/openqa test' but the scratch
build isn't done yet; we do that by having the scheduler consumer also
listen out for flag state change messages, and when it sees a 'packit
scratch build complete' message, it checks whether the PR in question
has an '/openqa test' comment and if so, it schedules the tests. (This
also means we re-run the tests any time there's a new scratch build).

It feels like something like this should also be possible if we come up
with some kind of related-PR design based around packit? As long as, in
that case, packit signals *somehow or other* (in a way that can be
picked up from fedora-messaging, though it doesn't have to be
*directly*) that it has finished a set of related scratch builds, we
should be able to have the openQA scheduler pick up that signal,
schedule the jobs with the correct set of scratch builds, and report
the results back to the correct set of PRs...
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @[email protected]
https://www.happyassassin.net



-- 
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new