[websites/linux-kde-org] content: Add documentation for openQA
Thomas Duckworth <[email protected]>
| Newsgroups | gmane.comp.kde.cvs |
|---|---|
| Message-ID | <[email protected]> |
Git commit fc3140304e358cdadc3a83463fd1721a45780adc by Thomas Duckworth. Committed on 25/07/2026 at 03:39. Pushed by ngraham into branch 'master'. Add documentation for openQA Adds a writeup for openQA, which explains the workflow for maintainers debugging an image as well as for developers testing changes and writing their own tests. M +1 -0 content/docs.md M +6 -0 content/docs/kde-linux-dev.md A +162 -0 content/docs/openqa.md https://invent.kde.org/websites/linux-kde-org/-/commit/fc3140304e358cdadc3a83463fd1721a45780adc diff --git a/content/docs.md b/content/docs.md index 56afa2a..cad2d6c 100644 --- a/content/docs.md +++ b/content/docs.md @@ -64,6 +64,7 @@ Life on a cutting-edge image-based OS can be unfamiliar! This page contains info * [Develop or debug the kernel](kernel-dev) * [Develop other non-KDE software](dev) * [Develop KDE Linux](kde-linux-dev) +* [Test changes to KDE Linux with openQA](openqa) <!-- * Develop general-purpose GPU software using OpenCL/CUDA/etc. (coming soon) --> diff --git a/content/docs/kde-linux-dev.md b/content/docs/kde-linux-dev.md index 870d0b5..2ca9189 100644 --- a/content/docs/kde-linux-dev.md +++ b/content/docs/kde-linux-dev.md @@ -87,3 +87,9 @@ The fastest way to test your changes is to boot the <code>.iso</code> image dire 4. **Important:** In the VM configuration, ensure the **Firmware** is set to **UEFI** and **Secure Boot** is disabled. For a more permanent setup or instructions on using other virtualization tools like VirtualBox or UTM, refer to the [Installing in a Virtual Machine](./install-vm.md) guide. + +### Testing Changes with openQA + +KDE Linux uses an automated openQA test suite to test system images before they are published. The suite boots images in virtual machines and exercises essential workflows, including installing a new system, upgrading an existing one, and testing the functionality of the installed system. This helps catch regressions that may not be apparent when testing a local build manually. + +If your change affects image building, installation, updates, system services, or the desktop environment, consider whether the automated tests need to cover it. See [openQA for Developers](./openqa.md) to learn how to write tests for it. diff --git a/content/docs/openqa.md b/content/docs/openqa.md new file mode 100644 index 0000000..a71df16 --- /dev/null +++ b/content/docs/openqa.md @@ -0,0 +1,162 @@ +--- +authors: + - SPDX-FileCopyrightText: 2026 Thomas Duckworth <[email protected]> +SPDX-License-Identifier: CC-BY-4.0 + +title: "openQA for Developers and Maintainers" +scssFiles: [/scss/main.scss] +hideMeta: true +--- + +KDE Linux uses [openQA](https://open.qa/) to automatically test every system image before it is published. openQA boots each image in a virtual machine and checks that it can be installed, upgraded, and used successfully. + +openQA has some specific terminology: a *job* is one execution of a test suite against an image, while a *worker* is the machine or container that runs the job. The [openQA documentation](https://open.qa/docs/) explains these and other concepts, including test modules, job groups, assets, and the web UI. + +## How the tests work + +The test worker runs alongside the CI pipeline, so it can use the images and virtual disks produced by the build without uploading those large files elsewhere. Tests are supplied to the virtual machine through a system extension and communicate with it over SSH. + +Desktop tests use the accessibility API through `selenium-webdriver-at-spi`, allowing them to interact with applications and check their state without relying solely on screenshot matching. The suite also checks for failed system services, crashed desktop processes, networking problems, and regressions in KDE Linux commands. + +## CI pipeline and maintainer workflow + +On the protected default branch of the upstream [KDE Linux repository](https://invent.kde.org/kde-linux/kde-linux), the pipeline works as follows: + +1. The *imaging* job builds, signs, and stages the image and its sysupdate channel on `storage.kde.org`. +2. The *trigger-openqa* job starts the pipeline in [os-autoinst-distri-kdelinux](https://invent.kde.org/kde-linux/os-autoinst-distri-kdelinux), passing the staged image URL and sysupdate channel URL to it. +3. The downstream pipeline runs the normal install-and-sanity flow and the upgrade flow. +4. The upstream *publish* job runs only after the openQA pipeline has succeeded. It promotes the already-staged image for public consumption. + +This makes an openQA failure a gate for image publish. Start by opening the downstream pipeline from the `trigger-openqa` job, then open the failing openQA job from its CI log. The job page shows the test modules in order and lets you identify whether the failure is in installation, the installed-system sanity tests, or the upgrade path. + +### Download a staged image + +The link provided after the log message *“In case of failure, you can inspect and download the .iso image and sysupdate tree at…”* opens a storage browser for the build under test. Download the <code>.iso</code> there to boot it manually or give it to a local openQA stack. The <code>sysupdate</code> tree contains the staged update artifacts used by the upgrade test. + +### Investigate a failed job + +Open the job in the openQA web UI and select the failed module first. The link to the web UI will appear after the *“<name> test job is now running.”* log message. Its details include the module result, screenshots or video where applicable, and diagnostic output. In the *Logs & Assets* tab, download or view `autoinst-log.txt` for the openQA worker and test-engine log. This is the primary log for the job itself. + +For tests that execute commands on the system under test, KDE Linux runs them in transient systemd services and records the service journal as diagnostic output in the test result. Inspect `autoinst-log.txt` for the command output and journal. + +## Run tests locally + +You can use openQA to test a locally built image before submitting a change. This is required when working from a fork outside `kde-linux/kde-linux` - its CI pipeline builds a test image, but does not trigger the openQA pipeline. It is also useful while developing tests for your own changes. + +The [KDE Linux openQA test repository](https://invent.kde.org/kde-linux/os-autoinst-distri-kdelinux) includes a local stack containing both an openQA web UI and a worker. It requires Podman and `podman-compose`, which are included in KDE Linux. + +Clone the test repository, then place the KDE Linux <code>.iso</code> image you want to test in its root directory. If no image is present, the test runner downloads the latest publicly available image instead. + +Start the local stack. + +<pre> +./mock.sh up +</pre> + +Once it is ready, open <code>http://localhost:1080</code> to inspect the openQA web UI. Jobs are not submitted automatically, so, open a shell in the container and submit the normal install and sanity-test flow. + +<pre> +podman exec -it openqa-single-instance bash +bash utils/jobs.sh +</pre> + +To run the upgrade flow instead, add <code>--upgrade</code>. + +<pre> +bash utils/jobs.sh --upgrade +</pre> + +The `sanity-test` job needs the `install-system` job to run first because it uses the virtual disk created by the installation. To concentrate on a particular test, adjust the submitted jobs in <code>utils/jobs.sh</code> while keeping that dependency in mind. + +When you are finished, stop the stack and remove its local volumes. + +<pre> +./mock.sh down -v +</pre> + +## Write a test + +The test definitions and the test code live in [os-autoinst-distri-kdelinux](https://invent.kde.org/kde-linux/os-autoinst-distri-kdelinux). Add the test to the appropriate flow in `main.pm` - the live-install flow, installed-system sanity flow, or upgrade flow. Tests are generally composed of a small openQA wrapper in `tests/` and a Python test that runs on the system under test from `extensions/openqa/usr/lib/kde-linux-openqa/tests/`. + +Choose the test type based on what you are checking: + +* Use a normal Python `unittest` for command-line tools, services, filesystem state, or other non-graphical behaviour. +* Use Selenium through `selenium-webdriver-at-spi` when the behaviour requires interacting with a graphical application. + +### Normal Python tests + +Create a `unittest` in the aforementioned system extension test directory. It should make assertions about the system under test and write its results to a JUnit XML file, which will be automatically collected as an openQA asset and a GitLab CI report. + +<pre> +import unittest +from lib.sut import openqa_junit_xml + +class ExampleTests(unittest.TestCase): + def test_expected_behaviour(self): + self.assertTrue(True) + +if __name__ == "__main__": + openqa_junit_xml.run(ExampleTests, "example") +</pre> + +Create a matching wrapper under `tests/` that gets openQA to execute the test on the host. + +<pre> +from testapi import * +from lib.openqa import cli_test + +def run(self): + cli_test.CliTest("example").run_python() +</pre> + +Finally, register the wrapper in the relevant flow in `main.pm`. The wrapper collects the JUnit result and command output so that failures appear in the openQA job and CI artifacts. + +### Graphical tests with Selenium + +Graphical tests are also Python `unittest` classes, but use the Appium/Selenium driver to find and interact with accessible UI elements. The Selenium runner enables the accessibility infrastructure and starts the driver for you. + +For example, a graphical test creates a driver for its application, interacts with accessible elements, and closes the driver afterwards: + +<pre> +import unittest +from appium import webdriver +from appium.options.common.base import AppiumOptions +from appium.webdriver.common.appiumby import AppiumBy +from lib.sut import openqa_junit_xml + +class ExampleTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + options = AppiumOptions() + options.set_capability("app", "org.kde.example.desktop") + cls.driver = webdriver.Remote("http://127.0.0.1:4723", options=options) + + @classmethod + def tearDownClass(cls): + cls.driver.quit() + + def test_expected_behaviour(self): + self.driver.find_element(AppiumBy.ACCESSIBILITY_ID, "example-control").click() + +if __name__ == "__main__": + openqa_junit_xml.run(ExampleTests, "example") +</pre> + +Run the matching wrapper with `run_selenium()` instead of `run_python()`. Pass the installed test user for desktop applications: + +<pre> +from testapi import * +from lib.openqa import cli_test +from lib import user_manager + +def run(self): + cli_test.CliTest("example").run_selenium(user=user_manager.installed()) +</pre> + +Similarly to normal Python tests, add its wrapper to `main.pm`, run it locally while developing it, and include it in the test flow that exercises the feature. + +For more information on writing Selenium tests, see [Appium automation testing](https://develop.kde.org/docs/apps/tests/appium/) and [GUI Testing with selenium-webdriver-at-spi](https://community.kde.org/Selenium). + +## Learn more + +For a more detailed explanation of the test flow and its implementation, see [openQA Testing in KDE Linux](https://blogs.kde.org/2026/07/13/openqa-testing-in-kde-linux/). The test infrastructure is developed in the [os-autoinst-distri-kdelinux repository](https://invent.kde.org/kde-linux/os-autoinst-distri-kdelinux).