Re: KCIDB: Support non-binary test outputs

Hi everyone,

On 8/5/24 12:56 PM, Nikolai Kondrashov wrote:
> Meanwhile I'd like to propose two small, but potentially very useful changes
> to the I/O schema for tests:
>
> * Supporting non-binary outputs beyond PASS/FAIL -
>   integers/floats/booleans/strings/etc. - useful for performance tests.
>
> * Supporting recording `compatible` values from the top of the device tree
>   inside the test environment, for machines which use them - useful for
>   correlating test results by hardware.
>
> Here's the corresponding schema PR:
>
>     https://github.com/kernelci/kcidb-io/pull/85
>
> I'll follow up with a separate message for each of the changes, going over the
> details and the rationale.

The support for non-binary test outputs introduces a new field to the "test"
objects called "value", being an "object" itself. It (and its abstract
meaning) are supposed to work together with the "status" field. That is, it
should only be considered when the test has actually executed. I.e. with a
"FAIL", "ERROR", "PASS", or "DONE" status only. Normally "DONE" should be
used, when the value is the test's output, and not an auxiliary value.

Here is the (abbreviated) schema for the new field:

    "value": {
        "type": "object",
        "properties": {
            "integer": {"type": "integer"},
            "number": {"type": "number"},
            "string": {"type": "string"},
            "boolean": {"type": "boolean"},
        },
        "minProperties": 1,
        "additionalProperties": False
    },

The meaning of the value itself depends on the particular test, that is the
"path" field value. Each property inside the value corresponds to a data type.
At least one must be specified, but if more than one type property is set,
each is considered a different representation of the *same* value, and not a
different value. E.g. these can be specified at the same time: "integer": 1,
"number": 1, "boolean": true, "string": "true".

Specifying multiple types at once can be used to assist transitioning
test output to a different type, but in normal use only one of them
should be supplied.

Here's an example test object using the "value" field:

{
    "id": "redhat:2876829c98e9878766a",
    "origin": "redhat",
    "build_id": "redhat:387d3459ef",
    "path":"redhat_ext4fs.performance.iops",
    "comment": "Red Hat Ext4 FS I/O performance, IOPS"
    "status": "DONE",
    "value": {"integer": 57324}
}

If the test in question has multiple values to report, then the submitting CI
system should create (synthetic) subtests under the test node, one for each
value. E.g. if the (imaginary) test above had random-read and random-write
IOPS to report separately, it could've been expressed as such:

"tests": [
    {
        "id": "redhat:2876829c98e9878766a",
        "origin": "redhat",
        "build_id": "redhat:387d3459ef",
        "path":"redhat_ext4fs.performance",
        "comment": "Red Hat Ext4 FS I/O performance"
        "status": "DONE"
    },
    {
        "id": "redhat:2876829c98e9878766a:rriops",
        "origin": "redhat",
        "build_id": "redhat:387d3459ef",
        "path":"redhat_ext4fs.performance.rriops",
        "comment": "Red Hat Ext4 FS I/O performance, random-read IOPS"
        "status": "DONE",
        "value": {"integer": 97524}
    },
    {
        "id": "redhat:2876829c98e9878766a:rwiops",
        "origin": "redhat",
        "build_id": "redhat:387d3459ef",
        "path":"redhat_ext4fs.performance.rwiops",
        "comment": "Red Hat Ext4 FS I/O performance, random-write IOPS"
        "status": "DONE",
        "value": {"integer": 46434}
    }
]

This separation allows us to control complexity, while at the same time
allowing KCIDB to analyze and track results in a more-or-less generic way.

Finally, if you need to monitor both the separate results and a combined
result, you simply add it to the parent test. Like adding back the overall
IOPS value to the parent test above.

Don't hesitate to send your comments/questions/objections here or in the PR!

Nick