RE: KCIDB: Support non-binary test outputs

> -----Original Message-----
> From: Nikolai Kondrashov <[email protected]>
> Hi everyone,
> 
> On 8/5/24 12:56 PM, Nikolai Kondrashov wrote:
> > Meanwhile I'd like to propose two small, but potentially very useful changes
> > to the I/O schema for tests:
> >
> > * Supporting non-binary outputs beyond PASS/FAIL -
> >   integers/floats/booleans/strings/etc. - useful for performance tests.
> >
> > * Supporting recording `compatible` values from the top of the device tree
> >   inside the test environment, for machines which use them - useful for
> >   correlating test results by hardware.
> >
> > Here's the corresponding schema PR:
> >
> >     https://github.com/kernelci/kcidb-io/pull/85
> >
> > I'll follow up with a separate message for each of the changes, going over the
> > details and the rationale.
> 
> The support for non-binary test outputs introduces a new field to the "test"
> objects called "value", being an "object" itself. It (and its abstract
> meaning) are supposed to work together with the "status" field. That is, it
> should only be considered when the test has actually executed. I.e. with a
> "FAIL", "ERROR", "PASS", or "DONE" status only. Normally "DONE" should be
> used, when the value is the test's output, and not an auxiliary value.

I'm not sure I'm reading this correctly, but as I understand it, this pre-supposes
that the value and the testcase are synonymous.  Is that right?  Are there
values that are not assigned to a testcase?

In my experience, many tests (particularly IO performance tests) produce
a whole lot of values (basically a matrix for combinations of different IO sizes,
IO direction (read/write), patterns (sequential vs random), and scheduling
classes).  It's quite common to have a tester select only a few values to
convert into testcases (that is, items that would cause a test to pass or fail).

> 
> Here is the (abbreviated) schema for the new field:
> 
>     "value": {
>         "type": "object",
>         "properties": {
>             "integer": {"type": "integer"},
>             "number": {"type": "number"},
>             "string": {"type": "string"},
>             "boolean": {"type": "boolean"},
>         },
>         "minProperties": 1,
>         "additionalProperties": False
>     },
> 
> The meaning of the value itself depends on the particular test, that is the
> "path" field value. Each property inside the value corresponds to a data type.
> At least one must be specified, but if more than one type property is set,
> each is considered a different representation of the *same* value, and not a
> different value. E.g. these can be specified at the same time: "integer": 1,
> "number": 1, "boolean": true, "string": "true".

I'm not sure what the intended use is for these different type properties.
Almost universally, benchmark data is expressed as numeric values (that is,
numbers).  Are these other types used to hold intermediate formats?
If so, for what reason?  I would suggest dropping the type field, and just
making them all numbers.
 
> Specifying multiple types at once can be used to assist transitioning
> test output to a different type, but in normal use only one of them
> should be supplied.
> 
> Here's an example test object using the "value" field:
> 
> {
>     "id": "redhat:2876829c98e9878766a",
>     "origin": "redhat",
>     "build_id": "redhat:387d3459ef",
>     "path":"redhat_ext4fs.performance.iops",
>     "comment": "Red Hat Ext4 FS I/O performance, IOPS"
>     "status": "DONE",
>     "value": {"integer": 57324}
> }
> 
> If the test in question has multiple values to report, then the submitting CI
> system should create (synthetic) subtests under the test node, one for each
> value. E.g. if the (imaginary) test above had random-read and random-write
> IOPS to report separately, it could've been expressed as such:
> 
> "tests": [
>     {
>         "id": "redhat:2876829c98e9878766a",
>         "origin": "redhat",
>         "build_id": "redhat:387d3459ef",
>         "path":"redhat_ext4fs.performance",
>         "comment": "Red Hat Ext4 FS I/O performance"
>         "status": "DONE"
>     },
>     {
>         "id": "redhat:2876829c98e9878766a:rriops",
>         "origin": "redhat",
>         "build_id": "redhat:387d3459ef",
>         "path":"redhat_ext4fs.performance.rriops",
>         "comment": "Red Hat Ext4 FS I/O performance, random-read IOPS"
>         "status": "DONE",
>         "value": {"integer": 97524}
>     },
>     {
>         "id": "redhat:2876829c98e9878766a:rwiops",
>         "origin": "redhat",
>         "build_id": "redhat:387d3459ef",
>         "path":"redhat_ext4fs.performance.rwiops",
>         "comment": "Red Hat Ext4 FS I/O performance, random-write IOPS"
>         "status": "DONE",
>         "value": {"integer": 46434}
>     }
> ]
> 
> This separation allows us to control complexity, while at the same time
> allowing KCIDB to analyze and track results in a more-or-less generic way.
> 
> Finally, if you need to monitor both the separate results and a combined
> result, you simply add it to the parent test. Like adding back the overall
> IOPS value to the parent test above.
> 
> Don't hesitate to send your comments/questions/objections here or in the PR!
> 
> Nick