RE: KCIDB: Support non-binary test outputs

> -----Original Message-----
> From: Mark Brown <[email protected]>
> Sent: Monday, August 5, 2024 6:33 AM
> To: Nikolai Kondrashov <[email protected]>
> Cc: syzkaller <[email protected]>; Dmitry Vyukov <[email protected]>; Vishal Bhoj <[email protected]>; Alice Ferrazzi
> <[email protected]>; [email protected]; Cristian Marussi <[email protected]>; Bird, Tim
> <[email protected]>; Johnson George <[email protected]>; Veronika Kabatova <[email protected]>; Guillaume Tucker
> <[email protected]>; [email protected]; Don Zickus <[email protected]>; Philip Li <[email protected]>; Denys
> Fedoryshchenko <[email protected]>; Michael Hofmann <[email protected]>; Tales da Aparecida <[email protected]>;
> Aditya Nagesh <[email protected]>; Jeny Dhruvit Sheth <[email protected]>; Sachin Sant <[email protected]>;
> Hambardzumyan, Minas <[email protected]>
> Subject: Re: KCIDB: Support non-binary test outputs
> 
> On Mon, Aug 05, 2024 at 02:28:24PM +0300, Nikolai Kondrashov wrote:
> 
> > Here is the (abbreviated) schema for the new field:
> 
> >     "value": {
> >         "type": "object",
> >         "properties": {
> >             "integer": {"type": "integer"},
> >             "number": {"type": "number"},
> >             "string": {"type": "string"},
> >             "boolean": {"type": "boolean"},
> >         },
> >         "minProperties": 1,
> >         "additionalProperties": False
> >     },
> 
> > The meaning of the value itself depends on the particular test, that is the
> > "path" field value. Each property inside the value corresponds to a data type.
> 
> Might it be useful to directly specify units for use with "number" to
> help with normalising data between different CI systems or hardware -
> for example with boot times both seconds and miliseconds seem like
> reasonable units to use?  It might also be useful for UIs, though we
> could also do that with a separate table for the tests that they can
> query.  Perhaps I'm just worrying too much about specialist cases where
> it's likely that CI systems won't just be picking up an off the shelf
> suite that has standard units.

I'll second this.  A number of benchmarks output their values in different
units, depending on the performance of the system, and it's valuable
to be able to detect that different units are being used, in order to 
compare results effectively.  For example, some IO tests will report KB/s
on a slow machine and MB/s on a fast machine, for the same measurement.

Tests sometimes need a policy for which unit to be the canonical one for the
test (and to express reference values  in that canonical unit format).  This
might require a test results parser to do units conversion, before comparison
with reference values in order to detect testcase results.
 -- Tim