Re: [Automated-testing] KCIDB: Support non-binary test outputs

(Sorry, re-sending, as I lost all recipients accidentally on the last try)

Hi Tim,

Thank you for your responses!

On 8/5/24 8:11 PM, Tim Bird wrote:
>> -----Original Message-----
>> From: Mark Brown <[email protected]>
>> Sent: Monday, August 5, 2024 6:33 AM
>> To: Nikolai Kondrashov <[email protected]>
>> On Mon, Aug 05, 2024 at 02:28:24PM +0300, Nikolai Kondrashov wrote:
>>
>>> The meaning of the value itself depends on the particular test, that is the
>>> "path" field value. Each property inside the value corresponds to a data type.
>>
>> Might it be useful to directly specify units for use with "number" to
>> help with normalising data between different CI systems or hardware -
>> for example with boot times both seconds and miliseconds seem like
>> reasonable units to use?  It might also be useful for UIs, though we
>> could also do that with a separate table for the tests that they can
>> query.  Perhaps I'm just worrying too much about specialist cases where
>> it's likely that CI systems won't just be picking up an off the shelf
>> suite that has standard units.
>
> I'll second this.  A number of benchmarks output their values in different
> units, depending on the performance of the system, and it's valuable
> to be able to detect that different units are being used, in order to
> compare results effectively.  For example, some IO tests will report KB/s
> on a slow machine and MB/s on a fast machine, for the same measurement.
>
> Tests sometimes need a policy for which unit to be the canonical one for the
> test (and to express reference values  in that canonical unit format).  This
> might require a test results parser to do units conversion, before comparison
> with reference values in order to detect testcase results.

JSON doesn't really have an integer or floating-point size, so I would prefer
that the submitting CI system stuck to a single exponent / metric prefix in
their reports, and scaled the values itself, to help us avoid (complicated)
pre-processing, and help the database indexes improve query performance. And,
after all, they know their tests best to pick the right unit/scale.

If we agree on prefix-less units, for example, then the dashboards would be
able to scale the data and apply their own prefix, as needed, automatically.

Please also see my response to Mark, for more considerations.

Thanks!
Nick