Re: KCIDB: Support non-binary test outputs

On 8/5/24 3:33 PM, Mark Brown wrote:
> On Mon, Aug 05, 2024 at 02:28:24PM +0300, Nikolai Kondrashov wrote:
> 
>> Here is the (abbreviated) schema for the new field:
> 
>>     "value": {
>>         "type": "object",
>>         "properties": {
>>             "integer": {"type": "integer"},
>>             "number": {"type": "number"},
>>             "string": {"type": "string"},
>>             "boolean": {"type": "boolean"},
>>         },
>>         "minProperties": 1,
>>         "additionalProperties": False
>>     },
> 
>> The meaning of the value itself depends on the particular test, that is the
>> "path" field value. Each property inside the value corresponds to a data type.
> 
> Might it be useful to directly specify units for use with "number" to
> help with normalising data between different CI systems or hardware -
> for example with boot times both seconds and miliseconds seem like
> reasonable units to use?  It might also be useful for UIs, though we
> could also do that with a separate table for the tests that they can
> query.  Perhaps I'm just worrying too much about specialist cases where
> it's likely that CI systems won't just be picking up an off the shelf
> suite that has standard units.
This is a totally valid concern. We could add a "units" field, e.g. beside
"value". However, I'm not sure how we could use it. Sure, we can put them next
to the value in the dashboard, which would look nice, but then we can also put
them into the "comment", as my examples do, and I'm not sure if we would be
able to do much with them in the database.

Even if we have the exponent separate, I'm not sure we can make use of it
(converting units on the fly won't work well with indices). I mean, we can
(and should) compare only values with the same unit, but which one, if we have
multiple? Both separately?

On the one hand I like having the explicit unit, on the other hand we can get
a similar result with simply using different test names for different units,
and have them specified in the comments, for humans 🤔

Nick