Re: [Automated-testing] KCIDB: Support non-binary test outputs

On 8/5/24 8:02 PM, Tim Bird wrote:
>> -----Original Message-----
>> From: Nikolai Kondrashov <[email protected]>
>> On 8/5/24 12:56 PM, Nikolai Kondrashov wrote:
>> The support for non-binary test outputs introduces a new field to the "test"
>> objects called "value", being an "object" itself. It (and its abstract
>> meaning) are supposed to work together with the "status" field. That is, it
>> should only be considered when the test has actually executed. I.e. with a
>> "FAIL", "ERROR", "PASS", or "DONE" status only. Normally "DONE" should be
>> used, when the value is the test's output, and not an auxiliary value.
>
> I'm not sure I'm reading this correctly, but as I understand it, this
pre-supposes
> that the value and the testcase are synonymous.  Is that right?

Yep. But don't take the meaning of "testcase" too literally. You can report as
many "synthetic" cases as you need, to report your data. They don't actually
have to correspond to actual "testcases". View them as a tool to express your
results. Just make sure you get the agreement of anyone else reporting the
same data on how you're going to do it.

> Are there values that are not assigned to a testcase?

I don't know? I haven't seen the gamut of the data we need. However, we can
always make up a "testcase" and assign them there, if needed, as I say above.

> In my experience, many tests (particularly IO performance tests) produce
> a whole lot of values (basically a matrix for combinations of different IO
sizes,
> IO direction (read/write), patterns (sequential vs random), and scheduling
> classes).  It's quite common to have a tester select only a few values to
> convert into testcases (that is, items that would cause a test to pass or fail).

Sure. To decide which values you want reported explicitly, consider which ones
you want KCIDB to be able to track, graph, and report deviations on
(eventually). You can put the the complete data into our free-form "misc"
field, if you'd still like to be able to get to it (but not query or analyze
it within KCIDB).

>> The meaning of the value itself depends on the particular test, that is the
>> "path" field value. Each property inside the value corresponds to a data type.
>> At least one must be specified, but if more than one type property is set,
>> each is considered a different representation of the *same* value, and not a
>> different value. E.g. these can be specified at the same time: "integer": 1,
>> "number": 1, "boolean": true, "string": "true".
>
> I'm not sure what the intended use is for these different type properties.
> Almost universally, benchmark data is expressed as numeric values (that is,
> numbers).  Are these other types used to hold intermediate formats?
> If so, for what reason?  I would suggest dropping the type field, and just
> making them all numbers.

We don't really have a type field, we just have separate fields for different
types. I don't really know what types we would want. I just showed some types
there, mostly as an illustration. But I think we might need the floating-point
and the integers separately, so different precision requirements could be
satisfied. That alone is enough to require us to support different types.

Regarding the strings, I suppose some tests (perhaps not performance ones)
could produce some discrete categorical outputs, and then we would be able to
graph and track them. But again, I haven't actually seen any.

The boolean was added just to completely cover the atomic types, although I
could probably think of a usecase, we can easily drop it, as well as the
strings, and add them when necessary, no problem.

Nick