Re: KCIDB: Support non-binary test outputs

On 8/5/24 7:32 PM, Mark Brown wrote:
> On Mon, Aug 05, 2024 at 05:26:52PM +0300, Nikolai Kondrashov wrote:
>> On 8/5/24 3:33 PM, Mark Brown wrote:
>>> On Mon, Aug 05, 2024 at 02:28:24PM +0300, Nikolai Kondrashov wrote:
> 
>>> Might it be useful to directly specify units for use with "number" to
>>> help with normalising data between different CI systems or hardware -
> 
>> This is a totally valid concern. We could add a "units" field, e.g. beside
>> "value". However, I'm not sure how we could use it. Sure, we can put them next
>> to the value in the dashboard, which would look nice, but then we can also put
>> them into the "comment", as my examples do, and I'm not sure if we would be
>> able to do much with them in the database.
> 
>> Even if we have the exponent separate, I'm not sure we can make use of it
>> (converting units on the fly won't work well with indices). I mean, we can
>> (and should) compare only values with the same unit, but which one, if we have
>> multiple? Both separately?
> 
>> On the one hand I like having the explicit unit, on the other hand we can get
>> a similar result with simply using different test names for different units,
>> and have them specified in the comments, for humans 🤔
> 
> Just thinking out loud here but perhaps what we want is something in the
> ingestion path which either validates a schema that says "Test X must
> have unit Y" or normalises the units on the way in.  I think my concern
> is more on the write side than on the read side (modulo the display
> stuff), or rather making sure that what's available to the read side can
> be joined up.

So far, one of the major principles of KCIDB was "what you send is what you
get back", that is, it doesn't ever modify the data submitted by CI systems
(well, except perhaps some corner cases like floating point or timestamp
precision differences). I'm not saying we should *always* keep to this
principle in the future, but it *does* simplify a lot of reasoning about how
data is processed and queried. Also, the need to preprocess submitted data is
a good sign that your schema needs improving.

In that light, I think validation is the right way here. And validating
submitter-provided units could be the right way. This is actually similar to
the situation with test "paths". Currently we're accepting everything, but our
target is to tighten that down to help correlation. Perhaps by sending
(aggregated) warnings about unknown test paths to submitters.

This looser approach allows us to admit new data to the database faster, as it
doesn't need to undergo cataloguing first. We certainly don't want to spend
time arbitrating *every* test name or a measurement unit at this point, and we
don't want to slow down adoption from CI systems, and introduction of new
tests. But I think letting them know about deviations could work. After all,
it's in the CI system interest to comply, as that improves result quality, and
raises the chance of reaching maintainers, which they're here for.

And then we need to consider how many different test paths there could be. In
the past six months the database has seen 88122 unique test paths. Yep, eighty
eight thousands. Of course, tests reporting a value would be a tiny fraction
of those, but even if it's just 1%, that's still almost nine hundred paths,
seen in six months. Most likely we can do quite well with regular expressions,
as many of those are only a bit different.

So, we can have such a catalog, and for example only validate the matching
entries, instead of requiring an entry to be there before accepting the data.
That would keep it easy to send new data, but with the catalog you would have
tighter control for stuff you care about.

However, I would leave it to the submitter to observe the correct exponent
(e.g. KB vs. MB vs. GB). We can perhaps standardize on units *without* the
metric prefix, and rely on either floating-point exponent, or larger integer
representations (like 8-byte bigints in PostgreSQL, JSON integers have no size
themselves) to handle the required ranges. This way the dashboard would be
able to display the units, *and* apply the prefix as necessary, automatically.
Or perhaps specify the prefix (or exponent) separately from the unit, so the
usual range would fit, but the dashboard could still scale the numbers on
display. OTOH, indices would be no use for separate value/exponent
representation.

So, we could start with adding a string field for the unit, and e.g. an
integer field for the exponent (10^E), both stored in the database, and
combined for display. And we could add the validation later.

Or we can bake in the exponent into the unit, and make use of indexes, but
require people to pay more attention to them.

Oooorr, we could implement as is for the start, and add units later, when we
see the situation better, based on what's actually coming in. This would help
us implement the dashboard support and regression tracking easier. As we would
only need to correlate across test paths (and other parameters, of course),
without the need to add the units to all the queries and considerations. The
submitters would be able to clarify the units in the test paths and the
comments.

And the final question we need to consider is how many people would actually
bother finding out what the units exactly are in their tests, and specifying
them?

Sorry for the wall of text :D
Thanks for reading!
Nick