Re: KCIDB: Support non-binary test outputs

On 8/6/24 10:20 PM, Mark Brown wrote:
> On Tue, Aug 06, 2024 at 08:03:00PM +0300, Nikolai Kondrashov wrote:
>
>> In that light, I think validation is the right way here. And validating
>> submitter-provided units could be the right way. This is actually similar to
>> the situation with test "paths". Currently we're accepting everything, but our
>> target is to tighten that down to help correlation. Perhaps by sending
>> (aggregated) warnings about unknown test paths to submitters.
>
> TBH as a submitter getting specific stuff back immediately (or at least
> the option for it) is really helpful.

Do you mean that you would like to know if you submitted something wrong
immediately?

Could you elaborate a bit here?

>> This looser approach allows us to admit new data to the database faster, as it
>> doesn't need to undergo cataloguing first. We certainly don't want to spend
>> time arbitrating *every* test name or a measurement unit at this point, and we
>> don't want to slow down adoption from CI systems, and introduction of new
>> tests. But I think letting them know about deviations could work. After all,
>> it's in the CI system interest to comply, as that improves result quality, and
>> raises the chance of reaching maintainers, which they're here for.
>
> Perhaps per test schemas of some kind (not sure how exactly you'd go
> about doing it) could help here, if the test is unknown then just let it
> in but if it's a test we know about and we've defined the units for then
> enforce those units?  That way there's the looser stuff and reporting
> that shows what we could work on standardising, and things that have
> been standardised are hopefully going to be more joined up?

Yes, that's the approach I was thinking about.

I'm not sure if we would be able to implement this kind of checking in JSON
schema, not likely. But we can certainly always implement a higher-order
check, which we can run on database data after it's updated.

>> However, I would leave it to the submitter to observe the correct exponent
>> (e.g. KB vs. MB vs. GB). We can perhaps standardize on units *without* the
>> metric prefix, and rely on either floating-point exponent, or larger integer
>> representations (like 8-byte bigints in PostgreSQL, JSON integers have no size
>> themselves) to handle the required ranges. This way the dashboard would be
>> able to display the units, *and* apply the prefix as necessary, automatically.
>> Or perhaps specify the prefix (or exponent) separately from the unit, so the
>> usual range would fit, but the dashboard could still scale the numbers on
>> display. OTOH, indices would be no use for separate value/exponent
>> representation.
>
> Putting the exponent in as a number does seem like it'd be much more
> helpful for machine processing.

It's helpful for normalizing to a single exponent, so data can be compared,
but that's not something that a database can do fast. So, not helpful in the
end.

>> And the final question we need to consider is how many people would actually
>> bother finding out what the units exactly are in their tests, and specifying
>> them?
>
> I suspect there's going to be a fair amount of stuff where there's a
> fairly clear specific unit for one reason or other that's commonly used
> when talking about the test (eg, things like I/O benchmarks, run times,
> or temperatures) so it'll be immediately obvious and also a bunch of
> things where the number is just a number for the benchmark and nobody
> cares about whatever the units actually are anyway.

Yeah, if we make the unit optional, and validate it only if it's specified in
requirements, then we can both have the cake and eat it too. That is have more
alignment where we want, and more freedom, where we want to experiment.

I'll post a new version of the schema with the optional unit string, which
would be expecting people to maintain it uniform, use wider data types, so
they don't need to switch the prefix/exponent, but would keep them constant,
and we can add validation later.

Thank you everyone for the feedback!

Nick