Re: KCIDB: Support non-binary test outputs

On Tue, Aug 06, 2024 at 08:03:00PM +0300, Nikolai Kondrashov wrote:

> In that light, I think validation is the right way here. And validating
> submitter-provided units could be the right way. This is actually similar to
> the situation with test "paths". Currently we're accepting everything, but our
> target is to tighten that down to help correlation. Perhaps by sending
> (aggregated) warnings about unknown test paths to submitters.

TBH as a submitter getting specific stuff back immediately (or at least
the option for it) is really helpful.

> This looser approach allows us to admit new data to the database faster, as it
> doesn't need to undergo cataloguing first. We certainly don't want to spend
> time arbitrating *every* test name or a measurement unit at this point, and we
> don't want to slow down adoption from CI systems, and introduction of new
> tests. But I think letting them know about deviations could work. After all,
> it's in the CI system interest to comply, as that improves result quality, and
> raises the chance of reaching maintainers, which they're here for.

Perhaps per test schemas of some kind (not sure how exactly you'd go
about doing it) could help here, if the test is unknown then just let it
in but if it's a test we know about and we've defined the units for then
enforce those units?  That way there's the looser stuff and reporting
that shows what we could work on standardising, and things that have
been standardised are hopefully going to be more joined up?

> However, I would leave it to the submitter to observe the correct exponent
> (e.g. KB vs. MB vs. GB). We can perhaps standardize on units *without* the
> metric prefix, and rely on either floating-point exponent, or larger integer
> representations (like 8-byte bigints in PostgreSQL, JSON integers have no size
> themselves) to handle the required ranges. This way the dashboard would be
> able to display the units, *and* apply the prefix as necessary, automatically.
> Or perhaps specify the prefix (or exponent) separately from the unit, so the
> usual range would fit, but the dashboard could still scale the numbers on
> display. OTOH, indices would be no use for separate value/exponent
> representation.

Putting the exponent in as a number does seem like it'd be much more
helpful for machine processing.

> And the final question we need to consider is how many people would actually
> bother finding out what the units exactly are in their tests, and specifying
> them?

I suspect there's going to be a fair amount of stuff where there's a
fairly clear specific unit for one reason or other that's commonly used
when talking about the test (eg, things like I/O benchmarks, run times,
or temperatures) so it'll be immediately obvious and also a bunch of
things where the number is just a number for the benchmark and nobody
cares about whatever the units actually are anyway.

-----BEGIN PGP SIGNATURE-----

iQEzBAABCgAdFiEEreZoqmdXGLWf4p/qJNaLcl1Uh9AFAmayd5QACgkQJNaLcl1U
h9B+kQf/YruqJnaYcX/+F9OXC1UQVaOcmK+fOwBLzTu7v1morU9dUC37OCjbNWJ7
3rZpxQlpBdfscF0fqdLUxbuHKvdCXVac83Bqds5Li/0dpgq3mlY4ojmbedCobeQi
hAl78mqcI1CuIGFt8RRpw0Fjaaj5WLelRnoxhdQt0EBVkdqWRN7R4nKP2ju9WaBC
Pxe2sMsT+7kR4mRnI0q0idTVmPp1j8P1wXUrUOc0uzqmkGLC+daSjABa/qMWLGw+
OJHquasfWMNFLZRd+E+lwdKGPUOUn4PQeyIegu5HGTdd0AfEXUSK4u9R4RUWa7T1
vfvTTEaQmQRVxVgvC5ALgBwJOS8U+Q==
=FzD1
-----END PGP SIGNATURE-----