Re: KernelCI report for stable-rc: v5.15.164

---- On Tue, 06 Aug 2024 07:06:00 -0300 Nikolai Kondrashov  wrote ---

 > On 8/6/24 1:03 PM, Nikolai Kondrashov via groups.io wrote: 
 > > On 8/5/24 5:40 PM, Gustavo Padovan via groups.io wrote: 
 > >>  > > Why does this have kernelci, syzbot and tuxsuite. Let's remove it for now. 
 > >>  > 
 > >>  > It lists all the CI systems the report has been aggregated with. Although 
 > > only maestro and broonie reported failures, we included 
 > >>  > builds and test stats from all the 5 CI systems. 
 > >> 
 > >> I see.  Although, we can't really guarantee the quality of data coming from 
 > >> the other origins, hence my request to use only maestro and broonie for the 
 > >> time being. We need a high quality report, not a report with as much results 
 > >> as possible. 
 > > 
 > > This is alright for a PoC. However, I'd like you guys to mention to Greg that 
 > > we have more data from other origins, when you engage him with this, and that 
 > > they're available on the dashboards. Or perhaps better just put that into the 
 > > notification message itself. 
 > > 
 > > Finally, as we've discussed before, if a maintainer has requirements for 
 > > particular result quality, it's best to directly apply them to data, where 
 > > possible (e.g. check that log_excerpt is there), instead of simply selecting 
 > > origins. That could be the next step, once we define what "good quality" 
 > > means exactly. 
 >  
 > This is what all the CI systems signed up for, after all: reaching out to 
 > maintainers. And we would be breaking our promise, if we make these decisions 
 > for them, or discriminate by origin. 

Not exactly. That would be the perfectionist take.  We are investing a lot in the test
and results data quality so we can be sure we can be sure we are showing good,
high quality data to maintainers we are engaging actively. 

That's exactly the case of the stable-rc report. Greg has said to us already that just
throwing data at him wouldn't help much. We have to make sure that data can actually 
help him. This has been our goal since we started the current stable-rc report process
since LPC. The Collabora team has invested dozens and dozens of ours into evaluating
and improving the quality of the data.

In summary, we are not breaking any promises. We can really throw any data we have into
maintainers and expect that would be helping them.

Best,

- Gus