[MAINTAINERS SUMMIT] Satisfied with the time it takes regressions to be fixed?

Linus' rule of thumb with regards to fixing regressions that have a
known culprit and were reported by a user afaics is: "within a week,
preferably before the next rc". He wrote that in January, for the exact
words and links to the full mails on lore from January see:
https://www.kernel.org/doc/html/latest/process/handling-regressions.html#on-how-quickly-regressions-should-be-fixed


I got the impression that most regressions I track take much longer to
get fixed. Which turned out to be the case when looked closer at a
recent development cycle, as more that half took more then two weeks to
get fixed; the analysis furthermore showed that fixes had usually been
available quickly, it just took them some time to get reviewed,
committed, and sent to Linus -- among others due to lack of maintainer
resources. More details on that below, let me first propose what we
might want to discuss the following here and later at the summit:


Is Linus asking too much here? If not, what can we do to improve the
time it typically takes to fix regressions -- without causing more
trouble for already overloaded maintainers?


Back to the analyses briefly mentioned above, which is based on a closer
look at 39 regressions I tracked during the 7.1 cycle that were reported
against either 7.1-rc or 7.0.

First I looked at how long it took to resolve those regressions once at
least a rough fix was known; the latter usually was the case right with
the report, as many stated that a revert helped. For 4 of the 39
regressions I tracked, a proper fix hit the next -rc that followed;
fixes for 8 in total (this thus included the 4 from before) hit mainline
within a week. 14 in total reached the second -rc within two weeks; 18
in total reached mainline within two weeks. The overall median time was
20 days; in 19 cases it took more than three weeks till a fix hit a -rc,
in 5 of those even more than five weeks.

Then I looked at how long it had taken the committed fixes for those 39
regressions from submission for review till hitting mainline: 6 of the
fixes managed to reach the first -rc within a week of the submission,
and 12 in total reached mainline within a week. 15 in total reached the
second -rc and 20 in total hit mainline in two weeks. The median was 18
days; 14 fixes took more than three weeks from the submission till
hitting a mainline -rc.

Occasionally it took longer than Linus' rule of thumb due to cases like
"mistakes happen" or "subsystem with single maintainer temporarily busy
or unavailable". There were also a few cases where things were delayed
by "developer/maintainer tried hard to avoid a revert". But the main
reason why it often took longer than one or two weeks, afaics, lies
somewhere else: only some subsystems regularly manage to review, commit,
and mainline regression fixes within a week. This assessment is based
less on the analyzed time frame and more on what I've seen in the last
two or three years of tracking regressions.

Quick reminder for those that might not have been aware of Linus' rule
of thumb mentioned initially: I brought "time it takes to fix
regressions" up two and four years ago at the maintainers summit (and
each time before that on this list). That two years ago led to a
reworked "Expectations and best practices for fixing regressions"
section in Documentation/process/handling-regressions.rst. What's
outlined there is somewhat similar to what Linus' expects -- albeit it's
a bit more nuanced and even a little laxer in some cases:
https://www.kernel.org/doc/html/latest/process/handling-regressions.html#expectations-and-best-practices-for-fixing-regressions


Also be aware that my tracking (and thus the numbers here) only cover a
subset of regressions. Among others, because I never become aware of
some and focus on those users reported. So more than the 39 regressions
got fixed during that timeframe, some of which might have been fixed
quicker. At the same time, me prodding maintainers in a few cases
prevented some regressions from taking even longer to get fixed.

With that out of the way back to the "only some subsystems regularly
manage to [...] mainline regression fixes within a week" aspect.
Mistakes occasionally happen in those that as well; but their
maintainers overall handle regressions better by doing the following
things (in order of importance based on subsystems where things don't
work that well from what I've seen over the years):

* They have the motivation and the resources (aka time) to handle
regressions well; the latter is apparently usually thanks to an employer
backing fully those efforts and/or by spreading the load among enough
maintainers.

* They regularly send all committed fixes to Linus shortly before a new
-rc, e.g., between Friday and Sunday -- even if that means sending a
second PR in a week (if all subsystems did that, the numbers above would
look quite a bit better).

* They review and commit regression fixes quickly, e.g., usually within
two or three workdays.

* They submit and apply reverts if no proper fix gets into reach within
a few days, e.g., usually within something like two or three workdays.

* Those that have them ensure all their downstream maintainers handle
regressions and fixes for them well (e.g. like outlined here, esp. the
three points above), too -- and step in to handle things themselves when
their downstream maintainers temporarily or generally are unable to do
so (for example due to a lack of resources -- which from what I see in
my tracking efforts is the case in a few popular sub-subsystems that
frequently show up on my radar due to unfixed regressions).

* They have multiple maintainers and are thus responsive, even if some
of them are unavailable due to vacation or something.

* If they have internal QA queues, it usually takes them no more than
about a day or two to chew on regression fixes.

* They ensure their developers handle regressions like Linus wants them
to be handled -- and when they fail to do that, step in to handle things
themselves.

* They enforce tags like "current" and "next" in the subject of patch
submissions for their subsystem; this forces developers to think about
where patches should head to, while also allowing reviewers and
maintainers to easily filter regression fixes to prioritize them over
feature work.

* They maintain separate -fixes and -next branches that are in
linux-next independently; they furthermore send "your patch was applied
to <branch>" mails when patches are applied. That way everyone
(developers, me, affected users, ...) can easily check the status and
speak up when a fix is on the wrong track or suddenly becomes more urgent.

* They expedite fixing regressions that affect a lot of people, for
example if the culprit made it to a proper (non-rc) mainline release or
to a stable-series through a backport. Linus to directly pick up a fix
directly from the list is one way to do that.

* They do not put an unnecessary burden on reporters (I see it more
often these days that developers or maintainers ask reporters to submit
reverts or improve POC regression fixes, even if the person has never or
rarely contributed to the kernel before and in no way indicated any
interest in getting their name into the kernel history).

* They generally prioritize dealing with reports about regression and
fixes over other kernel work.

* They know the lesser-known aspects about handling regression fixes,
for example, that -next can often be skipped or that most fixes for
recent regression (e.g., introduced in a recent or the current cycle)
are fine to mainline even post -rc7. Note, be aware of the "usually" and
"most" above, as that in the end needs to be a case by case decision
depending on the risk of the particular change.


I tried to keep things generic and mostly positive up to this point, but
sometimes concrete examples are helpful to properly understand problem
points. So here are the commit-ids of a few, all of them from the
analyzed 7.1 cycle introduced in either 7.1 or 7.0 (note, some commits
are in this list multiple times; getting the full picture sometimes also
requires following Link: tags or a search on lore):

* Fixes for regressions that could have been fixed way earlier by opting
for a revert: 2914709c914101 f78073e84c80 7495adaa0e45e1 f23bf992d65a42

* Fixes that took more than a week to get committed: 629279e2e798cd
2914709c914101 2eeb342aff6647 ed831e7ea1a860 c0c6ccd9828c3a
522567362b6340 f78073e84c80 4867cef03b58ca 71f42b2149a1 ec78a85d95e9c3
5772f6535227eb

* Fixes that took more than 9 days from committing till reaching -rc:
a39f0ce0c9da20 426e5846eba75f 635b5c6622f317 629279e2e798cd
e3ac0d9f1a205f 54cf41c969da66 5629eec1a28298 12f58a6caad3be
e4e9b7b38d5db2 c0c6ccd9828c3a 63d2059cd665c4 a8878e19d2f520 71f42b2149a1
7495adaa0e45e1 f23bf992d65a42 754d60ad1c9189

* Fixes that were delayed because they first were scheduled to be merged
in the next merge window by accident or by not thinking about it:
2914709c914101 593889c4014260

* Fixes that missed the 7.1 release where the regression could have
easily been fixed already by 7.1-rc7: a39f0ce0c9da20 426e5846eba75f
635b5c6622f317 12f58a6caad3be 7495adaa0e45e1

* Fixes for regressions that affected many users but nevertheless took
some time to fix despite available fixes: 12f58a6caad3be e3ac0d9f1a205f

As mentioned earlier, in some cases I prodded developers or maintainers,
otherwise it would have taken some even longer to get the regressions
fixed. Occasionally it works and I sometimes even get a "thanks" for
that. Most often is does not, among others because I get answers along
the lines of "this can wait till the end of the development cycle" (even
for regressions that made it into a stable release), "you are asking too
much here", "please leave us alone, we handle this already the best we
can". Sometimes I then point to to the "Expectations and best practices
for fixing regressions" or the rule of thumb from Linus initially
mentioned -- often without much luck. Which is why I think it might be
worth discussing this again on the summit.

Ciao, Thorsten

P.S.: Here is the table I used to generate the numbers:
https://docs.google.com/spreadsheets/d/1lp5bgLf8T4TRcAVqUmRviIAWxol8tQFUQ963HS0xscw/edit?usp=sharing