I think the #Linux#kernel's praised "no regression" rule it massively subverted by one aspect:
Developers that caused a regression in a mainline release (say today's 6.4) are only obliged to fix it in mainline (e.g. for the next release, 6.5 currently).
There is nothing that ensures the fix will be backported to affected stable (e.g. 6.4.y) kernels (ideally quickly, unless the fix is risky).
@kernellogger isn't it standard practice for the developer to explicitly cc: stable@... ?
I mean there's nothing that ensures a regression gets picked up/dealt with in first place right? It's all kind of dependent on the dev doing the right thing/people noticing as with regressions.
I guess perhaps some subsystems are better at picking things up than others?
@kernellogger yeah this is a pity. We maybe need a new tag that gets enforced by reviewers/bots/maintainers shouted at by Linus if not adhered to like like Fix-Regression: ...
I guess in mm we tend to generally do (fairly) well with these
new tag: maybe; we would not need one if we'd agree on "in case the culprit was merged during the past 12 months, mainline developers are obliged to use 'CC: <stable@...' (and help with backporting the fix to the latest stable kernel, in cases where that's needed)"
But -mm has another problem: from my point of view akpm often leaves regression fixes lingering in -next (aka his -unstable branches) for too long before he mainlines them; and he also often sends fixes to Linus after the weekend (at least last cycle), which is bad if the fixes need to go to stable quickly :-/
@ljs@kernellogger That’s just going to be more noise - the issue isn’t that we don’t have tags, multiple levels of Fixes tags seems like more work on the reviewer side and I’m not sure what problem it solves.
Personally I stopped bothering assessing and tagging things for stable because the stable bot is pulling in so much more stuff than I would consider tagging that it feels like it’s not worth the effort to spend time thinking about it.
I can totally understand that, but the problem with that approach is that important fixes then might fall through the cracks, if the bots miss them or the patch fails to apply
automation needs hints to ensure things we consider important (at least unless AI become better); a CC: <stable... is one; a CC: <regressions... could work in this case, too
@kernellogger@ljs If it’s that important someone (perhaps even me) is probably paying attention anyway, or will at some point. That’s a small proportion of what goes back to stable though - even for stuff I’d manually select there’s things where I’d question if anyone would notice.
2/ …stable team catches many of these regression fixes and backports them (often quickly); I keep an eye on those as well; but that's imperfect and leads to unnecessary delays.
@kernellogger And then there are cases where it’s vice versa: a fix introduced in a -rc kernel happens to be a regression itself. But, as it is supposedly fixing a bug, it gets backported to all the stable trees it applies to and thereby causes a much bigger damage. Now multiple kernels are broken, not only one.
It got fixed in time for the v6.4 release and backports are already queued up for the next round of stable kernels — except for v6.2, which is EOL by now but still got the bad backport :/
yeah, I'm aware of it. 🥴 But well, these things will always happen with backporting. Better tagging and things like that might help. But what bugs me most in your scenario:
Many mainline developers are not aware that they need to fix problems in mainline quickly if the culprit was backported, as Greg normally won't revert the culprit in stable when it's causing the same problem in mainline.
At least it soon will be (hidden :-/ ) in the docs:
@kernellogger Dunno how tagging could help here. But IMHO the real culprit is that commits, especially from the early rc phase, get backported too eagerly. These commits never made it to a release, thereby have very little real world test coverage. By pushing such commits down the LTS and stable branches, stability of these kernels weakens, causing more harm than good to end users.
It happened quite a few times in the past — even seems to happen more often in the recent past. So the advice for end users relying on a stable kernel (“stable” as in its intended meaning, not just the naming) should be to run a stable/LTS kernel that’s at least two weeks old, never the latest, as the latter may carry the gift of yet another regression some poor sucker still has to run into and report it so it, hopefully, gets fixed in time for the two week safety gap.
I don't think the "never made it to a release" aspect is that much important, but yes, I agree, some or a lot of commits are "backported too eagerly". Better tagging IMHO could solve a small part of the problem, as that makes it easy to differentiate changes that are likely worth a quick backport from those that can wait a bit longer.
But it's complicated and there are others things that would need to change, too. #sigh
@kernellogger well, such a tag already exists. It’s “Cc: stable@…” — or the lack thereof.
The original commit doesn’t have it, just a “Fixes:” tag. So, I guess, this is just another example of AUTOSEL picking up “random” patches that where never meant to be backported? 🤔
Add comment