kernellogger,
@kernellogger@fosstodon.org avatar

A question for experts on #git bisecting the #Linux #Kernel:

Assume someone runs into a regression when updating from 6.1.90[1] to 6.6.30 that needs bisecting. What do you suggest:

  • Check manually which mainline release (e.g. 6.2, 6.3, ...) introduced the problem and afterwards bisect between that and the previous release.

  • Bisect straight between 6.1 and 6.6.30.

1/ I guess I would definitely go for…

[1] let's assume that 6.1 was fine for this scenario to keep things simpler

SchwarzeLocke,
@SchwarzeLocke@ohai.social avatar

@kernellogger I would always avoid bisecting between two different stable series. The linear history of stable is very different from mainline, which also has a lot more commits. Effectively git bisect would spend most of the time outside of the linear commits of the stable series.

First, I would go down to see if the regression came in via mainline or via stable, by testing 6.1.0 and 6.6.0, and then bisect between those. As optimization, I would do the first steps manually, i.e. 6.4.0.

kernellogger,
@kernellogger@fosstodon.org avatar

@SchwarzeLocke

and, yeah, doing the first steps manually might be a good compromise

SchwarzeLocke,
@SchwarzeLocke@ohai.social avatar

@kernellogger If you want to describe that in the documentation:

Once you provided both a good and a bad version for git bisect, it will checkout automatically a commit in between. You don't have to test this commit: you may want to manually checkout a closer release tag, to avoid running into unrelated problems which got resolved after the merge window.

kernellogger,
@kernellogger@fosstodon.org avatar

@SchwarzeLocke

Hmmm. Wondering if that is worth it, but yeah, it likely is. Will keep that in mind for the next time I work on that document.

kernellogger,
@kernellogger@fosstodon.org avatar

@SchwarzeLocke

Well, sure, that's why I wrote " let's assume that 6.1 was fine for this scenario to keep things simpler"; apparently that was not clear enough, sorry.

SchwarzeLocke,
@SchwarzeLocke@ohai.social avatar

@kernellogger ah, I thought you meant "anywhere from 6.1.x" with "6.1", but meant what I mean with "6.1.0": what was released by Linus, not anything from stable.

Yeah in that case both of your original points work; from a git perspective (tooling) not really a difference.

I would still do the releases manually, to avoid commits from the merge window / early RC, as these might have other/unrelated issues, which got fixed in a later RC.

kernellogger,
@kernellogger@fosstodon.org avatar

@SchwarzeLocke

Sometimes I think somebody should submit a patch to remove the trainling .0 from the versioning when it's ".0", as kernel.org and Linus also just call mainline releases "6.9". but whatever, that is just a detail -- and fixing it would likely break something.

gromit,
@gromit@chaos.social avatar

@kernellogger I think I would start out by checking out which of the (packaged) stable kernel versions still suffer from the bug or if it not the "standard" linux package (i.e. linux-zen or linux-hardened) to see if the issue occurs with the tree that is closest to mainline. Then I would verify at the start of the bisection that the issue is not in 6.8 if the 6.8.2 kernel was the latest working kernel.

This is how I did it i.e. for https://lore.kernel.org/all/3iccc6vjl5gminut3lvpl4va2lbnsgku5ei2d7ylftoofy3n2v@gcfdvtsq6dx2/ which I bisected between 6.6 and 6.9rc4

gromit,
@gromit@chaos.social avatar

@kernellogger I like bisecting with the mainline kernel since I can just use packaging tools (a modified version of https://aur.archlinux.org/packages/linux-mainline) to do the bisection.

Edit: This would ofc also work for the stable trees, but I already have a setup for mainline 🤷 😃

kernellogger,
@kernellogger@fosstodon.org avatar

@gromit

out of curiosity: if you do that locally using the arch stuff, does it do a full rebuild for each step, or does building a new step build upon what the previous step left behind (which at the end of the build process makes quite a difference)

gromit,
@gromit@chaos.social avatar

@kernellogger We build our packages in clean environments with only the specified dependencies present ("clean chroots") and this is also what I use here. In the end its still pretty fast because I offload the build to our beefy buildserver with 96 cores :p

There also is a Wiki page on how to do it differently (https://wiki.archlinux.org/title/Bisecting_bugs_with_Git), but I think I will add some more info to it because I think it can use a few improvements.

kernellogger,
@kernellogger@fosstodon.org avatar

@gromit

yeah, rpm start with a clean env as well, but when it comes to kernel bisections in can be a downside if you don't have a beefy buildserver at hand ;-)

If you update that or a related page, you might want to add a link to https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html somewhere if suitable (which does more than a bisection, but that almost always is wise)

kernellogger,
@kernellogger@fosstodon.org avatar

2/ …the first option, if the user has access to pre-compiled kernel packages that make it easy to test those mainline releases (or stable kernels derived from it) .

But what's the better approach if the user has no such packages at hand?

klausman,
@klausman@mas.to avatar

@kernellogger One thing of note regarding precompiled kernel packages is that they rarely if ever are vanilla, so even as a "pre-bisect" step, there be Dragons there.

kernellogger,
@kernellogger@fosstodon.org avatar

@klausman

Yeah, definitely.

Another dragon in that field: .config changes where the packager enabled some new feature (say a new security technique that is known to cause problems for some old apps) that causes a regression.

Is rare, but does happen.

poeschel,

@kernellogger I understand you want to avoid compiling the kernel at a first step, right?
Well, at a certain point you will have to to bisect nevertheless. So you will invest in setting up the whole chain at some point. I would do that at the first step and let git bisect do the rest and be then safe with vanilla kernel and the same compiler etc.
I doubt that you spare much time testing precompiled kernels.

kernellogger,
@kernellogger@fosstodon.org avatar

@poeschel

"I understand you want to avoid compiling the kernel at a first step, right?"

what? no. As you said yourself: the users has to set up everything anyway.

"spare much time testing precompiled kernels": I guess that likely depends a lot on the horsepower of your machine; on my T14s Gen1 AMD I think compiling a localmodconfig kernel takes 10 or 15 minutes; downloading and installing would be quicker, even if I'd have to do it somewhat manually

  • All
  • Subscribed
  • Moderated
  • Favorites
  • linux
  • Durango
  • DreamBathrooms
  • InstantRegret
  • magazineikmin
  • osvaldo12
  • everett
  • Youngstown
  • khanakhh
  • slotface
  • rosin
  • thenastyranch
  • ngwrru68w68
  • kavyap
  • normalnudes
  • megavids
  • ethstaker
  • GTA5RPClips
  • modclub
  • cisconetworking
  • mdbf
  • tacticalgear
  • cubers
  • provamag3
  • tester
  • anitta
  • Leos
  • JUstTest
  • lostlight
  • All magazines