Am on LMDE6 with an ancient Nvidia card. Because I've had to resort to using the Nvidia OEM driver installer (which can be a pain to use), installed Xorg updates lurk quietly until a full reboot at which point they generally cause offloading of GPU tasks to the CPU instead because it hasn't figured things out properly.
Timeshift has been useful at least twice in getting me back to a less stressed system.
I think I have a procedure figured out now though (documented here for posterity even if it helps no-one today):
Make a Timeshift snapshot just in case
Install the pending Xorg update
Reboot so it's fully active
Check to see if GPU tasks are being offloaded to the CPU by doing something graphics intensive and noting temperatures or usage%. If not, a miracle has occurred and continuing isn't needed.
sudo remove the execute permission on /usr/bin/Xorg so that it can't immediately be restarted by subsystems designed to protect the average Mint user from command lines and consoles.
Kill Xorg
Log in through a console, via Ctrl+Alt+F1 or similar if not dumped to one by killing Xorg.
Re-install the Nvidia OEM driver
sudo put the aforementioned execute permission back on
Repeat steps 2 and 3 and hope that this time the GPU is doing the work.
Reboots ought to be replaceable by running specific commands, but I haven't gone deep enough into things to know the right things to do there. Reboots are quick and easy enough.
Obvious intermediate steps include not doing anything else important during this and saving important work before starting.
e.g. did you know it's possible to bookmark all open tabs? Well worth looking into.
The nvidia 545 drivers are an absolute dumpster fire. Even for beta drivers they are easily the worst drivers I’ve ever used. They claim to fix the vrr gsync bug tho… so as soon as they fix gestures broadly everything else, maybe they’ll be good
I will never install a Linux desktop without a snapshotting root filesystem ever again. Nvidia driver updates, /boot getting too full during kernel or driver updates, a bad update of pipewire half a year ago, and more I can’t remember. Was always able to boot to previous snapshot of the OS, and address whatever it was. Some ZFS here, some BTRFS there… and my small fleet of Linux desktops are as easy to recover as any immutable OS. Better even, because snapshots allow me to pull individual items or things between states easily, too.
Definitely. I use Timeshift on Linux Mint Debian Edition and set it to take weekly snapshots. Saved my bacon about 2 weeks ago when a kernel update borked my system.
Why does Timeshift only support btrfs? Is it just a lack of developers? LVM supports snapshots too, even if you’re just using ext4. ZFS supports snapshots too.
Timeshift. It has an easy to understand GUI that doesn’t really need much of an intro: You create snapshots of your system files and configs that can be restored if/when you bungle it up.
Timeshift works best if you use BTRFS for your root partition because snapshots can be taken instantly. I have mine setup to automatically take a snapshot every day.
There are many approaches, depending on what broke. In my case the system was fine, just xorg being completely borked. So I logged into the console and fixed it.
If regular console doesn’t work, something really went bad during boot, for which there’s single-user mode which is kind of similar to safe mode from Windows 98 (I’m sure there’s something similar in newer windows versions).
And of that doesn’t work, there’s the minimalistic rescue shell.
And if that doesn’t work, you can boot from a USB or some other external media and try to fix your system from that, maybe even using chroot to use the system somewhat normally.
Neat little guide (Arch, systemd) to set up automatic snapshots when you run a pacman/yay update and the option to chose snapshots in grub during startup. Really useful.
I started with linux begin 2019. I just use xorg so I don’t know about wayland problems. I think a long time ago nvidia broke once and I switched to nvidia-dkms and it has worked fine since until recently where a mesa update broke xorg but I don’t think that has to do with nvidia. Getting CUDA to work might be trouble though (I think I briefly tried once).
Another problem is that most manufacturers of laptops directed at the Linux crowd, for some reason I will never understand, insist on punching Nvidia hardware instead of AMD/ATI. How does that help?
Unfortunately for those of us that use Cuda features, AMD just really isn’t that viable of an alternative. Anyone who’s had to deal with ROCM can attest to this…
I’m on 545 and I have no issues. But I’m also not using Ubuntu…
Maybe it’s the distro that’s the problem not the backup. I mean I’d rather have a distro with smooth updates than one that makes me need snapshots.
What’s even the point with such a distro, ok so I restore previous working state, then what, I can’t do updates anymore? Living in fear of official updates sounds terrible.
So you’re implying there exists a distro that is perfect and never breaks anything? Sounds like denial. Having time shift in place is risk management and says nothing about the distro, which btw all are imperfect and may break eventually. Kinda confused how one can run Linux and be unaware of how complex and fallible ALL software is.
No, I’m implying that official updates breaking the system is insane and should not be accepted as the norm to the point you casually need to use snapshots just to keep your system working.
That’s not an accurate portrayal of anything though. You’re implying risk management means you’re accepting that the system sucks and the only way to keep it running is to have a backup system.
That’s not what anyone is doing. They are acknowledging the flawed nature of software and humanity in general and guarding against the consequences of this. It’s being smart. Name one distro that has never broken with updates? You cannot because there obviously couldn’t exist one.
A major distro breaking your system is the equivalent of a flower pot falling on your head walking down the street. Does it happen? Sure. Do I want to spend my life wearing a motorcycle helmet and looking up all the time? No.
I’m not saying distros can’t crap on you, I’m saying stop tolerating it. Raise a stink or switch distro. There are distros out there where you don’t have to live in constant fear and where nothing happens if you don’t have snapshots.
I do have backups, precisely because shit happens. But there’s a difference between a helmet and health insurance.
Okay what distros are we talking about? What’s an example of an unreliable one and what’s a reliable one?
I think it’s much more complicated than this. Honestly there’s a reason it took decades for Linux distros in general to get as stable as they are today. It’s really really hard to build an operating system.
I think a better analogy for installing updates on linux would be riding a motorcycle. Accidents happen all the time. They’re bound to, be prepared. Just because you can ride for 30 years unscathed doesn’t mean you can take that for granted.
My 2 cents, OpenSUSE Leap is stable as hell. lots of QA happening with their automated testing, and keeping in lockstep with SUSE releases (now sharing same binaries). Every distro upgrade has gone flawlessly, but when I have had the urge to tinker and do stupid things inf config files the built in btrfs snapshots have been a godsend.
What you wrote and what I’m reading about openSuse leap sound excellent. My only concern is support. Just about all the apps and tools I use are well supported on Ubuntu based systems but I don’t recall seeing nearly as much support for rpm based oses, including open suse . What’s your experience? If you go to install just any random software you just heard about, how well does it typically go getting it installed/working?
I haven’t had issues finding packages, often they may not be on the dev websites that host a deb package, but the main repos contain the general tools, if you need something more “fringe”,independent dev or new, then software.opensuse.Org allows you to turn on a search by community or experimental packages (which would be like Arch AUR and contains a lot of rpms) so you can install directly from the website, it will add the neccessary community repos during the install. Or if you don’t want to pollute your repo list they typically they have the option : Grab Binary directly. Or ther is an OPI ? Package you install that lets you search locally for community packages. For commerical apps like teamviewer , yubikey,webex etc the rpms were all available to support corporate SUSE users. If you still can’t find an rpm, then you run the Alien tool which converts a deb to the RPM installation format. The only issue I had once was the community package had dependencies that were not contained in the users repo so I had to find the dependencies and install those first. That felt like 1990s
No problem. Happy New Year. Also it is .org i mistakenly put .com for the opensuse software site. ( I will edit post) Also I should mention if you dont see an officially supported rpm on the software site, it does not mean it isn’t there, it now seems to mean it is in the main repos. I think years back they would all show, but maybe due to shared binaries with SUSE they don’t bother listing on the software.opensuse.org site. So best to check locally in the YAST or Zypper search of your repos first.
.deb distros are doomed from the start if you need to use third-party repos (which you do, for a desktop system) because they always end up undermining the stability of the packages from the core repos in the long run.
Try an Arch-based distro, they’re super stable because their compatibility model is more robust, and there are options depending on how much hand-holding you want — ranging from vanilla Arch to Endeavour to Manjaro.
Ya know, this is super interesting you mention Arch. The only person I’ve known IRL who uses and loves Arch champions it hardcore but with the caveat that you have to be okay with things breaking due to the rolling release model. Due to his guidance I have avoided arch specifically. I’ve been running Ubuntu based distros a couple of years and only had issues with updates breaking things like 2 times… Both of which didn’t require a wipe or anything.
Packages can break, not the distro. Packages can break at any time because there’s thousands of them and nobody can check all of them thoroughly. A rolling distro gets you both the bugs and the fixes faster.
Non-apt and non-rolling will limit your options considerably.
I might be confused. I thought that the distro itself was made up of packages and that’s what all updates did: update various packages bundled with the distro (plus any you installed yourself)
I think it’s just dumb to not make a backup before large updates. There’s so many things happening, a lot can go wrong, especially if you have added 3rd party repos and have customized core parts of the system, not just through config files but let’s say you switched to latest kde plasma from the one your distro ships.
And what happens if you have to restore the backup?
You can look up what’s the solution to your problem in peace while everything is still working. If it was a server, all the services are still available, if it was your desktop you don’t have to use a live linux usb that’s without all your configs to find the solution
You make a good point. Ubuntu gives you so many ways to shoot yourself in the foot that it’s pretty much a given that it will get messed up eventually. So you have to use snapshots.
On Arch based distros the updates just work. I’ve never had to snapshot anything. But having just one single community repo (AUR) contributes to that a great deal.
Of course it can. And your PC can also fall off the desk. I’m saying a snapshot tool is a really poor solution for distro problems, it’s really a bandaid for a problem that shouldn’t exist.
Use a decent distro, take proper backups, and use snapshots for what they were intended — recovering small mistakes with personal files, not for system maintenance.
That’s the point – your claim about deb-based distros is just anecdotal.
The example here is Nvidia updates borking the system. I’ve have that happen to me numerous times on Arch-based systems.
I’ve run deb-based distros on some boxes over years of updates with no issues. On the other hand I’ve had updates cause breakages on Arch-based systems pretty much every time I’ve run them.
Which is to say anecdotes are useless, updates can break systems, and being able to immediately roll back to a working system and deal with updating later is a simple, nice thing to have with no downsides.
Is it dumb that I only backup my docs and anything else I think is important? I can rebuild fairly quickly if something would happen. I ask since I know that people backup a variety of their directories
Timeshift with BTRFS kicks ass. I have mine set for daily snapshots, retained for a week. Only the changes between snapshots are stored, so the extra disk usage is minimal, and easily justified by the peace of mind in case of fuck-ups or broken updates.
Only the changes between snapshots are stored, so the extra disk usage is minimal
If you want to use a similar approach for backups, Borgbackup is a pretty nice piece of software. I have two backups of my most important files: One on my NAS at home, and one “in the cloud” on a storage VPS (ends up way cheaper than using S3, B2 or anything like that).
I’ve got one with HostHatch that’s 10TB of space for $10/month. It was an offer they had during Black Friday 2020. They had a similar offer during Black Friday 2023 but I think it was around $20/month, paid yearly.
I live in the San Francisco Bay Area and my storage server is in Los Angeles, which is around 10ms round-trip ping time from my home internet connection.
Hetzner is good too. They have relatively cheap “storage boxes” that are a shared environment rather than a VPS. You don’t get proper SSH access, but they do support FTPS, SFTP, Samba, Borgbackup, Restic, rclone, rsync and WebDAV. www.hetzner.com/storage/storage-box
Borgbackup encrypts the backups, so the host won’t be able to actually view your backups.
Add comment