Oh god I remember that. Luckily in 2003 my main computer was scraped together from discarded parts at my father’s day job, so it was ethernet only
In 2024 on a laptop I still have wifi problems though. Most recently, if I closed and opened the laptop lid (suspend + resume), the wifi hardware just disappeared off the face of the kernel.
Turns out that the iwlwifi kernel module just irreversibly crashes when the laptop suspends and can only be fixed with a reboot.
So I had the fun task learning about systemd pre-suspend hooks to unload the driver before suspend and load it again on resume.
I’ve found that the silliest desktop problems are usually the hardest to solve, and the “serious” linux system errors are the easiest.
System doesn’t boot? Look at error message, boot from a rescue disk, mount root filesystem and fix what you did wrong.
Wrong mouse cursor theme in some Plasma applications, ignoring your settings? Some weird font rendering issue? Bang your head against a wall exploring various dotfiles and rc files in your home directory for two weeks, and eventually give up and nuke your profile and reconfigure your whole desktop from scratch.
Yeah for some reason a single game ignores the system sound settings and goes straight to a line out. My system doesn’t see that the game is outputting sound and I can’t change it. (Arch with KDE)
Somewhat related on windows 11, for some reason teams volume will desync from system volume. I’ll put system volume to 0 and still be hearing teams. It’s the same audio device being selected. I don’t understand why it would ever work that way but here we are
Oh my god, you’ve put it into (really nice) words something I’ve felt since quite some time now. I’ve no trouble (in fact even joy) when something major is fucked up. But all this GUI shenanigans, I’ve usually no idea where to even begin. The lack of structure and hierarchy completely flummoxes me. Or maybe I just don’t have enough experience debugging userland stuff
A couple of weeks ago I moved Firefox to one side. Window disappeared, but Firefox was still running “somewhere” on my desktop, but was not actually be rendered to the screen. Killing the process and relaunching just resulted in it be rendered to this weird black hole. Log out of gnome and log back in? Same! Reboot? Same!
Ended up deleting it’s config folder and re-attaching to Firefox sync in order to have it working again. No idea what went wrong, nor will I ever most likely.
There really should be a hotkey for “move window to primary display” or somesuch. The worst is when just the top “cleat” of the window is inaccessible, making it impossible to simply move the window yourself.
Alternately, a CLI tool to just trash a specific app’s window settings, or a system control panel that lets you browse these settings, would be incredible.
In every GUI I’ve used, there are tiling or snapping hotkeys, something like Super + Arrow keys or something, that will usually put the window somewhere sane.
I feel like i had a disappearing window like that a lifetime ago and the fix was to change the resolution. I don’t know if that uncovered the void to the right or forced the window to reassign itself to usable space. But it worked then. Hell, it could have been windows for all I recall.
Fixed a typo in my /etc/fstab that prevented the NAS from mounting. I am a bear of little brain. But I’m also proof that you don’t have to be some master hacker to successfully run Linux.
Saved me from reinstalling. Made me realise that there really should be an alternative to typing into fstab by hand since us humans will make mistake. Either that or make fstab nog crash completly on an error but just skip it.
I have no idea how widespread it is among other distros, but ArchLinux’s bootable install disk/iso comes with a genfstabcommand that snapshots your current mount points and outputs it as a fstab.
You still need to figure out where and how to mount everything yourself, but at least it saves you from most typos that could otherwise end up in the fstab file.
Still trying to use Linux Mint on my 2013ish MacBook Pro as a daily driver. Got the MacBook for free and it wouldn’t update anymore, so installed Linux Mint and it’s been great for the most part. Still trying to access my NAS on it though. Having to manually mount drives is a new experience for me, and it’s not coming to me intuitively. Reached out via IRC, got some help but still working on it.
I have two, one is actually complicated and one was so obtuse that I never would have figured it out in a million years:
Actually complicated: I still don’t know how it happened, but somehow an update on Arch filled the boot partition with junk files, which then caused the kernel update to fail because of no disk space, which then kind of tanked the whole system. It took ages, but with a boot disk and chroot-ing back into the boot partition I eventually managed to untangle it all. I was determined to see it through and not reinstall.
Ridiculous: One day when using Ubuntu, the entire system went upside-down. As in, everything was working perfectly fine, but literally the screen was upside-down. After much Googling I had no luck figuring it out, then I accidentally found the solution - I’d plugged a PS4 controller into the USB on the laptop to charge it, and for some reason Ubuntu interpreted the gyroscope on the controller as “rotate the screen display” so when I moved it, the screen spun round. I only figured it out by accident when I plugged it back it and it spun back to normal lol.
I had a similar one. I had a usb-powered fan cooling pad that my laptop was sitting on. My laptop would randomly go into boot loops when I turn it on. I thought it was a grub issue so I always had my usb stick ready to re-install grub. Did some dusting one day and forgot to plug in the cooling fan, then the boot loop never happened again. Turns out it was the fan plugged into the usb that was causing it.
Semi-related note… displayport cables can cause a no-boot condition too. I think it was the existence of Pin#1. I had to duct tape that one pin and my computer finally booted up.
A couple years ago on Reddit I saw a story where a dude working IT support had to drive to a remote office or replace a workstation that wouldn’t boot. When he got there the lady whose desk it was had some shitty USB fan or maybe an led Christmas tree plugged into one of the USB ports. He unplugged that and the pc booted fine.
Ah I remember that one! Classic. I also remember a story about someone who lost an entire PC in their apartment. It was running and connected to the network, they could ping it, but couldn’t physically find it lol.
Ridiculous: One day when using Ubuntu, the entire system went upside-down. As in, everything was working perfectly fine, but literally the screen was upside-down. After much Googling I had no luck figuring it out, then I accidentally found the solution - I’d plugged a PS4 controller into the USB on the laptop to charge it, and for some reason Ubuntu interpreted the gyroscope on the controller as “rotate the screen display” so when I moved it, the screen spun round. I only figured it out by accident when I plugged it back it and it spun back to normal lol.
Not a Linux problem per se, but I had a 128GB image disk in a unknown .bin format which belongs to a proprietary application. The application only ran on Windows.
I tried a few things but nothing except Windows based programs seemed able to identify the partitions, while I could run it in Wine, it dealt with unimplementend functions. So after a bit of googling and probing the file, it turns out the format had just a 512 bytes as header which some Windows based software ignored. After including the single block offset, all the tools used in Linux started working flawlessly.
This is so arcane to me. Like, I more or less understand your high-level explanation, but then you gloss over “including the block offset” but how would one do that ??
Inspecting the file with a hex editor would give you lots of useful info in this case. If you know approximately what the data should look like, you can just see where the garbage (header) ends and the data starts. I’ve reverse engineered data files from an oscilloscope like this.
Well, in this scenario the image file had 512 bytes sections, each one is called a block. If you have a KiB (a kibibyte = 1024 bytes) it will occupy 2 blocks and so on…
Since this image file had a header with 512 bytes (i.e. a block) I could, in any of the relevant Linux mounting software (e.g. mount, losetup), choose an offset adding to the starting block of a partition. The command would look like this:
<span style="color:#323232;">sudo mount -o loop,offset=$((header+partition)) img_file /mnt
</span>
It was some combination of both, the system would post, past the bootloader, attempt to initialize drivers and other standard starting packages and then immediately panic and drop into an emergency terminal (/TTS), with a failure to mount the root partition, from what I can recall. It tried it a couple times and then there was an error message that said: “Bailing out, you are on you own, good luck”
Mine is close to that. I still had a working libc, but the dynamic library for C++ programs wouldn’t load, so most of the Gentoo tools and several other things I expected simply crashed on startup.
Found enough working programs to get the library restored and remove the bad arch flags from my configuration to start another emerge world.
After that, I was pretty confident that I could run Linux at least as confidently as I had previously run WinNT 4.
Generally if you remove a file, it won’t affect programs that already have it open. So if you delete libc, hope that you don’t lose power. If worse comes to worst, you’ll need to pull the drive and mount it on another machine.
More than a decade ago a user came into #ubuntu-server on Freenode (now libera.chat ) and said that they had accidentally run “rm -rf /* something*” in a root shell.
Note the errant space that made that a fatal mistake. I don’t remember how far it actually got in deleting files, but all of /bin/ /sbin/ and /usr/ were gone.
He had 1 active ssh connection, and couldn’t start another one.
It was a server that was “in production”, was thousands of miles away from him, and which had no possibility for IPMI / remote hands.
Everyone (but me) in the channel said that he was just SoL and should just give up.
I stayed up most of the night helping him. I like challenges and I like helping people.
This was in the sysv-init (maybe upstart) days, and so a decent number of shell scripts were running, and using basic *nix commands.
We recovered the bash binary by running something along the lines of
(If you can access “lsof” then “sudo lsof | grep deleted” will show you any files that are open, but also “deleted”. You may be surprised at how many there are!)
But bash needed too many shared libraries to make that practical.
Somehow we were able to recover curl and chmod, after which I had him download busybox-static. From there we downloaded an Ubuntu LiveCD iso, loop mounted it, loop mounted the squashfs image inside the iso, and copied all of /bin/ , /sbin/ , /etc , and so on from there onto his root FS.
Then we re-installed missing packages, fixed up /etc/ (a lot of important daemons, including the one that was production critical, kept their configuration files open, and so we were able to use lsof to find the magic symlinks to them in /proc/$pid/fd/ and just cp them back into /etc/.
We were able to restart openssh-server, log in again, and I don’t remember if we were brave enough to test rebooting.
But we fucking did it!
I am certainly getting a lot of details wrong from memory. It’s all somewhere at irclogs.ubuntu.com though. My nick was / is Jordan_U.
I just told this story to a friend but I did the standard rm -rf * as root while in the / directory. And this was back in the day where we nfs mounted every other machine and root privileges propagated through NFS. I think it was on the 2nd or 3rd machine when I thought – “this seems to be taking longer than I thought”.
I managed a CentOS system where someone accidentally deleted everything from /usr, so no lib64, and no bin. I didn’t have a way to get proper files at the time, so I hooked the drive up to my Arch system, made sure glibc matched, and copied yum and other tools from Arch.
Booted the system, reinstalled a whole lot of yum packages, and… the thing still worked.
That’s almost equivalent to a reinstall, though. As a broke college student, I had a laptop with a loose drive, that would fall out very easily. I set it up to load a few crucial things into a ramdisk at boot, so that I could browse the web and take notes even if the drive was disconnected, and it would still load images and things. I could pull the cover off and push the drive back in place to save files, but doing that every time I had class got really tiring, so I wanted it to run a little like a live system.
I have taken a drive with filesystem issues, mounted on a different machine and either backup data I wanted to keep or copy files to make the original machine runnable.
Are you including back in the day when we had to use windows device drivers via ndiswrappers?
I’ve managed to remove a critical library once but did manage to extract it from an RPM on another machine and manually install it. That was good enough to get me to the point where I could yum reinstall.
Pre-linux we had an HP workstation where the disc drive died and of course we had no backups. I managed to frankenstein the disc by connecting the platters on the broken disc to the circuit board of a working disc. This worked and I was able to back up the disk and reload on to a new drive.
And then we bought an 8mm tape drive for backups and I had to port some drivers to HP-UX to get it to work. But we had awesome backups after that!
I once removed most of X by trying to remove Gnome dependencies and it lead to an interesting couple of hours but I did have a working system when I was done.
There were countless dependency bugs and broken systems but at least I learned how to use the Gentoo Forum and also a lot of how Linux works.
A Gentoo upgrade package list with over 100 packages and conflicts all over the place. Then do it again when the list grows to the same size in a few months.
Add comment