Jordan_U, 2 months ago

More than a decade ago a user came into #ubuntu-server on Freenode (now libera.chat ) and said that they had accidentally run “rm -rf /* something*” in a root shell.

Note the errant space that made that a fatal mistake. I don’t remember how far it actually got in deleting files, but all of /bin/ /sbin/ and /usr/ were gone.

He had 1 active ssh connection, and couldn’t start another one.

It was a server that was “in production”, was thousands of miles away from him, and which had no possibility for IPMI / remote hands.

Everyone (but me) in the channel said that he was just SoL and should just give up.

I stayed up most of the night helping him. I like challenges and I like helping people.

This was in the sysv-init (maybe upstart) days, and so a decent number of shell scripts were running, and using basic *nix commands.

We recovered the bash binary by running something along the lines of

<span style="color:#323232;">bash_binary_contents="$( </proc/self/exe)"
</span><span style="color:#323232;">printf "%s" > /tmp/bash
</span>

(If you can access “lsof” then “sudo lsof | grep deleted” will show you any files that are open, but also “deleted”. You may be surprised at how many there are!)

But bash needed too many shared libraries to make that practical.

Somehow we were able to recover curl and chmod, after which I had him download busybox-static. From there we downloaded an Ubuntu LiveCD iso, loop mounted it, loop mounted the squashfs image inside the iso, and copied all of /bin/ , /sbin/ , /etc , and so on from there onto his root FS.

Then we re-installed missing packages, fixed up /etc/ (a lot of important daemons, including the one that was production critical, kept their configuration files open, and so we were able to use lsof to find the magic symlinks to them in /proc/$pid/fd/ and just cp them back into /etc/.

We were able to restart openssh-server, log in again, and I don’t remember if we were brave enough to test rebooting.

But we fucking did it!

I am certainly getting a lot of details wrong from memory. It’s all somewhere at irclogs.ubuntu.com though. My nick was / is Jordan_U.

I tried to find it once, and failed.