Yesterday night :pixelfed:.de had a few outtages. Why? Weeeell…
It starts with buypass having had an oopsie in their ACME service, which means they will revoke all certificates generated before 2023-12-22 in a few days. I wanted to force-renew my certs before.
The way to do it on #nixos is quite simple and well documented, buuut didn't work for me. The buypass API gave nothing but 500s, so that left me with self-signed certificates.
I decided to switch to LE to fix it quickly, but the host being nixos, I can't just run lego with the let's encrypt API endpoint, but have to change it in my config and rebuild the host instead. No big deal, except … I updated my system flake to 23.11 a while ago, but didn't apply that to abnoba yet, the host running pixelfed.de. The small config change turned into a release upgrade instead.
My flake was tested on other hosts enough, so the upgrade went smoothly, well, except for the last except. I run pixelfed in rootless podman containers from an unprivileged user, which means that user needs systemds linger enabled, or else running (podman) processes will get killed after logout.
Logout happens (for me) 10 minutes after I close the shell, as my SSH sessions persist for that time.
I do enable linger on every boot, but somehow the (accidental) upgrade seems to have killed that setting.
Well, it took me three rounds of wondering why the containers died to remember linger, re-enable it, and finally go to bed 😄
