I've been using #nixos on a #RaspberryPi to run some services for a year or two now, because I really wanted declarative configuration and to lose the stress of upgrades/etc., but it turns out Nixos SUCKS on the Pi (don't @ me, you know it's true or you haven't tried). I bought a laptop-in-a-mini-form-factor-desktop x86 machine to replace it, but ... I'm exhausted just thinking about it. Installing w/ ZFS was not painless. I want private services, but I don't want to sysadmin them.
@elb#raspbian (now Raspberry os) is fine, you don't have to manage most RPI specifics. You just run your k3s stack on it, leaving you the possibility to install native software when some are not correctly dockerized.
@elb just got my hands on a #RaspberryPi. I’m either going to put #k3s on there, or add it as an arm node to my #rke2 cluster. What do people recommend for running as the base os? I’ve read #nixos isn’t so hot on a pi.
@bashfulrobot I've had poor luck with it, but @ivan says it's working great. My install is very old and has difficulty updating to a newer NixOS, it may be that a fresh install would be fine.
I have historically had the best luck just sticking with Raspberry Pi OS (formerly raspbian), although it has its own special "features".
@elb half of the pain is in the Pi because it is weird, the other half is on the architecture because your laptop can probably not run it. Then just evaluate on a machine with more RAM and copy the result over. Old boards with arm 32bit are also annoying because you can't remote build on servers.
Also zfs is painless on NixOS. On some other distro you need to compile the kernel module for the live iso.
@sandro I don't understand the laptop comment. The small form factor x86 machine, which is a laptop CPU & chipset, is fine. The Pi is pain and not a laptop.
ZFS was not at ALL painless on the Pi. Root on ZFS was painful on the new machine. (For some reason it was INSISTING on installing i386-pc grub, which won't work. A regular install didn't do that.) I was able to get ZFS working as not-root on the new machine with little trouble. This pain is a fact, and not your opinion.
@jakehamilton Mostly that there's a bunch of badly documented stuff on the Pi, and so the stock kernel in nixpkgs frequently doesn't work well (or at all) on the Pi. (NB Pi 4 CM here, this may matter.) I would say that MOST updates that I've tried have failed to boot, by a wide margin. I've had to pick and choose very carefully.
@ivan This is on an SSD, and it's crap. You're just not using it hard enough.
I have been unable to update except in fits and starts from the very beginning. At many points in time, either the kernel, the ZFS modules, the combination of those two, the device overlay files, the boot loader, or some other friggin' thing has been broken, and I have to VERY carefully craft specific git revisions of nixpkgs that will boot and run correctly.
Needing ZFS and the serial console makes it harder.
@elb maybe I’m not, what constitutes “using it hard enough”?
I do use ZFS on mine (and LUKS), though I generally use it headless and don’t use the serial console. The only time it’s gotten broken for me was updating to the 6.1 kernel which has a broken display panel driver
@ivan Interesting. I had to fall back to / on the emmc, with /nix and then, by proxy, almost everything else on ZFS, and I still can't update most of the time, because of broken Pi-compatible versus ZFS-compatible kernel images.
I'm under the impression that the CM4 might also be somewhat less well-supported than the Pi 4, although I haven't dug into why.
I long to move to a better-supported platform, is the bottom line.
@elb this doesn’t match my experience as I’m running a CM4 here.
Which kernel are you using btw? I’ll using pkgs.linuxKernel.packages.linux_rpi4 which is a conservatively updated version of the raspberry pi linux kernel (currently 6.1, was 5.15 until recently)
If you’re trying to use a newer than LTS kernel I’d recommend using config.boot.zfs.package.latestCompatibleLinuxPackages which avoids accidentally using a kernel which wasn’t tested with ZFS yet
@ivan I've run many. I am no longer running the rpi4 kernel, I forget exactly why, but it was either because it had a long dry spell where it was unsupported by any zfs package, or because it quit booting.
Out of curiosity, how long have you been using Nix on a CM4? Maybe if I can manage to get updated to some new enough revision, some of my pains will have gone away. I started with 21.05 or so, which worked just fine, but then at some point before 21.11 things got bad.
@ivan I'm not entirely sure what I'm currently running, I'm in a holding pattern at the moment, because any time I touch anything, everything breaks. My goal is to migrate off the Pi.
Maybe there's an intermediate solution, but I only have so much time for fighting stuff that should Just Work.
@elb I’ve had it for a few months now, I put NixOS on it immediately.
In case you decide to tinker with it more before migrating off of it, feel free to look at my dotfiles on GitHub or my other write up on my installation process
Making a drastic forward update is hard because a lot of the services I'm running require staged updates from revision to revision; this is part of what has me stuck. If I can identify a path for those various services, maybe if I can get updated forward far enough, the suck will go away.
@ivan Part of my problem is that I am loathe to have down time. I bought a replacement CM4 and I/O board to do some testing of upgrade paths, but somehow killed the CM4 module (or at least it's dead; I don't know what happened to it), so I'm back to futzing with my prod system. Which I do not enjoy.
Add comment