My Nixos experiment was going so well… genuinely I am a bit miffed by this!
Alas, I am having repeated hard lock ups with large zfs replications.
Things I have tried:
memtest for 5hours. ✅
s-tui cpu stress test for 3hrs. ✅
booted in a live Ubuntu iso and performed the same zfs replication for 2hrs. No lock ups.
Because the machine just hard locks I get no stacktrace or kernel panic to help me here. No logs I can find or anything.
So it’s got to be one of the following.
A hardware issue. I’m not ruling this out because even though the majority of the system was untouched there is a new HBA (LSI 16i) in there. And a couple of new SSDs.
A software issue. Is there something in the Nixos implementation of zfs that’s causing this? Seems so unlikely but I think I owe it to try a different OS today for testing.
@vt52 journalctl showed absolutely nothing. It must have an absolutely instant lockup somehow.
I didn’t try the kernel thing. Good point.
But I’ve reverted to ProxMox now. This is the “Morpheus” old Intel server that’s been running for 3+ years. Never had any lock ups like this before. Im stopping short of blaming nix fully quite yet, but it’s looking likely unfortunately.
@ironicbadger@jinxd When I worked on cars for a living (VW/Porsche/Audi in fact) it was "loose nut behind the steering wheel." 🤣
Did you do any disk stress testing aside from heavy ZFS loads? I'd look at that to differentiate between S/W and H/W. I had some issues with ZFS 2.1.11 but did not result in hard lockup. I'm on 2.3.3 on Debian for about a month and just went through hell fixing an unbootable system. If you use bpool/rpool split DO NOT SNAPSHOT BPOOL.
@HankB@jinxd i leave for SCaLE in 2 days and would prefer to leave my wife and 3 year old with a stable server! so in the interest of time, i’m installing proxmox right now and will test the replication all day today.
fingers, toes, eyes, legs all crossed it's not a hardware issue. at least software is easily fixed.
@jinxd@HankB it’s my old Intel “Morpheus” media server. The only hardware change from the system running reliably the last 3+ years is a new HBA.
So it’s either that or Nixos. But so far the only situation that’s been able to repro is a Nixos install so that’s where I’m focused RN
@ironicbadger not necessarily. I have had hard restarts where it was so severe for the kernel it just restarted without being able to drop down to the code to print the stack when I was testing Core Scheduling when it was out of tree.
If this hardware works on other distributions then it signals to me it is a software issue.
Add comment