How to make setup more resilient? Proxmox mini-PC \w iSCSI to TrueNAS

unsaid0415, 8 months ago (edited 8 months ago)

Hey, OP here again.

Here’s what I ended up with:

• upgrading my TrueNAS CORE to TrueNAS SCALE - it was really easy, just upload a 1.3GB upload file through the web UI. CORE’s apps/plugins are based on BSD jails, where SCALE apps are based on Kubernetes/Docker, so I can any arbitrary Docker container from Dockerhub as I please, rather than being limited to BSD jails
• migrating all the VMs/LXCs to matching TrueNAS SCALE Applications. So e.g. my hand-made Navidrome LXC was migrated to the TrueNAS SCALE Application. Sometimes there was no equivalent TrueNAS app for what I was using - e.g. Forgejo, so I just ran an arbitrary container from dockerhub.
• decomissioning the Proxmox mini-pc (Lenovo M920q). I’ll sell it later or maybe turn it into a pfSense router.

I installed a custom TrueNAS app repository called Truecharts. It has some apps that the default repo doesn’t have, and it also has a nice integration with Ingress (Traefik), which allows you to easily create a reverse proxy using just the GUI.

I’m still yet to figure out how to set up Let’s Encrypt for the services I made available to the Internet. I can no longer do things the Linux way, i must do it the Kubernetes way, so I’m kind of limited. Looks like HTTP01 challenges don’t work yet and I’ll have to use DNS01.

Looking back, I’m happy I consolidated. The hypervisor was idling all the time - so what’s the point of having a second machine? Also, the only centralized machine has IPMI, so I have full remote control, and I’ll hopefully never have to plug a VGA cable again. Of course, there’s no iSCSI fault path anymore, though I’m happy I got to experiment with it.

The downside is as I said - I’m forced to do things the Kubernetes/Docker way, because that’s what TrueNAS uses and that’s the abstraction layer I’m working on. Docker containers are meant for running things, not for portability. I’m sad that I can’t just pack things up in a nice LXC and drag it around wherever I please. Still, I don’t thing I’ll be switching from TrueNAS, so perhaps portability isn’t that big of a deal.

I’m also sad that I … no longer have a hypervisor. Sure, SCALE can do VMs, but perhaps keeping TrueNAS virtualized would give me the best of both worlds.

TechAdmin, 9 months ago

VMs in ESXi have the same behavior when iSCSI connection is lost then restored later. Windows with iSCSI drive mounts shows the same behavior in that scenario too.

UPS would be a great addition no matter what option you choose.

ssdfsdf3488sd, 9 months ago (edited 9 months ago)

Get rid of iscsi. Instead, use truenas scale for nas and use a zvol on truenas to run a vm of proxmox backup server. Run proxmox on the other box with local vms and just backup the vms to proxmox backup server at a rate you are comfortable with (i.e. once a night). Map nfs shares from truenas to any docker containers directly that you are running on your vms. map cifs shares to any windows vms, map nfs shares directly to any linux things. This is way more resilient, gets local nvme speeds for the vms and still keepa the bulk of your files on the nas, while also not abusing your 1gbit ethernet for vm stuff, just for file transfer (the vm stuff happens at multi GB speeds on the local nvme on the proxmox server).

MigratingtoLemmy, 9 months ago

For a home environment, this is the correct idea

ssdfsdf3488sd, 9 months ago

I would argue it’s the correct idea up to a fairly decently sized business. Basically anything where you don’t have the budget or the need for super fault tolerant systems (i.e. where it’s ok to very rarely have a 20 minute to an hour outage in order to save 50k+ of IT hardware costs). You can take the above and go next step to a high availability proxmox cluster to further reduce potential downtime before you step into the realm of needing vmware and very expensive highly available and fast storage as well. It gets even more true when you start messing around with truenas and differential speed vdevs (i.e build a super fast nvme one with 10-25gig networking for some applications, a cheaper spinning rust one with maybe 10 gig networking for bulk storage. It’s also nice that, by using proxmox backup server as a zvol you can take advantage of all the benefit of both zfs replication/snapshotting and cloud (jstor/wasabi s3 bucket, another truenas server at a different location) for that zvol as well as your other data you are sharing as datasets.

MigratingtoLemmy, 9 months ago

I would have assumed that most businesses invested in resilient hardware, but perhaps not. Thanks for the note

melmi, 9 months ago (edited 9 months ago)

What is TrueNAS adding to this arrangement? Generally when people run two different servers at home, they keep the VM drives on the hypervisor and just use the NAS for storing bigger things like media files. Hosting VM drives over iSCSI works in an enterprise environment, but if you can’t guarantee uptime for your storage solution then all you’re doing is adding failure modes.

It seems to me that your best bet is to go down to one server, which means cutting out either TrueNAS or Proxmox. Both can handle both storage (ZFS included!) and VMs, so ultimately it’s a matter of which you like better.

Alternatively, if you’re hosting other stuff on your NAS, you could consider keeping both servers but just getting a few SSDs to stick in your Proxmox mini PC to serve VMs. That may or may not be viable for your situation, but it’s worth considering.

unsaid0415, 9 months ago

I edited my post to clarify that TrueNAS keeps more than just VMs. It has photos, documents etc. as well.

Generally when people run two different servers at home, they keep the VM drives on the hypervisor and just use the NAS for storing bigger things like media files

This is simple and makes sense as well. My TrueNAS is only 2 HDDs, which is not ideal for VMs. I could get a larger drive SSD/M.2 drive for the hypervisor, though the Lenovo M920q supports 1xM.2 and 1x2.5" drive.

Hosting VM drives over iSCSI works in an enterprise environment, but if you can’t guarantee uptime for your storage solution then all you’re doing is adding failure modes.

Well, my whole setup comes from the fact that I wanted to cosplay as an enterprise environment (famous last words for a homelabber). I’ve been powering the TrueNAS up and down a lot due to some electricity-related construction in my apartament, and it brought out this flaw in my setup. I guess an UPS would be in order, as another poster pointed out.

SeeJayEmm, 9 months ago

Well, my whole setup comes from the fact that I wanted to cosplay as an enterprise environment

I feel that. Experimenting I get, but I’d never have it be my primary vm backing storage. Esp. not on a 1Gb network, no ups, no redundancy.

Someone else suggested local vm storage and a PBS VM on the TrueNas box. I think that’s a solid solution to consider.

PuppyOSAndCoffee, 9 months ago

In general if you lose your iscsi storage you are hosed.

The way around this is replication where you write one byte to two locations and pseudo load balancing where you have an active and inactive link. When power on one storage fabric goes down you flip to the other. Iscsi isn’t really good for this use case

SheeEttin, 9 months ago

2 is the only possible one of those options.

You could also make the current setup more reliable by adding a UPS and/or second storage node for redundancy, so that when one goes down, the other is still available. Presumably TrueNAS supports this.

But nothing is going to help you recover if the iSCSI link is broken. It’s up to the host and guest OS to re-establish the link, and to the guest it usually looks like the hard drive has been unplugged, and I don’t know any OS that considers that a supported and recoverable condition.

unsaid0415, 9 months ago

Thanks for making it clear that iSCSI power down is in fact one of the more grim scenarios, I couldn’t make it out how bad of a situation it is. In an enterprise environment a SAN being down would require some type of incident report.

UPS - as you suggested - would solve most of my problems to be honest.

unsaid0415, 9 months ago

It’d be nice to have Proxmox and TrueNAS side by side on one machine, but since TrueNAS forums are against the virtualization of TrueNAS (yes I know people do that, but I’m not willing) I’m somewhat stuck with having to have one bare metal machine per appliance.

Kaavi, 9 months ago

I took my truenas some months ago and installed proxmox on that server instead.

I can’t from your post figure out what you need truenas for? Proxmox runs zfs, etc?

I made an lxc container with samba to share the discs in proxmox.

unsaid0415, 9 months ago

I edited my post to clarify. TrueNAS also keeps documents, photos, torrents, music. I also use the mount feature so that the music server LXC can access music

Decipher0771, 9 months ago

I’m sure you’ve heard plenty through the forums, but Truenas virtualized is perfectly fine so long as you’re passing through an HBA directly. It doesn’t affect reliability any, but it doesn’t add any features either.

“Can I virtualized Truenas” is probably the second most popular question after “do I really need ECC ram”