3 used MSFF PCs (i5, kingston SSDs, 24GB of ram each). All running proxmox, set up as a cluster.
1x Raspberry Pi 4 8GB. Running ubuntu.
1x Vultr 2vCPU/4GB RAM instance.
I've got a small kubernetes cluster set up using Talos with 3 controlplane / 3 workers in VMs on the proxmox nodes. The vultr node is also running Talos and attached to the same cluster. Their KubeSpan feature is pretty neat, automatic full mesh wireguard between all cluster nodes.
Traffic inside the cluster flows seamlessly between all nodes, and I can even use it as sort of a proxy server using Cilium's Egress Gateway function.
Meanwhile my Pi4 is running k3s, to host a few services needed to operate the main cluster, such as the Harbor registry operating as a cache and a zigbee2mqtt instance because I have a raspbee2 for a zigbee adapter.
The main reason I'm using K3S even on the single node Pi is because I very much like using flux to manage the deployments on the servers.
Network wise, I've got a USG-3P, one of the newer compact 16 port POE switch. And a pair of UAP-AC-LITE for APs.
Maybe one day I'll get around to switching the USG for something a little more capable. And maybe capable of doing IPS/IDS on my 500M/100M internet connection. But no idea what kind of specs I'd need for that.
Would also like a NAS but... eh.. Maybe I'll just see if i can add more storage to the proxmox nodes and expand the ceph cluster or something.
And seriously, Talos Linux is really, really, nice. If I ever manage to mess up a kubernetes node (which has happened a few times when I was messing around), I just wipe it, reboot it from the ISO, and reprovision it with the machine configuration.
This sounds a lot like my old cluster config (I stepped away from the lab for a few months and forgot how it works, so started over lmao), but basically it would spin up a talos cluster on proxmox using terraform, and then bootstrap FluxCD and the rest of the software would be setup using that. It was a pretty slick system.
I'm still mostly handling the initial creation of the VM instances manually. Though I got a powershell script to help me mass send Talos configs to all the nodes.
Though I am using terraform to bootstrap fluxcd tho. And TF controller for when I just need to do something with TF and inject outputs as a secret.
Intel NUC with a hard drive for local stuff (*arrs, jellyfin), but nowadays because I plan to go back to full-time motorhoming I fire up stuff on DO, hetzner, AWS, GCS, etc as required. At the moment just a Lemmy and general purpose instance, but I do pop up the odd gameserver I've dockerized on one of these services while playing with friends
Awesome! Yeah, my instances are currently running on DO, but it's pretty expensive hosting in the cloud when you have a lab at home. My internet here isn't very good though, that's the main thing stopping me from moving them on-prem.
Joe's datacenter & hetzner server auctions are good deals if you've got bad internet and want to run your own multiple smaller VMs! Depending on latency in the case of hetzner.
But yeah, hosting at home is always great. I did it for years, but electricity prices began creeping up and I got tired of the maintenance
Yeah, that's true, they do have pretty good prices. I like DO though because it's where I started and they have a DC not too far from me, so latency is very low.
That's my disappointment as well! I've done k3s on a droplet, and it was nice, but I'd like to handover the control plane to a cloud provider when I'm experimenting without burning my wallet.
I’ve just built my first NAS. I went very overkill for my needs, so bought 5x8TB drives, 32GB of RAM, a Ryzen 5600G, and 1TB of NVMe SSD storage to compliment my HDD space. I’ve installed TrueNAS and have plugged the HDDs into a RAID-Z2 config, yielding about 20TB of usable space, with the 1TB SSD being a separate pool for anything that can benefit from faster IO, which have generally been applications / VMs. Currently using it as a network drive for all my data, and am using BorgBackup to ensure everything’s backed up. At some point I’d like to get a remote BorgBackup repository going, but I’ve not gotten around to it yet.
I’m also using it to host a VM for my web dev projects I can access via RDP and Tailscale, and have started installing PhotoPrism for my photos, SyncThing for my dev projects (so I can continue coding across my PC and laptop seamlessly), and eventually I’d like to get Plex / Jellyfin going, though I already have a setup going via WhatBox.
Speaking of mass SSD storage, if you play a lot of Steam games, have you looked at setting up an iSCSI target on the SSD to hold your games? I did that for a while, but when I re-aranged my rack a while ago I lost 10gb to my desktop (needed the card elsewhere). Was a pretty performant setup though, all things considered.
I have a relatively small setup, because of space and cooling constraints, but in that setup:
Generic server with a Xeon E5-2697 v2, kinda old but it's still got 12c/24t, and 64 gigs of memory
Around 40TB of storage space, of which I'm using roughly 1%. I'm not even a datahoarder, I'm just a storage space hoarder.
Everything I self host runs through Proxmox, either as a LXC container or as a RHEL 9 virtual machine. I also have a RasPi running Pi-Hole for ad blocking.
My main machine is an Optiplex 7070 micro (i5 8th gen, 16gb ram, 500gb SSD + 4TB hdd). I also have a pi 3 + 4tb hdd for backups and a pi 4 for wireguard. I have a few other SFF computers, but I don't have a use for them at the moment.
For services, I host many of the popular ones (nextcloud, portainer, paperless, etc.), but here's 3 I haven't seen mentioned a lot:
komga (ebook reader, works well with tachyiomi on my tablet)
Noice, I've been meaning to setup something like paperless! Calckey looks like a good solution/alternative to Mastodon with an interesting user interface.
I haven't heard of the other two, but I'll definately check out kitchenowl, could use some more meal planning!
I have a pretty modest setup. This is just what’s in or on my cabinet rack.
Old two bay NAS
New five drive bay server I’m replacing the old NAS with and running local stable diffusion and language models on. I managed to fit my old nVidia 3070 in the rack mount case. There’s no way a card the size of a 3090 would fit
Some raspberry Pi’s
Rack mount firewall
Old Acer monitor, keyboard
Dumb PDU and an old battery backup that I replaced the batteries on
An old 802.11ac WiFi router set up as just a WAP, dedicated for home automation
Plan is to set up something like open stack but right now it’s just running unmanaged (orchestrated?) docker containers. I recently learned about ansible so may just automate the docker containers instead of figuring out open stack.
You fit a 3070 in a 5-bay NAS?? That's impressive! I haven't done much with ML, but it is a very interesting field of work. I've seen people do some pretty crazy things with it!
Ansible is nice, but have you heard of Terraform? Or, if you prefer programming/scripting as opposed to HCL/YAML, there is also Pulumi with lets you use terraform via a few different programming languages. (Ansible is nice though, I used to use it all the time in my lab, and it just works)
Add comment