First part of a new long term home project coming in. An #Ubiquiti PoE+ switch to power a small #Kubernetes cluster built using #raspberrypi nodes. Going to blog about every step once it has been completed. But it is going to be a few quarters long project doing bit by bit
Ok so got some #LEGO_Group to raise the #Ubiquiti PoE+ switch. #LEGO items 10701 and 10717 to have a quick and dirty start for the #Kubernetes cluster housing. Will design a proper housing later. I've never seen a #switch look this good, to be honest 😍
Hah, the initial #Raspberry_Pi master node has arrived! Still going to take a few weeks before the PoE+ hats will arrive, plus still need to arrange storage that doesn't nuke itself like SDcards by heavy usage. Will start prototyping and designing an initial #LEGO enclosure
First attempt at the bottom half of the #LEGO housing for a node. Need to find a way to get the RPI + PoE+ hat and the Switch into #BrickLink's designer and design a while enclosure
Preferably I'll have them in a ninety degree angle from each other. That should also help with keeping them cool. But in order to do that I need #LEGO Technic and I don't know if #LEGO_Group sells build your own sets like they do with classic lego
Which is fine until I realised I also needed to put the storage closer due to cable length. So I moved it a bit closer and also adjusted the storage to a bit more forward:
The last thing I did for today is making sure storage is stable. (Although I doubt it would have fallen over.) Now the wait is on for #kiwielectro ship the PoE+ hats for the master node.
Alright! Take over cluster from @ocramius just came in from Germany. This should speed up the project, plus makes it makes slightly repurposing those nodes in the future easier. (Yes there is a plan beyond jus the cluster and it's #lego housing.)
And exactly this is why I want to use #lego Technic for the skeleton. (Moved it from the box it was on the yellow thing it's on right now. The grey underplate is now fully resting on the surface it's on.))
#tinkerbell_oss Figured out what's wrong today and what is causing this error, TL;DR I need a custom kernel/bootloader to be able to run workflows, to be continued:
#tinkerbell_oss Decided to skip auto provisioning for now. Not because of #tinkerbell_oss but due to the fact that the RPI4 set up I've been following requires a custom kernel and initramfs to be able to run workflows from netboot: https://t.co/VejBOwwTP0
#tinkerbell_oss Still figuring out how to create that kernel and initramfs and that's going to take a while. And I really wanted to have at least SOMETHING running. So this is my MVP home #k8s cluster running #k3s. Didn't bother with the SSD yet, running purely from SDcards for now.
#tinkerbell_oss Got something else to fix for booting from SSD, but will also fix that later.
For now, the next step is getting GitHub Action runners on it to start building applications for it and have a way to deploy directly to it. There are several solutions for that, should be fun :D
Essentially lots of Docker images only have an amd64 version, maybe an arm64 but rarely an arm(7) image so running anything on the RPI3's in the cluster is unlikely unless I start building images for it.
Now the GitHub Actions Runner Helm chart I'm using also only has amd64 and
So that is going to be fun. The cool thing is setting that up is really really easy, like scary easy. Writing a Helm chart to add all the of the runner deployment and autoscaling definitions for that. Also considering putting them directly in a projects
deployment. But that results in the chicken and egg problem, so either the first deployment to the cluster has to be done manually or I'll have to store them at a central location.
However the first thing on the menu is getting Helm to work and be able to deploy from within the
Locking down the network didn't fully go as planned as I kept fully isolating the cluster from accessing anything in the network. This sucks if you want to host HTTP based services for inside your own network 🤣. Today I did something different however
So the second #LEGO Pick a Brick order to prototype with came in today. And with a more Technic focussed order, I've been able to build a simpler, more open, and more stable (still missing parts) casing for the RPI node.
Ok this is a crude but effective way of making a PoC/MVP enclosure. This is by far not what I had in mind for this, but the USB code tension is a major theme, and I need to solidify the structure a lot more, but this will do for the coming time:
It's alive! Well only the body, there is no SDcard in it yet as I need to first update the network provisioning for this VLAN. But now I have a node to experiment with that on while the master node does all the being a k8s alone (:P)
If there is anything I've learned so far is that building a solid enclosure for the node and SSD, is that it is harder than it looks. Partially because I don't have all the required parts because I'm ordering as I'm learning. So I have a couple 100 parts already, but everytime
I order it's mainly parts I didn't have enough of the previous time. So it's a slow process, but it means that even though I'm ordering way more than I need per node. It also means I have a whole bunch of parts I need anyway to build the enclosures for the other nodes.
Argh, one step forward two steps backward. I really love the idea of #tinkerbell_oss and everything call that you can do with it. But I haven't even gotten around getting the workflows to run and do their job. (It's also not #tinkerbell_oss that is to blame here for the record!)
It's that the RPI's make you jump through all kinds of hooks with PXE and net booting it. I'm probably better off building my own image that streams the k3sos iso to the SSD and kexec's into that or something.
Because all I want is a fresh node when it comes up, no reuse of whatever was previously on that node. It's maybe not what you'd normally do for a "home lab" but I'd like it because there is no litter left behind.
That p in sdap1 shouldn't be there when using an SSD over USB, but it has to be there for when doing this from an SDcard. The script I'm using has this somewhat hardcoded, and it took me long enough to release that the "fix" pointed out in this issue: https://t.co/RWujfwXkFF
Solves it, and makes the whole thing boot and works without a hitch. Next up is making sure I'm using the latest #k3os version, as for some reason the script doesn't pick up the latest version as provided. (Or I can just let it upgrade it self until the latest version
One of the things I wanted SSD's for, is A) SDcards wear out fast under high I/O, B) speed, but C) https://t.co/0nq5EhdEMv for persistent volumes. (With a S3 based backup/restore for real persistence.)
One of the things I want to try now knowing how that script works. Is to hardcode sda in it, and boot from SDcard when SSD doesn't have an MBR. Now when booting from SDcard it will install k3os on the SSD, and up on reboot k3os supports scripts and I'm looking into removing the
Installed https://t.co/0nq5EgW3UX just now (through terraform through a GitHub Actions self-hosted runner on the cluster (yes it's a bit meta)). And due to the amount of pods (24!!!!), it took the cluster a while to download all OCI images, extract them, and spin the pods up
Alright so with the latest #k3os and #raspberrypi firmware the #PoE+ fans are kicking in. The downside, they are audible when they ramp up to cool. Which happens every few 1 - 20 seconds pretty much. Need to tweak that they are pinning 10 RPM higher by default, I think
Waiting for a #LEGO Pick a Brick order with "some" parts for the nodes housing. One of the major lessons from the last few days was that the USB <-> SATA adapter #LED blinking during the night can affect our sleep. And I prefer a good night rest, so will attempt to build a less
It was fun to do this #LEGO build for the 3rd time with all the new insights from the 2nd build. This is what became the SSD enclosure (with human sleep improvement change (a.k.a. known as let us not leak #LED light)):
Aside from the need to block the SSD USB <-> SATA adapter's LED, another thing is sound from the PoE+ hat's cooling fan. It's switching between 64 and 128 RPM a lot to cool the CPU off by a few degrees celcious. Rather noise, especially if you can hear it in the bedroom at night.
So one of the things I took time for today was to tweak when to switching from 64 to 128 RPM. And it's currently set to 55 degrees celsius. Meaning it gets about 5 degrees hotter then when it would previously kick in, and instead of every few seconds it now only kicks in a
On the other side of that coin, I don't want it to get hot at all because it's still held together by #LEGO. Which can held a maximum of 80 degrees celsius according to: https://t.co/bP2FxZX8hx
But judging by the mention of Polycarbonate for transparent bricks could be interesting as well, at least for the contact points with the Pi. runs off to the pick a brick page
Ow, another neat detail is that before letting this node join the cluster. I only had to turn it on once, to get the MAC address of the board. It booted straight from USB after that 😍
Resetting the cluster to apply the PoE+ hat fan speed changes, one SSD at a time. And yes at this point I ran a USB cable from my desktop to the cluster because taking the SSD's off would be a hassle:
#geerlingguy After this wipe of the cluster was done, and all nodes were back up. It took 15 minutes for #terraform to reprovision every service running on the cluster: https://t.co/cfnhZvXfAo
Ok, this might look like exactly the same thing as the previous tweet. But this time for my current #Kubernetes cluster #Terraform crafted the kubeconfig that was used to do the deployment. Refs: https://t.co/CYBqnPqQFb
Also terraforming my current cluster will secure it more, plus make a lift and shift, or booting up a clone in another region a lot simpler. (Application specifics excluded.)
#traefik Was hoping to use #soloio_inc's Gloo instead of #traefik, but no arm64 images available by default made it, for now, an easy call go with #traefik. (Nothing against #traefik FYI, just want to experiment with Gloo more on this cluster.)
Another important milestone today. Started preparing to move an existing project over to use #RabbitMQ on the cluster instead of running on my #Synology NAS. Went all in and set up a 3 node cluster. Up next is configuring the ingress for AMQP
#RabbitMQ#Synology Been iterating over that thing, and have been duping from traffic from the one running on my NAS to see how it works with more than one node: https://t.co/mDtpYwdG6n
Found the cause of the master outage. Deathwing somehow managed to disconnect the SATA to USB adapter from the SSD used for storage so the node didn't come back up after powercycling it.
Here is something interesting I didn't notice yet. The spikes you see are the other two nodes frantically attempting to reconnect with the master node:
My lego order with special parts for a, hopefully, more stable construction also finally shipped yesterday. So early next week more experiments are starting :D!
Made scale model pillars with diagonal beams for sturdiness but still some movement possible (had bigger ones with the green bars but didn't take a photo), for connecting the blades to:
So preparing (with k3d) for a triple master cluster, S3 backup and snapshot and automatic recovery before starting it again. Will then do a series of chaos engineering tests to make sure it's resilient against power outages etc etc.
The triple master set up will have each master node on a different switch (!!!) (or access point). Mainly because one of the masters will replace my Raspberry Pi 1 that reads information from the smart meter in this house. The other extra master will have a special purpose.
Yay! The fine folks at #LEGO_Group seem to have caught up with the custom bricks orders after the holidays zerg 🎉. So I can put together the node #lego enclosures in a few days and start the cluster back up.
Still going for the 3 master setup in the long term, but since I can already start it up with a single master going to focus on building and testing power outage and recovery testing. Hopefully including having a fresh node every time it starts, including the master 🤐
This is a first for me! #LEGO_Group put the custom bricks order in a box 😱. Building the node kube's tonight 🎉. Probably powering the cluster back up next week, and hopefully the more space between components should make it require less active cooling
Hah, progress! But seems I'm missing parts, connector pegs to be specifically so I can't do anything at the moment. Good thing I ordered 200 of those the other day. Might need a few more parts, so another #lego order will go out soon 🤣. But really loving the progress here 😍
There is only one issue. Doing rapid rebuilding of the cluster to test things out is going to be a lot harder because this one won't be as easy to access as the others currently are...
Got it up and running at least https://t.co/0s4snsqtBh and while I love the #argon_forty casing I still think defaulting to using the USB3 instead of the USB2 port will leave plenty of users with I/O issues.
#argon_forty All the nodes in my #k8s cluster run on SSD's on the #USB v2 port because the v3 port has stability issues. Couldn't even write the image to the M.2 #SATA#SSD using the v3 port. Hence the extension cables to hook it to the v2 port. So it has a tail:
Whoop whoop, the #LEGO parts I ordered 6 weeks ago just came in! Time to finish the current enclosures and start with building the tower to put them in
The end result of tonight's thinking with the new #lego parts that came in. Fully hanging node enclosure with room to make it more blade-like/slide it in and out for easy maintenance but that is for another time. Plus the black technic baseplates are amazingly solid <3
The end result for tonight, it is not perfect, and there should be another set of parts arrive in two weeks, but we're getting there. Going to have to order more baseplates tho, those things are awesome and will make the whole project stable af.
Had a small set of #LEGO pieces come in a few days ago. And I finally managed to get a good and sturdy solution for the SSD, and also the RPI, which was a bit tricky with the SSD already in it. Making it about 90% done, the final thing I really want is a way to slide it in and
And that is the 3rd TF apply done. The (almost) full cluster back up in 3 commands and about 30 minutes of waiting for everything to be installed and back to running:
Next up is rook as a potential replacement for longhorn to use NFS/ISCSI for storage instead. Longhorn is really cool, but with my current number of cluster reinstalls I'm looking for something easier to recover data with
Another win for the cluster today is the set up of NFS as PVC storage backend. The SSD's are useful and Longhorn did do what is intended to do. But NFS survives cluster reinstalls. And trying to KISS it here: https://t.co/AwqmBDit7w
Installed #linuxserverio's Heimdall on the cluster today. A bit sad the #Unifi app isn't an enhanced app, but I'm pretty sure I can muster up writing some #laravelphp to turn it into an enchanted app 🤔
#linuxserverio#laravelphp Had lots of fun this weekend building a platform above the switch. Starting out with these half perfect towers (the top still needs diagonal support before adding more nodes):
#linuxserverio#laravelphp It looks big, and it is big but this finally means I can start working on a long term cabling plan and have plenty of space underneath the switch to do so and for cooling:
#linuxserverio#laravelphp After fixing that it has been standing fine like this for a day now. Just need to make the space between the top of the switch and the node platform smaller:
Really happy with the results of this weekend. Swapping all of this when all the nodes where up and running and my wife was playing online games that went through this switch made it "fun" as at some point I had the live running nodes on my lap while trying to get the switch in
Worked on a PoC to make adding and removing node enclosures easier by not directly making them part of the structure. But instead by putting them on a cart you can take out:
The #LEGO node enclosure for Reality came out pretty well last night. Been iterating over this design for months and it's come together really well. Both the #raspberrypi and the #SSD in it are well kept in place. All supervised by #deathwing_t_cat who just wanted a nap 🤣.
Came upstairs this morning and found out a certain cat has been on the cluster during the night and made a mess. Now the cool part is that this shows the latest iteration of node enclosure kept the SSD in place for those two nodes:
It came back up with the switch in its new raised position after I took the old raise down for redesigning. And you might notice that the #IKEA cabinet it was on is no longer standing, but is now lying on it's side providing double the space and a lot more height to work with:
Now I can put all the nodes directly on the cabinet instead of stacking them on unstable towers. There will be a better more epic #LEGO#MOC to it all once I've figured it all out. But for today I'm happy with the progress:
Also experimented with a new rear closer for cable management but folder it behind it put a lot of strain on the cable. Might need to do a few varriations:
Another important part of this project is cable management. So I started creating a hole different groups of cables can go into the structure. This was the first attempt
This is the second attempt. (Got to love me some #glowinthedark#lego.) Not fully happy but getting somewhere with this. It needs two more whole like this for all 3 groups. (Downstream switch, nodes, and upstream switch.)
Part of getting this to work was to put all my DNS records for that domain in #digitalocean's nameserver service and let #certmanager do a DNS01 challenge.
So I made the call to put all my DNS records in #terraform making managing it easier and more atomic. This makes migrating
#LEGO_Group This will make sure I can keep the cool external aesthetics of the coloured cables from the switch, without having to pull that throughout the MOC to the nodes (it just makes maintenance also a lot easier): https://toot-toot.wyrihaxim.us/@wyri/111193914559469567
Also lowered the switch, these legs are both more solid and require 100 or 200 connector pegs less to keep it up. The previous design had a flaw where the top was turned/twisted a few degrees out of alignment with the bottom due to the tensions in the design.
Also with this new design I can put the fan you see to the right inside to cool the switch off extra when needed. (It should only be needed during heatwaves; plus temperature is monitored through several systems.)
It took me a bit of timing, and cable patching, to get all cables through the cable management holes. The #Kubernetes cluster never went down as it was shifting workloads between nodes as the cables were patched:
The #lego parts for the RJ45 plug holders came in today. Here is one of them I build to make sure the parts and idea fully worked. But since we put the parts in a personal advent calendar it's going to take up to christmas before all of them are assembled and on their spot.
3/6 RJ45 holders pick up from the advent calendar, with 3 to go and 4 days left it should be straight forward getting the last 3 as well before I can start hooking them up at the rear of this MOC
#SSD and the #USB to #SATA adapter are coming later this week and early next week. If all goes well with https://www.talos.dev/ it will replace k3os. This is also the first time in 9 months I got my hands on a #raspberrypi 4 8GB, thanks to #kiwielectro
#kiwielectro Put the #RJ45 blocks in place, it's going to make managing cable a lot easier now. In the end, each node will have matching cables because both sides of the row are on the other side cables 🤣.
Cables are still somewhat messy but this is a good step in the right direction. Already have some ideas for the next steps, including a nice exit hole for the power cable
Had another experiment yesterday to be able to slide the #raspberrypi#kubernetes node enclosure in and out of the #lego MOC. The bottom is pretty solid. But the higher I got the worse it got. Sliding in and out is great, needs some tweaks but it will work. The network and #USB
cables I wanted to plug in behind it. But that would mean getting deeper and under the roof (the "roof" shown in the photos for sure won't make it into the final version). So considering making plugs in the side to plug the node into MOC.
#SideroLabs One of the things that is always important with computers is cooling them. So as such, I have a fan from #hema to cool the nodes. And right now it is always on, but I already connected a #shelly_IoT plug to be able to only turn it on when the nodes get hot enough that they need
If the experiments with https://www.talos.dev/ are positive, it will replace k3os. One of big plusses is that this is #k8s and not #k3s so I should be able to use the clusterautoscaler for autoscaling. Which needs some code to work with #unifi switches.
One of the biggest things is that since half a year ago the cluster is now fully on https://www.talos.dev/ and managed through #terraform. It's been a use success and a quick and easy move. (Well it took me a day tho.) It is also a lot easier to rebuild nodes:
The second biggest thing is that, since a couple of days ago, the cluster has not 1 but 3 control plane nodes. And for some reason hours before I started working on adding the other two the only one running then crapped out: https://toot-toot.wyrihaxim.us/@wyri/111165411454560918
Meet Deathwing, one of our #cats, he has a medical history and sometimes likes to retreat. So he started claiming the spot next to the switch tucked away in that corner. So we decided to turn it into a catbed for him. Including the roof, which needs to be added again in the current iteration but the places where needed to extend the cluster surface area.
Here is a reason why I love #kubernetes; just made #RabbitMQ replicas run in different zones. Downstairs where 2 of the control plane nodes are is a zone, my desk for the other control plane node is a zone, and the rest of the cluster behind me is a zone. So I now forced a broker in each zone, but also told it not to get on the same node as #homeassistant.
This will get me some more usage out of those nodes. Because honestly 3 nodes as control plane for a home cluster? That is overkill.
Today the long enough cables for the SFP modules came in (the other cames were 4cm shorter than advertised and about 3cm too short). So hooked it up. This now means there is a patch location to the side of the MOC. So now I have the last two ports I need for the last two nodes:
While yanking on some cables I took out the support tower for the cat bed on top. So no cat bed there for Deathwing tonight. Will do the cables better tomorrow and rebuild it. Might have to take part of the foundation loose, going to be fun.
Next up getting each node it's own fan instead of using those #hema that @erikaheidi showed cased a year ago on nodes to cool them down. They need some cleaning once in a while tho.
Put in air flow directors for some extra cooling for the switch in place as well. It's currently running at a fine 57 degrees (inside the outside is always touchable but can feel hot), but can go up to 70 especially when it is hot up here.
Monitoring everything, currently building something that queries #Prometheus and tosses it over #MQTT to @homeassistant so #nodered can turn on the cooling fan, and in the worse case scenario cut the power.
The roof was technically on, and while it's not intended for the #cats to walk on, they will. Any light force would bend the walls. So going to reinforce those first and come up with a solid skeleton and then give it another go.
Finished building most of the blade in stud.io, only thing left is the front/rear node suspension (the side with all the USB and Network sockets) and be happy with a final design.
Got several designs in the cluster, from sloppy to tight but not happy with any of them honestly.
Dust sucks, but have to keep it in mind. This is the controlplane node on my desk, didn't clean it on purpose for a few months to see how bad it would be.
Took less than a minute to clean, and the dust barely affected the cooling. But it was still a lot more than expected.
Now with these #Noctua on each node dropped the temperature per node by about 10 - 15 degrees. The reason there is a slight increase on the graph is because my wife and I started playing WoW. So two gaming PC's and two humans affected the temperature in the room by that much.
Started building the full cluster in its current state in stud.io. Almost 1100 parts for just the bare basics. Adding the cable management and patch racks next. This thing is going to be massive.
Did some #lego patch panel maintenance aligning it with improvements in stud.io I made yesterday. Also started working on and closing the cable gutter. Not per se the most boring part. The gutter behind the nodes will require patch spots for each node for #RJ45 and #USB:
Achievement unlocked: Write a #kubernetes node autoscaler in an afternoon that scales up and down with demand. Completely hacked together in #php to my particular requirements. (Absolutely not built for the long term.) So now I have two #k8s nodes behind me that are off 💙.
With some tweaking to the autoscaling, it's pretty clear it only sometimes scales up when needed. Got a bit batch of World of #Warcraft datamining scheduling for tomorrow, so will see how that impacts node usage. Overall I'm pleased with the makeshift node auto scaling. #kubernetes#k8s
Did another node cleaning, first time with the fans on all of them. They work great cooling them, but also at getting dust of then. Judging by where the dust settles it was a good call to pull the air through instead of blow it in the nodes. But nothing a (paint)brush can't remove. All nodes where as clean as if they would be new when done.
To help combat the dust, and as planned I've started working on an enclosure for the enclosures. Just a first stab at it, that worked out to have some stability issues to address.
Aside from roofing up, cable management is the next big topic. Need to take care of those cables so created a few more patch boxes for each node so it can just patch directly into the cable gutter:
One of the reasons I haven't published a blog post yet on the nodes, even though they are pretty much done. Is because I didn't finish the design yet. Well, earlier today I finished the design. Reinforced it in some locations, and included options to stack them.
The stacking will have to be designed. But finalizing the design is already a big milestone. Designs will be available (for sale) on Bricklink and/or ReBrickable at some point.
Been updating the node enclosures to this design and still very happy with it. Will design the stacking build soon and then do a blog post on the whole thing. Stacking is going to be fun with the whole big cooling fan in front. Which I'll make optional in the design so people can opt out of it. One major thing I want to do, before blogging about or and putting the design up on Bricklink/ReBrickable is to get some photo's with a camera to support the reasoning for this design.
One of the things I want to have once all nodes are in a "rack" so to say is to have some lights to indicate which node is on. Got a sonof mini wifi for each node so I can turn off the lights when no one is here. For the same reason I have big cooling fans: Our sleep at night. Light pollution is an issue with this many LED's around. So with the sonof mini wifi, and some node-red utilizing control through home assistant, with data from the everything presence 1. It will only turn on those lights
After that, I did the node patch boxes and hooked each node into it. Tomorrow I'm putting a wall around the gutter to keep everything in one place. And after that figure out how to create a patch box for the USB lighting cables.
This is my current power supply situation for the cluster worker nodes, #PhilipsHue, and the switch supplemental cooling fan. Not ideal, but perfectly fine if it wasn't in the way of the #lego MOC I've been building around the cluster. So it has to move:
The initial thought was to use this one and put it under the MOC. Except it has 3 ports .... which is problematic because of the bulkyness of one adapter:
So instead I went with this tower, we have the same one downstairs powering the living room #lego display. Plenty of space and nothing will block each other. Plus it comes with 4 #USB powers we will utilize for (more) lego display lights and planned #ESPHome sensors on the stairs to the home office. Just need to pick and time and date as doing this will take that entire switch down, and create a network split between two and one control plane #kubernetes nodes.
The fun part is that I've already accidentally ran with 3 nodes, while I need 4 over the past week occasionally. Tweaked my home brow #kubernetes cluster autoscaler a bit to aggressive and it took out a node while it shouldn't.
Thing is that I used to have a fixed always on #PoE+ node. But changed it so that all of them can be turn on and off. And because it will always turn the longest running off, it's now cycling through all the nodes over time and each one of them gets run down time.
Add comment