The next step I would try would be to boot an other install, like a liveusb or a raspbian, on the same usb port, to completely eliminate a hardware problem if it boots properly.
Good advice. I moved the usb drive from another (working) PI and attached it to the same USB port. It boots correctly. It is not the USB port nor power.
If it is a software problem, it seems to happen very early in the boot process, so my bet would be a corrupted initramfs/initrd (or what is equivalent on a Pi). No idea how you could debug and fix that on Ubuntu, though (especially on a Pi where /boot isโฆ different).
I believe it is something like that. Or it is not mounting the drive correctly and not finding it, or it is something else. I just wish there was a better (or any) error printed on the console. I tried to attach a keyboard to get to a shell with no success. I honestly could just reformat the drive and use a clean install, but it is the last resource. I would like to understand what happened so I can learn from it and avoid it in the future (or learn a path to fix it).
I could format it and try but the moment I do I lose the ability to debug the issue and learning from the problem. So I may temporary solve it but it may happen again. I wonder if someone knows what I can check/test/run to identify what broke.
Hi, just writing it here in case you were curious of the cause of the problem.
I finally had some time today to work on it. What I tried was to copy the content of the system boot partition of another PI, one that I configured at the same time/day, and compared file by file with the content of the partition in the broken one. Now there were some diff in the some binary files as expected, but what surprised me was that one file was ONLY present in the working oneโฆ initrd.img XD Now, donโt ask me why the hell the file was not there. Maybe it got corrupted somehow, since nothing touches this partition as far as I am aware aside at boot time. Luckily, there was .bak file present in the partition which I renamed andโฆ it worked.
Lesson learned: I now have a copy of the boot partition of every PI I managed (it is only 167MB pre tar-zip, so it is not a cost) and I have it backed up safely on another system. Should another file get corrupted in the future (maybe this time without a .bak) I have an older working copy if it, and I can restore the service without the need to format everything.
Tried it out in the past couple of days to manage k8s volumes and backups on s3 and it works surprisingly well out of the box. Context: k3s running on multiple raspberry pi
Two Pi4 Model B 8gb and one Pi4 Model B 4gb. OS I am using the latest ubuntu 64. A word of warning, I am having some issues with longhorn breaking the fs of my pihole pod making them โread onlyโ. I work on this stuff as a job and I still donโt find a good explanation about wth went wrong. I have to be honest longhorn github issues are not very helpful nor there are good logs about it. I am starting to think there are too many microservices working behind the scene and too many gRPCs. tl;dr: it is hell to debug - I still donโt find a good alternative however
Eh, I will have to find my notes on the issue with the pihole, I can see if I can dig them out this weekend and send it to you (I wonder if you can send PM in Lemmy ^^).
To stay on the point of this discussion: just, and I am not joking, this afternoon I got hit by this: longhorn.io/โฆ/troubleshooting-volume-with-multipaโฆThe pod (in this case wireguard) was crashing because it could not mount the drive and the error was something like โalready mounted or mount point busyโ. I had to dig and dig but I found out the problem was the one above and I fixed it. I will now add that setting in my ansible and configure all three the PIs. However this should not happen for a mature-ish system like longhorn which may cater a userbase which may not know enough to dig into /dev . I think there should be a better way to alert the users for such an issue. Just to be clear, longorn UI and logs were nice and dandy, all good on the western front, but all was broken. Longorn reconciler could have a check that is something should be mounted, and is not, and the error is โalready mountedโ, but is not โalready mountedโ, check for known bugs. However I think the issue is what I said above. It is too fragmented and working with a miriad of other microservices, so longhorn is like โI gave the order, now whateverโ. I will share what is in my longhorn-system ns, there is no secret in here but I want to give an idea (ps: I do nothing fancy with longhorn at home - obvs some are ds so you see 3 pods because I have 3 nodes):
Please do not use my bad experience stop you! Longhorn is a nice tool and as you can read online and in other posts it works very well. I may have unlucky, have a bad configuration or had my PIs under too much pressure. Who knows! My advice is try something new: k3s, longhorn, etc. That is what I use the PIs for. I would not use Longhorn at work :D
Iโm really not sure what I even want the distributed FS for. I guess I wanted to have redundancy on the pod long term storage, but I have other ways to achieve that.
I am not using replicas :) I use longhorn for the clean/integrated backup mechanism instead of using something external. Maybe one day when I have the same-ish disk speed on all 3 PIs I will enable replicas but for now I am good like this.
For backups of important stuff maybe use something else or ALSO something else. I was personally thinking to use another backup too for longhorn devices like github.com/backube/volsync or velero to have a secondary source in case something happen. Also longhorn is always getting better. This is just out of the press github.com/longhorn/longhorn/releases/tag/v1.5.0
My advice? Try it out! If not, it will still be a source of learning and fun (but I am strange, I like to debug stuff).
Sorry to hear. If you had NO connection with kubectl, I would have adviced you to check the ports; but if sometimes it replies and most of the time not, it must be something else. Good luck with the debug and if you have any specific problem you could also try to create a post on any of the self-hosted communities here on lemmy. From my experience people is more friendly and more technical than what we used to have on reddit.
Yet another lemmy instance proudly hosted in Montreal, Canada. Still need to get a icon created, if there are any designers here who would like to contribute please feel free to post!
I think the issue with feddit.de is that, "I think", it is blocking the federetaion with this server. I may be totally wrong, since I am digging how this works for like 1h, but I can see this server as "blocked" here: https://lemmymap.feddit.de/ I wish there was a list instead of the map because it is unreadable, but you can search for sh.itjust.works there. Remember to select "blocked" first in the bottom left.
Yes (I think). The two approaches above are the ones that comes to mind. Again, I am super-new here so I really hope I am not misleading you ^^; And what @manifex just commented (reaching out ot the admin).
[SOLVED] Help debugging an issue with Raspberry Pi 4B boot
Hi all,...
Longhorn: Cloud native distributed block storage for Kubernetes (longhorn.io)
Tried it out in the past couple of days to manage k8s volumes and backups on s3 and it works surprisingly well out of the box. Context: k3s running on multiple raspberry pi
Welcome All!
Yet another lemmy instance proudly hosted in Montreal, Canada. Still need to get a icon created, if there are any designers here who would like to contribute please feel free to post!