My homelab is essentially my own passion project and only really I access it except for when I spin up the occasional game server for friends.
I’m currently running Proxmox and run a debian LXC container for each docker stack I have, and have OpnSense routing incoming traffic with Haproxy with ssl offloading. My currently running LXCs are: mediawiki, amp game server(2 Minecraft servers), freshrss, and currently playing around with n8n.
I’m looking to collapse my LXC’s to just VMs. I’d like to be able to have 3 VMs running in a Docker Swarm together so I can upgrade a VM at a time and just swing my running containers to another docker node and then swing back when the VM is stable again.
I’ve looked at k0s, k3s, and k8s and it just seems way too much work and overhead for what I’m willing to do. I also want to keep using docker compose and want a decent webgui to manage my containers/nodes/swarm. I’m using DockHand right now, but need to research swarm support.
Anyone have any advice for something like this? Any specific terms, tech, software I should look into?
Also, gonna throw a curveball, but what would the effects be of running 3 different distros as my nodes in my swarm? Like a Debian node, Rocky Linux node and potentially arch node? I’m guessing I shouldn’t due to docker engine differences potentially.
I’m just trying to have fun with things, break things, fix them, learn, etc.
Regarding different distros: yes, absolutely possible! Considering that swarm development is basically stopped the risk of inconsistencies is lower than back in the days but I’d pin the engine version on all nodes and only explicitly upgrade all at the same time to a newer version.
Regarding swarm itself, even though I understand your desire, I’d recommend to have a look at something else. Have you had a look at Nomad for instance? It’s not compose compatible but I guess these days it’s a simple prompt to convert the compose files, it’s supposed to be simpler than K8s and still offers many perks (like the failover you are looking for).
Also, from what I just read, Proxmox is working on improving the container support to allow “native” containers (inside VMs similar to Kata containers). They’re not there yet but I am wondering if it’s worth the effort from your side right now 😅
Either way, have fun! A fellow homelab buddy 😄
Glad to hear I’m not crazy with different distros haha. I wanted to be able to have different “enviroments” to keep familiar with release schedules, package managers, and just the flow of the distro. I’ve been using nothing but Debian/Ubuntu in my homelab for ~10 years now, at work we use RHEL, and for my desktops I’m on Arch. I’ll have to look into pinning.
I’ve never heard of Nomad(love the name) so I’ll definitely add it to my list of things to research. Looking at their site it looks solid, but want to weigh my options once I’ve loomed at everything.
And thanks for your comment! I’ve been doing this a long time and nothing “tickles” my brain more than something in my homelab breaking and I have to figure out how to fix it and then prevent it going forward.