I currently run all my self-hosted apps either on Podman in a VM or in LXCs on Proxmox. For hardware, I’m using a Chinese-made mini computer with an Intel N150 and 16GB of DDR5 RAM that I bought before the whole AI hype started. I also have a Synology NAS that I use mainly for media and photo storage.
I’ve been thinking about tinkering with Kubernetes in my homelab for a while now (I already use it extensively at work, so I’m quite familiar with it), and I started looking around for used hardware to use as bare metal nodes. Nothing fancy—I’m looking for 1 or 2 mini servers or SFF with at least 16GB of memory and a decent CPU (4–6 cores). But with current prices, even decently priced used hardware (~200–250€) is quite difficult to find in Europe, and most of it is HP stuff with Lenovo being a rare breed around here. I won’t even get started on newly bought hardware…
If you’ve bought hardware in this market recently, how did it go for you? Or are most of you holding out for now, waiting for better times?
Honest question, why are you looking to buy hardware to do this when you can mess with K8S in VMs on your existing proxmox host?
Fair point. If I don’t want to just run a single node cluster, I will need at least 2-3 nodes aka VMs. Looking at like 2-4GB (as a bare minimum imho) memory for each, I would already eat into like half of my current 16GB of available memory. And this mini server is already running a lot of my critical apps, using more than half of the memory.
That’s fair.
I tried this on proxmox and ran into a lot of installation issues. Any tips?
Have you watched the Learn Linux TV video series on YouTube? He does a good job at walking you through k8s from nothing using cloud init images, but you don’t need that part.
Will check it out thanks
Nothing fancy—I’m looking for 1 or 2 mini servers or SFF with at least 16GB of memory and a decent CPU (4–6 cores)
Not in the EU, so…
The Optiplex 7020 SFF with the i7-4790 chip are pretty robust little computers, regularly sell on Amazon for less than $200. The 9020 MT is a bit older, however still capable @ well under $200 USD on Amazon. I’d upgrade the 9020 to a i7-4790 chip which you can normally find, used, on ebay for less than $50 USD. The models in that range take DDR3 which you can find good prices for at https://www.memorystock.com/. I’ve been using them for years. The bad part about shopping for used computers on ebay is that, while you might be able to find some killer deals initially, most often they stick it to you good with shipping.
Also: Dell OptiPlex 3050 Micro Not too shabby either.
I just recently got a Optiplex 9020 MT 16 GB RAM for $124 USD / free shipping off of Amazon. Shopped around ebay for the i7-4790 which cost me $35 USD, and 16 GB DDR3 RAM for $50. Not a bad little set up for right at the $200 USD mark. I’m setting it up for a friend of mine’s son who wants to get into selfhosting at the age of 10. It’s no threadripper but a solid piece.
Non hardware answer: but for bare metal Kubernetes install I really recommend Talos OS. It is easy to install, secure and very lightweight. So you can squeeze more out of your hardware.
I personally have a handful of pi’s doing things, with plans to get more to do more (I have one in the mail to become my OpenWRT router). I was thinking of getting something simple to be able to do STT and TTS as I want to replace my HomePods as I slowly leave Apple and have have no interest in joining Amazon or Google for home automation, so I looked at my university’s used computers. I am not really a fan of buying some random persons old hardware but maybe a business is selling their old stuff.
I assume you are using pi 4 and/or 5? I would have some 3b+ laying around which don’t see any use nowadays. But can’t quite imagine to build a cluster only made up of these, I think they are too underpowered, unless being used as like a small additional worker node.
And buying new Pi 5 with 16GB of memory would set me back at least 200€ a unit, not including power supply or any case.
I’m also in the PI 4 group. Its worth watching ebay, etc for people selling pi clusters. I picked up my 8 node 4GB pi cluster fairly cheap.
Also if you’re learning and find yourself struggling with yaml files, I found Claude helped a lot to get running. Create the file, dump it into Claude and let it re-output the file. Give it the error log helps if it gets stuck.
Also if you explain your cluster setup, I built mine with Talos Linux so setup VIP, etc and it’s pretty good at generating a working setup.
I’m not a huge fan of AI but I struggled with yaml formatting so it might suck for a lot of stuff but formatting stuff properly is great… that and scanning logs for errors! 😎
I liked my Pi 3B+ for tinkering around and run some services like vaultwarden or a VPN node with PiVPN (first OpenVPN, later Wireguard). They are great for stuff like that and don’t use much power. Not sure it makes sense to add them as nodes though as 2GB is stretching it, although Talos OS has a minimal requirement for worker nodes of 1GB, so I might still try 😄
As you can see I plan to use Talos as well, it sounds really promising what I read about it. For a CNI I plan to use Cilium and for distributed storage I might try Longhorn and see how it works. Apart from that I am still on the fence if I should run my nodes bare model or put a virt layer in between with Proxmox.
I have a pi 4 running HAOS, I have an 8 gb pi 5 running a bunch of services, with head room to spare. I have a pi 5 with 4gb ram running Jellyfin with headroom to spare. Both pi 5’s are also running pihole.
I also have a pi 5 with 2gb that will hopefully handle my personal blog and email, hopefully! I have a pi 5 with 1 gb showing up on Thursday that will run my OpenWRT and LuCI. I had a pi zero 2 that handled wireguard but it could only handle it for DNS, with OpenWRT on a pi 5 I will be able to do more than just DNS over wireguard.
Do not run kubernetes nodes as LXC. I mean you can for testing etc… BUt you will have to break sandboxing because of networking capabilities kubernetes requires. At least that was the case for K3S. You could probably have better experience with Kind.
Not planning to run nodes as LXC at all. Just using them in my current setup for running services like Wireguard or Jellyfin.
At most I would create VMs for nodes, but still on the fence if I should just go bare metal or proxmox with virtualized nodes (VM).
I would go with two (or more) proxmox nodes working as one. And then running even more VMs on them as kubernetes nodes. Proxmox handles lots of stuff for yout that is harder to handle in phusical machine.
