Hi selfhosters,
I’m considering improving the security of my homelab/server setup and I’d love to hear some more experienced opinions on my plans.
Like most self-hosters, I started out with a single-server setup: I cobbled together some old hardware, slapped on a NAS-style OS, bolted on Docker and started experimenting. As things go, my setup has grown a lot over the years. I’m running a lot on that machine - NAS capability, local DNS via pihole, a Wireguard VPN endpoint (so I can tunnel into the LAN from outside), many private services (accessible only from inside the LAN), but also some public ones accessible from the open web.
Although I’ve tried to do my homework in terms of security (various hardening measures for SSH and Docker, reverse proxy, proper SSL termination for everything), I realise I’ve put a lot of eggs in that one basket and I might want to introduce some network segmentation.
I don’t think I’m ready for getting a hardware firewall or a router/switch capable of VLANs just yet. What I’m considering, though, is putting my services available from the open internet on a separate server inside my network - a little Pi 5 I’ve still got lying around. My idea is that if a container on that is breached and if an attacker manages to escape the container to gain root access, compromise would be limited to the few of publicly available services I have on the Pi, thus limiting blast radius. If combined with (software) firewall rules that make the Pi a ‘pariah node’ (dropping all incoming connection efforts), lateral movement to other devices on the network could be contained.
An alternative would be to finally make the move to Proxmox. Everything would still be running on one physical machine, but I’d create several VMs: there would be one for private, one for publicly available services, and I’d firewall the latter off rigorously so it cannot connect to anything else.
I’d like your opinion on these plans. Which one would you recommend? Are they both sound, or is there anything I haven’t considered? Are there other (easy, non-hardware intense) things I could do to improve security even more?
You must log in or register to comment.
Security architect at a large enterprise here, so grain of salt with what I say as it may be overkill for your use case. First question you have to ask yourself is so what? If they completely compromise your devices, what impact will that have? Establish what your hard and soft limits are. Are you ok with your device becoming a crypto miner? If not, what controls can you put in place? e.g. CPU/GPU monitoring and alerting. Are you worried about threat actors stealing your personal data? Consider encrypting your drives and applying least privilege (don’t have a single user account). Worried they will do the favour of encrypting your drives for you? Look into off site storage.
If you are only using docker, the switch to Proxmox may give you trouble as docker isn’t supported. That said, I run Proxmox and have everything as an LXC and I think it is an excellent technology. If you stay on Docker, look up the OWASP Docker cheat sheet. It will help you to lock everything down tightly. You could also migrate to podman. Still not comfortable with the risks and controls at that point? Look at creating docker networks or a SDN. Also, log into your router. You may find that you can create a DMZ to limit an adversaries ability to pivot.
Feel free to reach out and ask more.
Thanks for your detailed reply and expertise! I really appreciate it!
First question you have to ask yourself is so what?
You’re right to call out my lack of threat modeling. I am aware that the “cost” of what I’m planning here likely outweighs the actual risks. That’s a deliberate choice though: one of my objectives in homelabbing is to “learn how the big guys do it”, i.e. learning and implementing, bit by bit, networking and security best practices.
That said,
- all of my drives are (LUKS) encrypted (although this will only help against evil maid attacks physically stealing my disks, since data is only encrypted at rest, not while the server is in operation)
- I do have some CPU monitoring in place
- I have applied the principle of least privilege (as far as I can see at least)
- I have set up an off-site backup server running (although I need to improve automation and deduplication; I’ll probably go with Borg here)
Thank you also for pointing me towards OWASP Docker cheat sheet! I didn’t know this resource yet, although I seem to have found others that said the same thing. I already have in place:
- no docker socket exposed to services
- user set for every service that didn’t give me a headache doing that (some assume root and won’t accept setting a user)
- username remapping is active
- all containers have their caps dropped and selectively only put those back in which they need
- I expose almost no container ports, and only those necessary; instead, communication with the containers is handled on my reverse proxy’s network (i.e. via the internal docker network)
- privilege escalation is disabled
I have played around with rootless docker (again, stubborn containers assuming root prevented me from going all-in) and considered Podman, although just putting containers in a VM looks like a more frictionless route to me.
All things considered, your advice seems to be:
- to first consider whether I actually need all this security. I really don’t, yet I want to learn how to set it up anyway.
- to first explore and all other hardening avenues. I have, to a certain degree, and beyond that run into brick walls.
If I might exhaust a little more of your time, I’d be curious how you judge the next step I’ve suggested: splitting duties between separate servers, thus segmenting public from private physically, or going the virtual route (again, by splitting into a public and a private VM and separate them through strict software firewalling).
I’ll start again with the threat modelling. How worried are you about an adversary acting with a compromised container? Is it just fear of an attacker or are you trying to protect something specific? If so, identify that service and/or data and isolate it. Start with what you want to protect. Does it need to be on a different physical device? If not, do you need kernel level separation (VM) between systems? That separation will have overhead associated with it. Does the compute cost reduce your risk enough to justify the separation? Moving onto multiple different boxes has its challenges as well. Yes you are more segmented but do you have the network bandwidth to support the services you want? Also, do you need those publicly available services actually exposed to the internet? If you are already using wireguard, what difference does it make to create keys for your other devices to be able to access the environment? In this age of AI running rampant, reducing your external attack surface is the single greatest thing you can do to protect yourself.
Opinion time. Build on what you have and know. If you haven’t physically separated your machines yet and everything is running on docker, look into building docker networks. If you have the networking expertise and multiple devices to run things, bite the bullet and migrate to a new set of systems. From what it sounds like, you don’t have much experience with Proxmox and that is fine. Running a few Ubuntu servers with vms and containers works great as well. Look into your firewall rules. You can use AI to help you draft a firewall table. What services need to talk to what systems? Whitelist those and blacklist everything. Once you have an understanding, look into VLANs. I personally have 3 machines running on my home domain and they are separated by functionality with like 8 separate vlans woven between them. Adversaries care about home servers for two reasons (mainly): crypto mining and botnets. Your description of the hardening already manages most of these threats. Physical or virtual both work but build your controls and networks around what you want and know. Does that make sense?
+1 for owasp resources.
I personally go with VMs. Except of some services that I run on Hetzner.
I got one Debian server that runs multiple VMs with tailscale. Then one Hetzner VPS with Caddy. This one acts as a public gateway.
Some things are still running bare metal, like Immich or Jellyfin because of apple hardware. However the level of entropy I have now is th opposite of all eggs in on basket. I sometimes forget what do I have where and have to look it up in my docs.
This sounds a lot like my second option, except yours is a little more custom-made. What do you use to set up and run your VMs?
virt-manager for qemu. I also did some cloud init script with LLM to have a nice way of spinning up new vms with a server user, ssh and my access keys. VMs have also on big advantage. It’s easy to back up entire qcow2 disk image and then if necessary move entire VM to a different host.
Ah! I use virt-manager + qemu on desktop Linux, mainly to try things and and contain dodgy software (Windows). I didn’t know it was commonly used on headless servers. Thanks for pointing that out!
combine your VMs with a little layered LXC garnish for that special touch :-)
I respect what you said about your own firewall, however I disagree with it! ISP modem/firewall’s are rarely if ever updated/upgraded. I thought I was fine without a hardware firewall until I started getting into some of my more darring adventures into home labbing. I was using DuckDNS for things, but decided I wanted to be able to do more so I needed a static IP. Luckily my ISP offers two static IPs for free so I got those.
I hope part of your hardening SSH is changing the port number! If you want to keep your LAN safer with a firewall, and something small like a pi zero (I use a pi zero 2), you can setup a honey pot vLAN. Again I realize you said not interested in a firewall, but I am saying a honepot helps.
Thanks for your input. It is correct that a lot of ISPs/consumer grade hardware is poorly maintained. Mine is a rare exception, however: I’m running an AVM FritzBox, which are somewhat famous for still receiving updates even after 10+ years on the market. They have only recently pushed a security update my way, so I’m good enough on that front - for now.
I shall consider a firewall in the future though.
I hope part of your hardening SSH is changing the port number!
I used to do that, but then realised that doing so will do very little. Scanning all possible ports is a matter of seconds, and SSH (unlike Wireguard) will signal to a potential attacker that it’s listening on port X instead of just remaining silent. In other words: in my eyes, it’s security by obscurity and will have very little effect.
What I have done to harden SSH is:
- set up pubkey authentication and disable password authentication
- disable root login
- not expose my SSH port publicly; it is accessible only from my LAN and through Wireguard
I have also tried out IP whitelisting (which may harden my setup against attacks from the LAN), but found it may be overkill for now. I might turn it back on just for shits and giggles.
I was in your position a few years ago. Went the Proxmox route. For many services, don’t make separate VMs, but instead separate LXC containers. Saves time and space.
I’ve done my research on that, and apparently LXCs share the host kernel, meaning: if the LXC is compromised, so is the host system. So if I go with Proxmox, I’ll probably swallow the (somewhat) larger overhead of running a full VM.
