Many Network Interfaces per VM/CT - Good Practice?

Trainguyrom, 10 days ago

It really sounds like you need to dive into firewall rules. Generally you lean on your firewall to allow and restrict access to services. Probably the easiest place to start is to setup pfsense/opnsense since it has a really clean interface for setting up rules. Proxmox’s built in firewall is nice too, but configuring the firewall per VM would probably get annoying and difficult after a while

And as you learn more about firewalls learning how subnetting works will allow for more efficient rules (for example, if you have 192.168.0.0/23 192.168.2.0/24 and 192.168.3.0/ 24 for your networks that you’re allowing traffic to/from you can just enter one firewall rule for 192.168.0.0/22 rather than 3 separate rules)

anon2963, 8 days ago

Thank you, that is a very good point, I never thought of that. Just to confirm, best standard practice is for every connection, even as simple as a Nextcloud server accessing an NFS server, to go through the firewall?

Then I could just have one interface per host but use Proxmox host ID as the VLAN so they are all unique. Then, I would make a trunk on the guest OPNsense VM. In that way it is a router on a stick.

I was a bit hesitant to do firewall rules based off of IP addresses, as a compromised host could change its IP address. However, if each host is on its own VLAN, then I could add a firewall rule to only allow through the 1 “legitimate” IP per VLAN. The rules per subnet would still work though.

I feel like I may have to allow a couple CT/VMs to communicate without going through the firewall simply for performance reasons. Has that ever been a concern for you? None of the routing or switching would be hardware accelerated.

Trainguyrom, 7 days ago (edited 7 days ago)

So from my experience you generally will have different zomes of security. Outside Internet is obviously entirely untrusted so block every incoming connection except those you really need, and even then ideally all remain blocked (especially for a home network). Then you generally have your guest network which might need access to some hosted resources but is largely just used by guests to connect to the internet, next is your client network where your computer likely lives which probably gets access to all hosted resources but no management access (or depending on how much you want to trust your primary PC, limit that to just your main PC) and finally your datacenter network where you hopefully trust everything running in there.

You generally work with these zones and write rules based on the zone the traffic is coming from, with some exceptions, such as I might not want to give the guest network any access to my data center network, except for access to my jellyfin so I’ll create a rule allowing only tcp web traffic from that network to a specific port on a specific IP/hostname.

A common way to achieve this is with a DMZ network, a network that sits between all of your networks and relies heavily on routing and firewalls. Public services and routers get IP addresses on the DMZ, and your firewall only allows specific paths. The outside Internet can open connections to the web ports of the web server and nothing else, the web server can’t open connections to your other networks, only specific machines/networks are allowed to access the SSH port of the web server, etc. the DMZ is where trusted and untrusted connections mix, hence why its named after the zone that belongs to both North and South Korea where both are allowed but also neither are allowed, where one only goes with specific purpose and explicit permission

I was a bit hesitant to do firewall rules based off of IP addresses, as a compromised host could change its IP address

Realistically any identifier you can write firewall rules based off of can be forged in some way. A rogue machine can change it’s host name, IP address and MAC address (and many do randomize their MAC address these days) in enterprises this is generally mitigated through limiting a network to only Ethernet access or via 802.1X authentication on WiFi and potentially even Ethernet. (You can also take the approach of MAC address whitelists, and some switches even allow for “sticky” MAC addresses where the first MAC address that connects is whitelisted until either the switch is rebooted or an administrator explicitly clears/allows the MAC address)

However, if each host is on its own VLAN, then I could add a firewall rule to only allow through the 1 “legitimate” IP per VLAN

You could go crazy and do everything at L3 (which your idea is basically doing but with extra steps) but that sounds like far more effort than it’s worth, since now you’re making every client also act as a router, and you lose a ton of efficiency both in configuration and in routing & switching, plus you’ve now changed the type of threats you’re vulnerable to.

Generally in the enterprise, risks like what you’re trying to mitigate are handled through reporting. An automated alert email is sent when a new device connects to a network that should never have new devices connect to it, then you kill the connection and verify with the team of that was any of them and investigate if it wasn’t.

Realistically as a home network your threat model is automated scripts and maybe a script kiddie trying to get in. You really just need higher than average security to mitigate such a threat model (and average security is a shit show)

I feel like I may have to allow a couple CT/VMs to communicate without going through the firewall simply for performance reasons. Has that ever been a concern for you?

Security is always a trade off of convenience and speed. You have to decide what is an acceptable compromise between security and efficiency

Generally anything virtual when you aren’t sure what to do, you should look at what the physical solution would be. For example, network storage is very bandwidth intensive, latency sensitive and security intensive. This is usually secured at the physical level as a separate network with no routers so that most security can be disabled. So at the virtual level these would be tackled with a separate virtual network connected to a second interface, and firewall rules on other interfaces to disallow incoming and outgoing connections to the storage network

Edit: I just realized I never answered your first question. In short, from what I’ve seen most enterprises put one firewall from a vendor like Fortinet, Zscaler, Palo Alto, etc. right on the edge of the network closest to the internet then either entirely rely on that for firewall or rely on that for firewalling off the outside Internet then do additional firewalling with a different tool inside the network. For example, a bank I worked at had a pair of redundant L3 switchs (Nexus N9ks specifically) which handled all of the routing for all of the bank’s networks, and connected between those and the internet was the Fortinet box which was managed by an outside vendor and while i was there as part of hardening ahead of a scheduled red team audit we setup firewall rules (I’m blanking on the Cisco term for it, but they’re ultimately just firewall rules) on the L3 switches to limit access to more sensitive networks and services

anon2963, 7 days ago

Thank you for the detailed reply. You seem very knowledgeable. I will implement your suggestions as I redesign my network.

Trainguyrom, 7 days ago

No problem! I’m just an information sponge and I’ve lucked out with really good mentors so far in my career to learn from

pyrosis, 10 days ago

I use using docker networks but that’s me. They are created for every service and it’s easy to target the gateway. Just make sure DNS is correct for your hostnames.

Lately I’ve been optimizing remote services for reverse proxy passthru. Did you know that it can break streams momentarily and make your proxy work a little harder if your host names don’t match outside and in?

So in other words if you want full passthru of a tcp or udp stream to your server without the proxy breaking it then opening a new stream you would have to make sure the internal network and external network are using the same fqdn for the service you are targeting.

It actually can break passthru via sni if they don’t use the same hostname and cause a slight delay. Kinda matters for things like streaming videos. Especially if you are using a reverse proxy and the service supports quic or http2.

So a reverse proxy entry that simply passes without breaking the stream and resending it might ook like…

Obviously you would need to get the http port working on jellyfin and have ipv6 working with internal DNS in this example.

<span style="color:#323232;">server {
</span><span style="color:#323232;">    listen 443 ssl;
</span><span style="color:#323232;">    listen [::]:443 ssl;  # Listen on IPv6 address
</span><span style="color:#323232;">
</span><span style="color:#323232;">    server_name jellyfin.example.net;
</span><span style="color:#323232;">
</span><span style="color:#323232;">    ssl_certificate /path/to/ssl_certificate.crt;
</span><span style="color:#323232;">    ssl_certificate_key /path/to/ssl_certificate.key;
</span><span style="color:#323232;">
</span><span style="color:#323232;">    location / {
</span><span style="color:#323232;">        proxy_pass https://jellyfin.example.net:8920;  # Use FQDN
</span><span style="color:#323232;">        ...
</span><span style="color:#323232;">    }
</span><span style="color:#323232;">}
</span>

testfactor, 11 days ago

I wouldn’t let every VM have an interface into your management network, regardless of how you implement this. Your management network should be segregated with the ability to route to all the other VLANs with an appropriate firewall setup that only allows “related/established” connections back into it.

As for your services, having them on separate VLANs is fine, but it seems like you would benefit from having a reverse proxy to forward things to the appropriate VLAN, to reduce your management overhead.

But in general, having multiple interfaces per VM is fine. There shouldn’t be any performance hit or anything. But remember that if you have a compromised VM, it’ll be on any networks you give it an interface in, so minimizing that is key for security purposes. Ideally it would live in a VLAN that only has Internet access and/or direct access to your reverse proxy.

Im_old, 11 days ago

Having multiple interfaces in each vm can lead to issues with routing if you screw something up.

Like you said I’d expose the services via reverse proxy in the public vlan, and enable ssh access on the firewall only from a jumpbox or the ip of your pc (or maybe the vlan you are in).

testfactor, 11 days ago

Yeah, it can for sure. Definitely worth mentioning. Gotta watch what interface is set as the default router, or you’re bound to have a bad time. That said, the same is true with his originally proposed solution of pushing a trunk port to the VM, so it’s not any worse in that regard.

But yeah, full agreement on the correct solution. Keep it simple.

realbadat, 11 days ago

Agreed, I prefer trunk with native to the vlan for services, each container that the reverse proxy will hit in its own vlan (or multiples for differing sets of services, but I can be excessive).

I’d block any traffic initiated from that vlan to all others, and I’d also only allow the specific ports needed for the services. Then fully open initiated from the general internal vlan.

pyrosis, 10 days ago

I agree with this. The only vm I have that has multiple interfaces is an opnsense router vm heavily optimized for kvm to reach 10gb speeds.

One of the interfaces beyond wan and lan is an interface that links to a proxmox services bridge. It’s a proxbridge I gave to a container and is just a gateway in opnsense. It points traffic destined for services directly at the container ip. It keeps the service traffic on the bridge instead of having to hit the physical network.