Anybody have any docs or performance figures for how much CPU is "enough" for routing/firewalling a given amount of bandwidth?
My current core router (Debian box with an i3-4160 and a dual 10G NIC plus some 1G interfaces) is getting long in the tooth and doesn't have enough PCIe lanes to add all the stuff I want, so I'm planning its retirement some time later this year.
I'm targeting ~10 year lifetimes for new builds at this point, so it needs to have the oomph to last through a bunch of network upgrades.
I'm thinking I would be happy with something that could handle 40G or 100G cross-vlan traffic (plus a few hundred Mbps to/from the internet but that's negligible) in a router-on-a-stick configuration.
But I have no idea how much CPU you actually need to do this. Currently eyeing a Xeon 5416S (EDIT: or a 5415+ which is half as many cores but significantly higher Fmax) somewhat arbitrarily, but I have no idea if this is overkill or insufficient.
@azonenberg I vaguely recall that the 2018’ish generation Xeons could push 40Gbit/sec/core using the X710 NICs, SR-IOV, and Intel’s DPDK packet acceleration stuff - through a virtualisation layer
@azonenberg I guess a fair bit of the answer is gonna depend on what type of offload capabilities your NICs have. if you're using a high end NIC with all the bells and whistles you can probably get away with pretty much anything mid tier and relatively modern.
if you're trying to do 100G or more on DDR3 or slower DDR4 it's generally advisable to max out your memory channel count if you're doing anything more than straight up forwarding. on DDR5 it's less of an issue.
@gsuberland I'm looking at a sapphire rapids xeon with eight DDR5 channels (that was going to be one of my next questions, do I need to actually fill all those channels if I want to saturate 100G? I assumed that routing was mostly going to live in LLC and packets wouldn't sit around long enough to touch external RAM).
NIC wise I'm looking at mellanox connectx but haven't decided for certain yet.
@gsuberland In addition to routing/firewalling this machine will likely also run internal-facing DNS (world-facing DNS will be a separate service VM on a different box), DHCP to some of the VLANs, and an internet-facing VPN server endpoint.
But I don't expect any of those to be particularly demanding of CPU, my current internal DNS server is running on a pi.
@gsuberland There won't be any heavy routing protocols, I'm not running BGP and at the moment this is largely a single-site network with a few static routes for a site to site VPN link.
@azonenberg in an ideal world with that kind of system I'd pick the fastest DIMMs (balancing clock and CAS latency) that are supported in a fully populated config, then fill half (or fewer) of the slots with them and upgrade later if needed.
@azonenberg in fact this is generally good advice for any system that has a lot of memory channels and where you don't have an immediate requirement to fill them all. memory with the same clock and timings will be cheaper down the line.
@gsuberland Yeah I was thinking of using only like two DIMMs to start.
That would probably still be like 64GB of RAM with server grade DDR5 lol. My current router has 16GB of DDR3 and barely touches it (htop shows 377 MB used right now).
@gsuberland My main goal here is to not overpay buying more hardware than necessary, but also futureproof to the point that I'm meeting my anticipated demand over the next decade.
@gsuberland I don't think the extra cycle or two of latency for LRDIMM would be a problem for my use cases.
Most of the instrumentation stuff is going to be bandwidth bound, but a few us of latency will be negligible. All the hard realtime stuff will happen on FPGA before touching the network.
@azonenberg fair, although the other reason to avoid them is that LRDIMMs with matching specs will be harder to find for upgrade later due to lower production volumes.
@azonenberg@gsuberland I worry about the energy intensity of doing 100G routing in software if it’s really going to be just some static routes (unless you specifically want to work on that routing software). Something like the MikroTik CRS504 will do it and pull <30 watts from the wall as a router-on-stick.
@karppinen@gsuberland I'm looking for something off the shelf, but I also want to be able to run DHCP and DNS and VPN servers and a bunch of other network infrastructure stuff on the same box.
That system looks a little light for doing all that?
@karppinen@gsuberland In any case replacing my ancient ~150W 24 port Catalyst 2970G's with a my modern FPGA based systems aiming at maybe 30W TDP for 48 ports is already on the todo list and should save a couple hundred watts net across the lab once I reach that point.
@azonenberg@gsuberland what, 64MB RAM and 16MB flash not enough for you??? yeah, it won’t do all that but for the static routes plus perhaps the DHCP, it’d do the trick
@karppinen@gsuberland Yeah I want an OpenVPN endpoint, DHCP server, DNS server, and possibly one or two other network infrastructure type services to be able to run on it.
I''m not opposed to hardware acceleration but I do really want a full Linux system with a nontrivial amount of ram/flash that can run a standard distro.
@erincandescent@gsuberland Yeah my NAT demands fit comfortably into the decade-old i3 on my current router. I'm limited by DOCSIS here.
I'm not opposed to switching to a different firewalling platform to take advantage of acceleration. This is all new to me, I've only ever done software based stuff on Linux and, years ago, used a Cisco 2600 series router as an edge router for my parents going from a single 10baseT interface to the internet via a WIC-1ADSL lol.
Add comment