Amazing that it’s been… 6 years (as of this writing) since I decided to pursue 10GbE.
First trying to build a custom switch, then dropping all that when I learned that a lot of retired Quanta 10GbE switches dropped on eBay. Then dropping that switch two years later for the far quieter, lighter, and just better overall MikroTik CRS317. Even ordering it direct from Latvia. And then last year replacing the fans with the far quieter, Noctua NF-A4x20 FLX.
So why am I now talking about building a router?
Google Fiber’s buggy interface
Before Google Fiber, I was with Time Warner Cable (now Spectrum), and I used my own cable modem and router. Never had any issues as a result. With Google Fiber, though, we were given their router box from the outset. As much as I don’t like not being able to use my own hardware, I didn’t really have a choice here. (Or so I thought, actually… Apparently I could’ve used my own router from the outset, but their documentation didn’t make it look that way.)
Google Fiber has changed how their routers are configured a few times. Initially, like most every router out there, you connected to it directly via the IP address. Then they made it so everything is configured by the Google Fiber site. The latter was better, since it allowed you to handle things remotely but still securely, such as enabling or disabling any port forwarding, allowing you to enable/disable it more-or-less on demand from anywhere.
Recently this has become more frustrating and buggy. Port forwarding in particular. Plus I didn’t have nearly as much control over other aspects as I would like.
Thankfully Google Fiber has an account option allowing me to use my own router and put theirs into “bridge mode”. So I did just that and switched over to using the MikroTik CRS317 as the router.
[Insert Nuke’s Top 5 voice-over]: It did not go well.
RouterOS performance
Sure port forwarding was far easier than using Google Fiber’s buggy interface. But performance… fell off a cliff. Instead of getting 2Gb down, I was getting around 500Mb. Something my research told me was largely unavoidable. Both with RouterOS versions 6 and 7.
Hardware is the primary reason. It’s just too underpowered with a dual-core ARM 32-bit processor running at only 800Mhz. That’s more than capable as a 10GbE switch, especially if you’re not loading up all of the ports. (I’m using 7 of 16 as of this writing, one being a link to a MikroTik CSS610.) As a router, though… not so much.
So the solution then is… building my own router using spare hardware I have lying around.
Requirements and Specs
The requirements are simple: gateway between the MikroTik switch and the Google Fiber box while being able to handle 2Gb up, 1Gb down without a problem. So what level of hardware would work?
Linus Tech Tips most recent video about building a router used an old Dell Optiplex 7010 with an Intel i5-3770. And with that being just a Gigabit gateway, the CPU was barely being touched.
And the hardware for the official pfSense appliances is also very lightweight. The Netgate 4100 is the lightest that would still meet my requirements. And it has an Intel Atom C338R 1.8GHz dual-core processor with 4GB RAM and sipping only a few watts of power.
I’m going a little overkill merely because I have this lying around not being used:
| CPU: | AMD A8-7600 APU with Noctua NH-D9L |
| Mainboard: | Gigabyte GA-F2A88X-D3HP |
| RAM: | 16GB DDR3-1600 |
| PSU: | EVGA 650 G2 |
| Storage: | Inland Professional 128GB 2.5″ SATA SSD |
| WAN NIC: | 10Gtek X540-10G-1T-X8 10GbE RJ45 |
| LAN NIC: | Mellanox ConnectX-2 10GbE SFP+ |
| Chassis: | Silverstone GD09 |
| Operating system: | OPNsense (with latest updates as of this writing) |
Okay, not all of it I had lying around. The 10Gtek card I needed to acquire, along with replacing the fans in the chassis, but that was it.
Now why a 10GbE card for the WAN link when I only have 2Gb service? So I don’t need to upgrade it later.
Google Fiber is rolling out 5Gb and 8Gb full-duplex service starting early 2023, so I’m already set for either option. I don’t need to swap out any hardware to support it. And with the 10GbE switch as the backbone of my home network with a 10GbE card in mine and my wife’s desktop systems, we’re already well positioned to take full advantage of it.
And if your router needs to handle faster-than-Gigabit traffic to the Internet, pay attention to PCI-E lanes with your mainboard and processor combination, in particular with slot bandwidth when you have certain slots populated to ensure you’re not cutting off bandwidth to your card(s). 2.5GbE NICs should run in a PCI-E 2.0×1 slot without issue. 5GbE and 10GbE cards require additional consideration.
Thankfully the FM2+ board and APU have enough lanes. The PCI-Express slot with the Mellanox card is wired for full x16 while the full-length slot with the 10Gtek card is wired for x4. PCI-E 2.0×4 is more than enough to handle 10GbE.
And to keep the NICs running at peak performance and cooler temperatures while still remaining nearly silent, I used 3M VHB to attach a Noctua 60mm fan to the 10Gtek NIC, and a Noctua 40mm fan to the Mellanox.
And I went with OPNsense due to it running on the newer version of FreeBSD – pfSense still uses FreeBSD 12 as of this writing but will update to version 14 with the next major release, which isn’t slated to release until July 2023.
OPNsense and Mellanox
The Mellanox card wasn’t being used out of the gate. Some searching led me to an obscure article mentioning the solution. I needed to create the file /boot/loader.conf.local with this line, which comes from the FreeBSD documentation:
mlx4en_load="YES"
But that leaves the question of why OPNsense does not have support for Mellanox cards enabled by default. Given how popular Mellanox cards are with DIY and homelab setups, they really need to have that enabled by default in future distributions. TrueNAS has that support by default. And I’m pretty sure pfSense has it, too.
So why did OPNsense not do that?
Router-hosted VPN
I have been relying on OpenVPN for a while. First installing it in a Docker container, then moving to a dedicated virtual machine. Neither was optimal, but it was really the only way I could have a self-hosted VPN.
OPNsense allowed me to move the VPN service to the router, allowing me to jettison one of my VMs. This cuts out the extra steps of the router sending traffic to what is, in essence, a second router to determine where to send the traffic.
OpenVPN is installed by default with OPNsense, but I took this as a chance to change over to the lightweight and better-performing Wireguard. And the VPN performance has been much snappier as well. Moving to Wireguard was probably a lesser part of that performance jump compared to being able to have the VPN service on the router.
Going wireless
WiFi 6 is integrated into the Google Fiber router. I do have an older Tenda AC1900 wireless router, but I wanted to keep the WiFi 6 capability. Enter TP-Link and their EAP670 WiFi 6 access point. It has a 2.5Gb RJ45 port that can also be powered via POE+ or the included 12V adapter. I have it connected directly to the 10GbE switch through another RJ45 adapter.
The beauty here is not just cost – I found it for about $150 at Micro Center – but expansion. If I need greater coverage of my house, I can install a second and set up a virtual machine as an Omada controller for hand-off with all of that configuration staying local. It also has the capability for guest networks, though I haven’t used this yet.
Performance and recommendations
My network configuration is now back to what it once was but with a couple slight improvements.
First being the custom router itself. Objectively and subjectively, it’s allowing for a much better connection to the Internet. The speed test when I put the new router into service was higher than the initial speed tests when I first got the Internet service upgrade. Probably about 15% better and it was the first time I saw >2000Mbps on the downlink during a speed test.
And there are two reasons for that improvement. The custom router being one, being able to perform a lot better than the Google Fiber router. The hardware providing the physical connections being the other.
In my last article about the CRS317, I said I used a MikroTik S+RJ10 module to connect the switch to the Google Fiber router. That’s a very high latency connection. Even with a Cat7 cable. Higher still than using dedicated RJ45 hardware. It’s just the nature of the beast.
This changeover allowed me to use an optical fiber connection between the switch and router – the first time I’ve been able to do that. Optical fiber has virtually zero latency across short runs.
And the connection from the router to the Google Fiber box is going through dedicated RJ45 hardware, not an SFP+ RJ45 module that gets very hot. No, seriously. Even with a fan, it was running at over 60°C continuously while the optical fiber modules had no issue with temperature. And with this upgrade, I was able to remove the fan I had blowing down onto the SFP+ module.
So what can you take away from this if you want to build your own router?
1. Have a high-performance switch as the backbone for your network
Avoid the cheap desktop switches. Like the ones that are under $30 for 8 ports.
Two things to look for are 1. whether it supports full-duplex and 2. the switch bandwidth. The switch bandwidth should be higher than the all the ports combined at half-duplex – e.g. an 8-port GbE switch should have switch bandwidth higher than 8Gbps. If the switch specifications don’t even mention “switch bandwidth”, then don’t bother with it as your network’s backbone.
The uplink of the switch will also matter as you’ll need to make sure it’s faster than your Internet connection. So if you’re sticking with Gigabit Ethernet but have a faster-than-Gigabit Internet connection, then something like the MikroTik CSS610 will be perfect as a backbone switch. Just make sure, again, to use an optical fiber connection between that switch and your custom router.
2. Build the router with only one (1) WAN and LAN port, if possible
Don’t build your custom router to also act as a switch. Build it only as a router. This means one port for the LAN, one for the WAN. The LAN port goes to your backbone, the WAN port to your modem or, in my case, ISP-provided router configured to act as a bridge. Even if you want to segment your network so one part is isolated from another, you can generally accomplish that far better and still maintain line-speed or near line-speed performance with a managed switch – e.g., the MikroTik CSS610.
Both ports should be also faster than your Internet connection. For example, if you have a Gigabit Internet connection, buy 2.5GbE NICs. This should ensure that you are able to max out your Internet connection. And if you have less-than-Gigabit Internet, don’t rely on any onboard Ethernet controller unless it’s an Intel chip.
Your custom router will rely on software for moving packets around, so keep it relegated to just one task – moving packets into and out of your home network while blocking everything else you didn’t explicitly request. Having it also move packets between other interfaces will only degrade performance.
So if you’re acquiring hardware to make your custom router, stick with a single dual-port card. I have two separate cards only because I’m using different media – optical fiber between the router and switch, Cat7 between the router and the Google Fiber box. Just make sure the mainboard and processor combination will have enough PCI-E lanes to allow for it. Use an AMD APU or integrated Intel graphics where possible to free up slots and lanes.
3. Connect only the switch to the router. Nothing else.
Sure this kind of seems like a duplicate of #2, but I’m mentioning it in case you decide to use a card with more than two ports.
The switch will handle everything about funneling traffic to and from your router. And if you have any other services on your network, it can prevent traffic from clashing so you can still access those services (e.g., a Plex Media Server) without impacting or being impacted by anyone else’s Internet activity. Provided you aren’t relying on a cheap switch.
4. Don’t forget the UPS
Unfortunately OPNsense appears to support only APC via a plugin you can install, but that only matters if you require monitoring and auto-shutdown. Make sure to get one rated for about… double what your router requires to operate and pay attention to the half-load battery runtime.
