This is the steps of my home network upgrade to 10Gb/s.
1G LAN, 1G WAN
For almost two years my network was basically like this:
- Blue lines: 1000Base-TX 1G Ethernet lines. All unmarked ones are regular Cat 6 / 250 Mhz cables.
- Red lines: Fiber optic cables (all SMF here).
- OTO: Optical Termination Outlet, where FTTH is terminated at my location.
I have been using HP 1920 series switches for more than 5 years and very happy with them. They support everthing you need in an home network (VLANs etc.) and non-PoE ones are fanless.
I have been using UniFi Access Points (first AC-LR, and now nanoHD) for more than 5 years. I dont like they need a separate controller software for configuration, but they work great so I am still using them.
I used EdgeRouter4 for more than two years and if you need a small router/firewall to support up to 1G speed, it is great.
I used QNAP TS-431+ for around 5 years. There is nothing I particularly like or dislike about it, but on the other hand I had almost no issues with it, it just works, so I dont hesitate to recommend a QNAP NAS.
You may ask why I did not connect WAN fiber to EdgeRouter4 directly but it goes to HP 1920 first. The reason is I can mirror that port on HP 1920 and listen it, very useful for debugging or gathering network statistics. You might also ask why there is a fiber between EdgeRouter4 and HP 1920. It is only for fun, because I had these SFP modules at hand.
1G LAN, 1G WAN with IDS/IPS
Recently, I was thinking if/how I can integrate IDS/IPS to this network, and also maybe open a way to improve LAN/WAN speeds to 10G. Since I already have a few and familiar with Ubiquiti products, I decided to give a try to Dream Machine Pro (UDM Pro). So the network became something like this:
You will definitely ask why there is still EdgeRouter 4 there. The main reason is IPv6 RA configuration on UDM Pro is not configurable enough for my need. So although it is probably OK for many people, I was not very happy with it, and this setup is too complex than it needs to be.
10G LAN, 1G WAN
Meanwhile, my ISP, the one and only init7, announced they are going to introduce 10Gbps and 25Gbps (!) connections for end-users at the same price of 1Gbps. I also had in my mind to upgrade the home network to support 10G speeds. So I decided to go with this and opt for 10Gbps WAN. They are rolling out 10G/25G in phases, and while I wait for it, I changed my network to this:
- Bold lines are 10G lines. Blue ones are copper, red one is fiber optic.
- If not particularly mentioned, a normal blue line is a regular Cat 6 / 250 Mhz.
Many changes happened here.
I replaced both EdgeRouter4 and UDM Pro with pfSense running on a Dell T140 (Xeon 3.4Ghz / 32GB RAM). Dell T140 has dual 1G ports, and I also installed a dual 10G (SFP+) ports Intel X710 based NIC. The two 1G blue lines in the figure are for management network and for iDRAC remote management port of T140.
I replaced the main switch (HP 1920) with QNAP QSW-M408-4C. This is maybe a little expensive but a simple nice switch. It offers 8 1G ports, and 4 combo SFP+/1G/2.5G/5G/10G ports. I mainly wanted to have this switch because I did not know if I can reach 10G from my PC (more on this below), so maybe I could use 2.5G or 5G ports. It is a simpler switch than HP 1920 series, but it is fine for me. There are only two features I was using and missing on this one: 1) management VLAN is not configured and it is always VLAN 1, 2) there is no port mirror capability, neither is a blocker.
I installed a single 1G/2.5G/5G/10G port QNAP 10G1T NIC to my PC. I would prefer to have a NIC with Intel (or maybe Mellanox) chipset but they are 2x more expensive and this one supports 2.5G/5G speeds, so I decided to give it a try.
I replaced my NAS with a recent more powerful QNAP TS-932PX model with dual 10G SFP+ ports.
I replaced the cables from my PC to QNAP switch with Cat 7. Naturally I cannot replace the cable inside the wall so it is still Cat 6 / 350 Mhz. I know Cat 6 can theoretically support 10G up to 55m or so, but I was not sure. It seems there is no problem and it can easily support 10G. So it seems actually I did not need Cat 7 (or even 6A).
The network speed from my PC to pfSense tested with iPerf 3 is 9.4 Gbits/s.
The sequential IO performance from my PC to QNAP NAS is roughly:
- for HDDs: 700-800 MB/s read, 300-400 MB/s
- for SDDs: 700-800 MB/s, 500-600 MB/s
It was around 100MB/s with my previous NAS with 1G connection (keep in mind HDDs are same). I tried various setups (RAID-0, 1, 10, 5) and interestingly the results are not very different. Only difference I can clearly see is with SSDs you have better write performance especially random write performance is much better. However, considering the difference with my previous NAS, I think the most important thing is not SSD but having a 10G connection. Because even with HDDs, sequential IO with any RAID configuration is faster than 1G network. I think in my current setup 10G is also the limiting factor, because SSDs have ~500MB/s performance, so in a RAID configuration this can easily pass 10G.
10G LAN, 1G WAN #2
Edit: This step is completed today on 13.07.2021 without any issues.
Edit (2022/05/01): I am still waiting for the 10G WAN connection. Meanwhile I have changed the 10G network adapter on my PC to an Intel X550 based 2 ports network adapter.
You probably realized HP 1920 switch above is unnecessary. The reason is I do not have right SFP module yet (waiting for it). When I have it, it will be like this:
It is almost same as before, the only difference is WAN connection goes directly to pfSense, and HP 1920 24 ports switch is removed.
10G LAN, 10G WAN
Edit (02.03.2022): This step is completed today without any issues.
Finally, when my WAN connection is upgraded to 10G, it is going to be like this:
Same as before, only the 1G SFP module is replaced with a 10G SFP+ module (FS.com 10GBASE-BX SFP+ 1270nm-TX/1330nm-RX 10km DOM LC SMF Transceiver Module).
On Speedtest from my PC to init7 (Winterthur), I see over 9Gbps download and upload.
When I have another 10G devices around, I guess I am going to replace the 8 port HP 1920 with another 10G switch like the QNAP I use.
10G LAN, 10G WAN #2 (08.04.2022)
Edit: This step is added later to this post.
I installed a Proxmox VE virtualization server and need some flexibility with the 10G ports. So I decided to get a QSW-M1208-8C and use that with my PC and the server. I still use the HP 1920 to connect the management and 1G ports. So the network is like this now:
What I learned from this upgrade
- Cat 6 easily supports 10G at home (meaning in short distances).
- DAC cables for 10G are quite nice, cheap and easy to use. They are more expensive than Cat 7 cables but the overall cost depends on the price of the 10G switch also.
- pfSense is very nice, much better than my experience with consumer products.
- Running pfSense on a host with remote management capability (e.g. Dell iDRAC) is very useful.
- Having a NAS with 10G ports is a very effective upgrade.
- 10G switches are still expensive, and there are limited options. I think, usually ones with RJ-45 (10GBase-TX) ports are more expensive. SFP+ ports are more flexible since you can use fiber, RJ45 or DAC cables, but compatibility with SFP+ modules or DAC cables might be an issue, so it requires more attention. I purchased another QNAP with 12x 10G ports which is the best fit to my environment and also the most cost-effective solution with basic L2 layer management capabilities.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.