Upgrading the Home Network to 10G

This is the steps of my home network upgrade to 10Gb/s.

Edit: I keep updating this post, so it became like a history of the changes. It starts with a 1G LAN/1G WAN and goes to 10G LAN/10G WAN.

1G LAN, 1G WAN

For almost two years my network was basically like this:

• Blue lines: 1000Base-TX 1G Ethernet lines. All unmarked ones are regular Cat 6 / 250 Mhz cables.
• Red lines: Fiber optic cables (all SMF here).
• OTO: Optical Termination Outlet, where FTTH is terminated at my location.

I have been using HP 1920 series switches for more than 5 years and very happy with them. They support everthing you need in an home network (VLANs etc.) and non-PoE ones are fanless.

I have been using UniFi Access Points (first AC-LR, and now nanoHD) for more than 5 years. I dont like they need a separate controller software for configuration, but they work great so I am still using them.

I used EdgeRouter4 for more than two years and if you need a small router/firewall to support up to 1G speed, it is great.

I used QNAP TS-431+ for around 5 years. There is nothing I particularly like or dislike about it, but on the other hand I had almost no issues with it, it just works, so I dont hesitate to recommend a QNAP NAS.

You may ask why I did not connect WAN fiber to EdgeRouter4 directly but it goes to HP 1920 first. The reason is I can mirror that port on HP 1920 and listen it, very useful for debugging or gathering network statistics. You might also ask why there is a fiber between EdgeRouter4 and HP 1920. It is only for fun, because I had these SFP modules at hand.

1G LAN, 1G WAN with IDS/IPS

Recently, I was thinking if/how I can integrate IDS/IPS to this network, and also maybe open a way to improve LAN/WAN speeds to 10G. Since I already have a few and familiar with Ubiquiti products, I decided to give a try to Dream Machine Pro (UDM Pro). So the network became something like this:

You will definitely ask why there is still EdgeRouter 4 there. The main reason is IPv6 RA configuration on UDM Pro is not configurable enough for my need. So although it is probably OK for many people, I was not very happy with it, and this setup is too complex than it needs to be.

10G LAN, 1G WAN

Meanwhile, my ISP, the one and only init7, announced they are going to introduce 10Gbps and 25Gbps (!) connections for end-users at the same price of 1Gbps. I also had in my mind to upgrade the home network to support 10G speeds. So I decided to go with this and opt for 10Gbps WAN. They are rolling out 10G/25G in phases, and while I wait for it, I changed my network to this:

• Bold lines are 10G lines. Blue ones are copper, red one is fiber optic.
• If not particularly mentioned, a normal blue line is a regular Cat 6 / 250 Mhz.

Many changes happened here.

I replaced both EdgeRouter4 and UDM Pro with pfSense running on a Dell T140 (Xeon 3.4Ghz / 32GB RAM). Dell T140 has dual 1G ports, and I also installed a dual 10G (SFP+) ports Intel X710 based NIC. The two 1G blue lines in the figure are for management network and for iDRAC remote management port of T140.

I replaced the main switch (HP 1920) with QNAP QSW-M408-4C. This is maybe a little expensive but a simple nice switch. It offers 8 1G ports, and 4 combo SFP+/1G/2.5G/5G/10G ports. I mainly wanted to have this switch because I did not know if I can reach 10G from my PC (more on this below), so maybe I could use 2.5G or 5G ports. It is a simpler switch than HP 1920 series, but it is fine for me. There are only two features I was using and missing on this one: 1) management VLAN is not configured and it is always VLAN 1, 2) there is no port mirror capability, neither is a blocker.

I installed a single 1G/2.5G/5G/10G port QNAP 10G1T NIC to my PC. I would prefer to have a NIC with Intel (or maybe Mellanox) chipset but they are 2x more expensive and this one supports 2.5G/5G speeds, so I decided to give it a try.

I replaced my NAS with a recent more powerful QNAP TS-932PX model with dual 10G SFP+ ports.

I replaced the cables from my PC to QNAP switch with Cat 7. Naturally I cannot replace the cable inside the wall so it is still Cat 6 / 350 Mhz. I know Cat 6 can theoretically support 10G up to 55m or so, but I was not sure. It seems there is no problem and it can easily support 10G. So it seems actually I did not need Cat 7 (or even 6A).

The network speed from my PC to pfSense tested with iPerf 3 is 9.4 Gbits/s.

The sequential IO performance from my PC to QNAP NAS is roughly:

• for HDDs: 700-800 MB/s read, 300-400 MB/s
• for SDDs: 700-800 MB/s, 500-600 MB/s

It was around 100MB/s with my previous NAS with 1G connection (keep in mind HDDs are same). I tried various setups (RAID-0, 1, 10, 5) and interestingly the results are not very different. Only difference I can clearly see is with SSDs you have better write performance especially random write performance is much better. However, considering the difference with my previous NAS, I think the most important thing is not SSD but having a 10G connection. Because even with HDDs, sequential IO with any RAID configuration is faster than 1G network. I think in my current setup 10G is also the limiting factor, because SSDs have ~500MB/s performance, so in a RAID configuration this can easily pass 10G.

10G LAN, 1G WAN #2 (13.07.2021)

Edit (2022/05/01): I am still waiting for the 10G WAN connection. Meanwhile I have changed the 10G network adapter on my PC to an Intel X550 based 2 ports network adapter.

You probably realized HP 1920 switch above is unnecessary. The reason is I do not have right SFP module yet (waiting for it). When I have it, it will be like this:

It is almost same as before, the only difference is WAN connection goes directly to pfSense, and HP 1920 24 ports switch is removed.

10G LAN, 10G WAN (02.03.2022)

Finally, my WAN connection is upgraded to 10G:

Same as before, only the 1G SFP module is replaced with a 10G SFP+ module (FS.com 10GBASE-BX SFP+ 1270nm-TX/1330nm-RX 10km DOM LC SMF Transceiver Module).

On Speedtest from my PC to init7 (Winterthur), I see over 9Gbps download and upload.

When I have another 10G devices around, I guess I am going to replace the 8 port HP 1920 with another 10G switch like the QNAP I use.

10G LAN, 10G WAN #2 (08.04.2022)

I installed a Proxmox VE virtualization server and need some flexibility with the 10G ports. So I decided to get a QSW-M1208-8C and use that with my PC and the server. I still use the HP 1920 to connect the management and 1G ports. So the network is like this now:

Virtualized pfSense, re-organization (07.11.2022)

I decided to move Proxmox VE virtualization server and move pfSense from the physical server to a VM. Because of this, I also decided to exchange the locations of QNAP 10G switches, so the switch with more 10G ports goes next to Proxmox VE server, network patch panel and where FTTH enters to the apartment. At the moment, I do not need any other ports, so I removed the HP 1920 switch from the setup.

I did not use the Intel X710 interface I was using with the physical pfSense server. Instead, I installed a Chelsio T520-LL-CR interface I had to Proxmox VE server and I use PCIe passthrough to use it solely with pfSense. The server already had two 10G ports, so I am using them for other VMs.

The performance of virtualized pfSense is not bad at all, I cannot say if it performs 100% same as the physical server as I did not make any extensive test, but I can see >9Gbps download/upload speed, so even if it is not same, it is very close.

Virtualized pfSense, re-organization 2 (12.11.2022)

The setup above works pretty OK but the problem was the entry point of fiber connection to my apartment where the Proxmox VE virtualization server was staying is pretty small and enclosed so it was getting too warm even in winter. I have not observed any extra noise or an increase in temperature sensor readings of the server but for the sake of the device, I decided to take it out. The cabling in the home is CAT 6 (but 350 MHz), so I had to terminate it at the switch and then carry the WAN link through CAT 6 to pfSense.

I am still using the Chelsio T520-LL-CR interface for pfSense. I had to deploy HP1920-8G switch this time and I use it mainly for 1G devices, network Management (MGMT) links and for low speed or temporary devices. The SIMATIC X208 in the diagram is the industrial switch I use with my PLC setup.

Media Converter, Intel X540-T2 and X710-BM2, Cisco 1852 (17.11.2022)

In order to free some ports on the switches, I decided to convert the Single Mode Fiber WAN connection to 10GBaseT at the entry so I can directly carry it inside home (since the network cables inside the walls are CAT).

Because now pfSense gets the WAN connection over 10GBaseT, I installed an Intel X540-T2 NIC and replaced Chelsio with the previous NIC I was using, Intel X710-BM2.

Also, I have replaced the Unifi nanoHD WiFi AP with a Cisco 1852. I like nanoHD but I do not like it needs a separate controller software and that software is very integrated to be used with other Unifi products not only Wifi APs. Cisco AP is normally supposed to be used with a separate controller (hardware) but they also have a mode that you can change the software and run standalone (run the controller on the AP), so I started using like that.

Cisco 2960-CG (21.12.2022)

I have been using HP 1920 switches for a very long time. I think they are great but unfortunately they do not get any updates anymore. Because of this and also because I want to try a Cisco, I purchased a second hand Cisco 2960-CG and it replaced HP 1920.

What I learned from this upgrade

• Cat 6 easily supports 10G at home (meaning in short distances).
• DAC cables for 10G are quite nice, cheap and easy to use. They are more expensive than Cat 7 cables but the overall cost depends on the price of the 10G switch also.
• pfSense is very nice, much better than my experience with all consumer products.
• Running pfSense on a host with remote management capability (e.g. Dell iDRAC) is very useful.
• Having a NAS with 10G ports is a very effective upgrade.
• 10G switches are still expensive, and there are limited options. I think, usually ones with RJ-45 (10GBase-TX) ports are more expensive. SFP+ ports are more flexible since you can use fiber, RJ45 or DAC cables, but compatibility with SFP+ modules or DAC cables might be an issue, so it requires more attention. I purchased another QNAP with 12x 10G ports which is the best fit to my environment and also the most cost-effective solution with basic L2 layer management capabilities. (Update: the price of 10GBase-TX switches are decreasing and more alternatives appears. There are now cheaper than QNAP, 8x 10GBase-TX switches.)
• Virtualized pfSense functions very well. If you need a virtualization server at home, it makes sense to run all loads virtualized like this.