After struggling for over 20 hours, I wanted to share the results of my investigation regarding very poor Internet upload erformance.
Setup
- Proxmox Server with a Single 10GbE NIC
- OPNsense VM on Proxmox
- OPNsense uses VirtIO NICs tied to the 10GbE Linux Bridge
- upstream Gateway is a OpenWRT router with 1GbE uplink
- Zyxel XS1930 Switch connecting Proxmox Host and Gateway
Problem
Internet download speeds are fine (900Mbit/s) but upload speeds are not (5-15MBit/s instead of 50MBit/s)
Solution
Various OPNsense tunables (configured for 8 CPU cores)
hw.ibrs_disable = 1net.isr.maxthreads = -1net.isr.bindthreads = 1net.isr.dispatch = deferrednet.inet.rss.enabled = 1net.inet.rss.bits = 6kern.ipc.maxsockbuf = 16777216net.inet.tcp.recvbuf_max = 4194304net.inet.tcp.recvspace = 262144net.inet.tcp.sendbuf_inc = 16384net.inet.tcp.sendbuf_max = 4194304net.inet.tcp.sendspace = 262144net.inet.tcp.soreceive_stream = 1net.pf.source_nodes_hashsize = 1048576net.inet.tcp.mssdflt = 1240net.inet.tcp.abc_l_var = 52net.inet.tcp.minmss = 536kern.random.fortuna.minpoolsize = 128net.isr.defaultqlimit = 2048
Enabling Multiqueue in Proxmox for the VirtIO NICs
(binary stepping, 1 Queue for 2 cores, 2 Queues for 4 cores, 3 Queue for 8 cores ect, total amount of all Queues mustn’t be greater then the VMs CPU cores)
Enabling Flow Control on all involved Network devices
- Proxmox hardware NIC:
ethtool -K nic0 rx on tx on - OpenWRT lan interfnace:
uci set network.lan.txpause='1'
uci set network.lan.rxpause='1'
uci commit
reload_config
- Zyxel Switch:
Port -> Port Setup - Checked all Ports
Enabling Port Buffering
Zyxel Switch:
Port -> Port Buffer - Checked the Port with the Gateway
Reason
The Main reason for this problem seems to be the down-stepping of 10Gbit traffic to 1Gbit devices. Without Flow control enabled on all involved devices, the sending rate can't be adjusted. But without enabling Port Buffering, the Switch won't allocate resources for adjusting the traffic flow rate for slower devices.
This Problem should only affect people who use devices with different link speeds on the same switch.
iperf3 between OPNsense and OpenWRT reached 900Mbit/s, so there was no bottleneck for all kinds of traffic. It only plummeted to 15Mbit/s when there were a lot of small packets to be transmitted. When I ran a speedtest on OPNsense itself, the CPU would hit 100% and I would see 50000+ Interrupts. Because OPNsense and consequently Proxmox were both connected to a 10Gbit/s Switch Port, OPNsense just flooded the OPNsense with as much traffic as possible and without buffering on the switch OPNsense was just drowning in packets. Instead of discarding all packets OpenWRT can't handle, I think it's a way more elegant solution to use Flow Control to throttle the transmission to what the router is capable of. I have a hard time believing that this would affect any other traffic (especially on my LAN), because the 10GbE NIC has 8 Queues which should handle different flows and TX/RX pause packets should only throttle the affected flows. Matter of fact, I just tested and I can still hit consecutive 770Mb/s transfer from my client to my NAS while running a speedtest on the OPNsense (TrueNAS is running on the same Proxmox host as OPNsense). And when disabling flow control all together, I only hit about 750Mb/s transfer speed... So I will stick with my current configuration, as it results in reliable, 0 packet loss transmissions with maximum speeds.