cross-posted from: https://lemmy.nocturnal.garden/post/387129
Hi, I've had issues for the last days where my services were unreachable via their domains sporadically. They are scattered across 2-3 VMs which are working fine and can be reached by their domain (usually x.my.domain subdomains) via my nginx reverse proxy (running in it's own Debian vm). The services themself were running fine. My monitoring (Node Exporter/Prometheus) notified me that the conntrack limit on the nginx vm was reached in the timeframes where my services weren't reachable, so that seems to be the obvious issue.
As for the why, it seems that my domains are known to more spammers/scripters now. The nginx error.log grew by factor 100 from one day to the next. Most of my services are restriced to local IPs, but some like this lemmy instance are open entirely (nginx vm has port 80 and 443 forwarded).
I never heard of conntrack before but tried to read up on it a bit. It keeps track of the vm's connections. The limit seems to be rather low, apparently it depends on the memory of the vm which is also low. I can increase the memory and the limit, but some posts suggest to generally disable it if not stricly needed. The vm is doing nothing but reverse proxying so I'm not sure if I really need it. I usually stick to Debians defauls though. Would appreciate input on this as I don't really see what the conseqences of this would be. Can it really just be disabled?
But that's just making symptons go away and I'd like to stop the attackers even before reaching the vm/nginx. I basically have 2 options.
- The vm has ufw enabled and I can set up fail2ban (should've done that earlier). However, I'm not sure if this helps with the conntrack thing since they need to make a connection before getting f2b'd and that will stay in the list for a bit.
- There's an OPNsense between the router and the nginx vm. I have to figure out how, but I bet there's a possibility to subscribe to known-attacker-IP-lists and auto-block or the like. I'd like some transparency here though and also would want to see which of the blocked IPs actually try to get in.
Would appreciate thoughts or ideas on this!
Connection tracking might not be totally necessary for a reverse proxy mode, but it's worth discussing what happens if connection tracking is disabled or if the known-connections table runs out of room. For a well-behaved protocol like HTTP(S) that has a fixed inbound port (eg 80 or 443) and uses TCP, tracking a connection means being aware of the TCP connection state, which the destination OS already has to do. But since a reverse proxy terminates a TCP connection, then the effort for connection tracking is minimal.
For a poorly-behaved protocol like FTP -- which receives initial packets in a fixed inbound port but then spawns a separate port for outbound packers -- the effort of connection tracking means setting up the firewall to allow ongoing (ie established) traffic to pass in.
But these are the happy cases. In the event of a network issue that affects an HTTP payload sent from your reverse proxy toward the requesting client, a mid-way router will send back to your machine an ICMP packet describing the problem. If your firewall is not configured to let all ICMP packets through, then the only way in would be if conntrack looks up the connection details from its table and allows the ICMP packet in, as "related" traffic. This is not dissimilar to the FTP case above, but rather than a different port number, it's an entirely different protocol.
And then there's UDP tracking, which is relevant to QUIC. For hosting a service, UDP is connectionless and so for any inbound packet we received on port XYZ, conntrack will permit an outbound packet on port XYZ. But that's redundant since we presumably had to explicitly allow inbound port XYZ to expose the service. But in the opposite case, where we want to access UDP resources on the network, then an outbound packet to port ABC means conntrack will keep an entry to permit an inbound packet on port ABC. If you are doing lots of DNS lookups (typically using UDP), then that alone could swamp the con track table: https://kb.isc.org/docs/aa-01183
It may behoove you to first look at what's filling conntrack's table, before looking to disable it outright. It may be possible to specifically skip connection tracking for anything already explicitly permitted through the firewall (eg 80/443). Or if the issue is due to numerous DNS resolution requests from trying to look up spam sources IPs, then perhaps either the logs should not do a synchronous DNS lookup, or you can also skip connection tracking for DNS.
Thanks for your answer, pretty much what I've been looking for I think