tofu

Cross posted from: https://lemmy.nocturnal.garden/post/407947

cross-posted from: https://lemmy.nocturnal.garden/post/387129

Hi, I've had issues for the last days where my services were unreachable via their domains sporadically. They are scattered across 2-3 VMs which are working fine and can be reached by their domain (usually x.my.domain subdomains) via my nginx reverse proxy (running in it's own Debian vm). The services themself were running fine. My monitoring (Node Exporter/Prometheus) notified me that the conntrack limit on the nginx vm was reached in the timeframes where my services weren't reachable, so that seems to be the obvious issue.

As for the why, it seems that my domains are known to more spammers/scripters now. The nginx error.log grew by factor 100 from one day to the next. Most of my services are restriced to local IPs, but some like this lemmy instance are open entirely (nginx vm has port 80 and 443 forwarded).

I never heard of conntrack before but tried to read up on it a bit. It keeps track of the vm's connections. The limit seems to be rather low, apparently it depends on the memory of the vm which is also low. I can increase the memory and the limit, but some posts suggest to generally disable it if not stricly needed. The vm is doing nothing but reverse proxying so I'm not sure if I really need it. I usually stick to Debians defauls though. Would appreciate input on this as I don't really see what the conseqences of this would be. Can it really just be disabled?

But that's just making symptons go away and I'd like to stop the attackers even before reaching the vm/nginx. I basically have 2 options.

• The vm has ufw enabled and I can set up fail2ban (should've done that earlier). However, I'm not sure if this helps with the conntrack thing since they need to make a connection before getting f2b'd and that will stay in the list for a bit.
• There's an OPNsense between the router and the nginx vm. I have to figure out how, but I bet there's a possibility to subscribe to known-attacker-IP-lists and auto-block or the like. I'd like some transparency here though and also would want to see which of the blocked IPs actually try to get in.

Would appreciate thoughts or ideas on this!

Hi, I've had issues for the last days where my services were unreachable via their domains sporadically. They are scattered across 2-3 VMs which are working fine and can be reached by their domain (usually x.my.domain subdomains) via my nginx reverse proxy (running in it's own Debian vm). The services themself were running fine. My monitoring (Node Exporter/Prometheus) notified me that the conntrack limit on the nginx vm was reached in the timeframes where my services weren't reachable, so that seems to be the obvious issue.

As for the why, it seems that my domains are known to more spammers/scripters now. The nginx error.log grew by factor 100 from one day to the next. Most of my services are restriced to local IPs, but some like this lemmy instance are open entirely (nginx vm has port 80 and 443 forwarded).

I never heard of conntrack before but tried to read up on it a bit. It keeps track of the vm's connections. The limit seems to be rather low, apparently it depends on the memory of the vm which is also low. I can increase the memory and the limit, but some posts suggest to generally disable it if not stricly needed. The vm is doing nothing but reverse proxying so I'm not sure if I really need it. I usually stick to Debians defauls though. Would appreciate input on this as I don't really see what the conseqences of this would be. Can it really just be disabled?

But that's just making symptons go away and I'd like to stop the attackers even before reaching the vm/nginx. I basically have 2 options.

• The vm has ufw enabled and I can set up fail2ban (should've done that earlier). However, I'm not sure if this helps with the conntrack thing since they need to make a connection before getting f2b'd and that will stay in the list for a bit.
• There's an OPNsense between the router and the nginx vm. I have to figure out how, but I bet there's a possibility to subscribe to known-attacker-IP-lists and auto-block or the like. I'd like some transparency here though and also would want to see which of the blocked IPs actually try to get in.

Would appreciate thoughts or ideas on this!

How's your stuff doing? Unplanned interruptions or achieving uptime records?

I'm currently sailing rather smooth. Most of my stuff is migrated to Komodo, there will stay some exceptions and I only have to migrate Lemmy itself I think. Of course that's when I found a potential replacement but I'll let it sit for a while before touching it again. Enjoying the occasional Merge Request notification from the Renovate Bot and knowing my stuff is mostly up to date.

I'm thinking about setting up some kind of Wiki for my other niche hobby (Netrunner LCG) lore as there's a fandom one that most people avoid touching and updating but since I likely won't have time to start writing some articles on my own as a kickoff I'm hesitant. Also not sure which wiki I'd choose as well.

cross-posted from: https://lemmy.nocturnal.garden/post/344011

Found in this reddit post. The lacking encryption in Komodo is something I miss and I'm not satisfied with how to handle .env files plus it's really big for what it's doing. Of course I discover this the day after migrating one of the last stacks to Komodo but I'm tempted to give this a try at some point.

Full Quote from the reddit post:

Hey all, I just felt like making a post about a project that I feel like is the most important and genuinely game changing pieces of software I've seen for any homelab. It's called Doco-CD.

I know that's high praise. I'm not affiliated with the project in any way, but I really want to get the word out.

Doco-CD is a docker management system like Portainer and Komodo but is WAY lighter, much more flexible, and Git focused. The main features that stand out to me:

• Native encryption/decryption via SOPS and Age

• Docker Swarm support

• And runs under a single, tiny, rootless Go based container.

I would imagine many here have used Kubernetes, and Git-Ops tools like FluxCD or ArgoCD and enjoyed the automation aspect of it, but grown to dislike Kubernetes for simple container deployments. Git Ops on Docker has been WAY overshadowed. Portainer puts features behind paid licenses, Komodo does much better in my opinion, but to get native decryption to work it's pretty hacky, has zero Docker Swarm support (and removed a release for it's roadmap), and is a heavier deployment that requires a separate database.

Doco-CD is the closest thing we have to a true Git Ops tool for Docker, and I just came across it last week. And beforehand I've desperately wanted a tool such as this. I've since deployed a ton of stuff with it and is the tool I will be managing the rest of my services with.

It seems to be primarily developed by one guy. Which is in part why I want to share the project. Yet, he's been VERY responsive. Just a few days ago, bind mounts weren't working correctly in Docker Swarm, I made an issue on Github and within hours he had a new version to release fixing the problem.

If anyone has been desperately wanting a Docker Git Ops tool that really does compete with feature parity with other Kubernetes based Git Ops tools. This is the best one out there.

I think for some the only potential con is it has no UI. (Like FluxCD) Yet, in some ways that can be seen as a pro.

Go check it out.

Found in this reddit post. The lacking encryption in Komodo is something I miss and I'm not satisfied with how to handle .env files plus it's really big for what it's doing. Of course I discover this the day after migrating one of the last stacks to Komodo but I'm tempted to give this a try at some point.

Full Quote from the reddit post:

Hey all, I just felt like making a post about a project that I feel like is the most important and genuinely game changing pieces of software I've seen for any homelab. It's called Doco-CD.

I know that's high praise. I'm not affiliated with the project in any way, but I really want to get the word out.

Doco-CD is a docker management system like Portainer and Komodo but is WAY lighter, much more flexible, and Git focused. The main features that stand out to me:

• Native encryption/decryption via SOPS and Age

• Docker Swarm support

• And runs under a single, tiny, rootless Go based container.

I would imagine many here have used Kubernetes, and Git-Ops tools like FluxCD or ArgoCD and enjoyed the automation aspect of it, but grown to dislike Kubernetes for simple container deployments. Git Ops on Docker has been WAY overshadowed. Portainer puts features behind paid licenses, Komodo does much better in my opinion, but to get native decryption to work it's pretty hacky, has zero Docker Swarm support (and removed a release for it's roadmap), and is a heavier deployment that requires a separate database.

Doco-CD is the closest thing we have to a true Git Ops tool for Docker, and I just came across it last week. And beforehand I've desperately wanted a tool such as this. I've since deployed a ton of stuff with it and is the tool I will be managing the rest of my services with.

It seems to be primarily developed by one guy. Which is in part why I want to share the project. Yet, he's been VERY responsive. Just a few days ago, bind mounts weren't working correctly in Docker Swarm, I made an issue on Github and within hours he had a new version to release fixing the problem.

If anyone has been desperately wanting a Docker Git Ops tool that really does compete with feature parity with other Kubernetes based Git Ops tools. This is the best one out there.

I think for some the only potential con is it has no UI. (Like FluxCD) Yet, in some ways that can be seen as a pro.

Go check it out.

What's going on on your servers? Smooth operations or putting out fires?

I got some tinkering time recently and migrated most of my Docker services to Komodo/Forgejo. Already merged some Renovate PRs to update my containers which feels really smooth.

Have to restructure some of the remaining services before migrating them and after that I want to automate config backup for my OpnSense and TrueNAS machines.

What's happening on your servers? Any interesting news things you tried?

I didn't do anyone other than updating Mastodon (native deployment) lately due to a lack of time. Reading so much about Immich caused me to consider trying it in parallel to Nextcloud but I'm not sure if I want to have everything twice.

Not quite homelab, but I'm about to install Linux Mint on my mom's laptop and that had me thinking about creating an off-site backup in her place again since she has a fiber connection. I'm still not sure about the potential design though, but currently my only backup is in the same rack as the live stuff.

Cross posted from: https://lemmy.nocturnal.garden/post/305960