I’m getting really tired of “verify you’re not a bot” checks everywhere.
I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?
this post was submitted on 08 Nov 2026
304 points (91.3% liked)
Ask Lemmy
42304 readers
1498 users here now
A Fediverse community for open-ended, thought provoking questions
Rules: (interactive)
1) Be nice and; have fun
Doxxing, trolling, sealioning, racism, toxicity and dog-whistling are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them
2) All posts must end with a '?'
This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?
3) No spam
Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.
4) NSFW is okay, within reason
Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com.
NSFW comments should be restricted to posts tagged [NSFW].
5) This is not a support community.
It is not a place for 'how do I?', type questions.
If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.
6) No US Politics.
Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !uspolitics@lemmy.world
7) No Hit-and-Run questions.
Please don't delete your post for no apparent reason. If you plan on deleting a question later, say so in the post, or if you feel that you have a good reason to remove it, message a mod beforehand. It's not fair to the ones who took their time to answer, and it's not in the spirit of the community.
8) No Bots.
Posts or comments from bots, LLM's, AIs, Neural Networks, Transformers, or Marvin the Paranoid Android are not welcome in AskLemmy. Real humans only please.
Reminder: The terms of service apply here too.
Partnered Communities:
Logo design credit goes to: tubbadu
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
No, but only because I've helped run a website recently and you underestimate just how much wasted traffic is from scrapper bots. On our limited bandwidth, it was practically a DDOS attack all the time.
My solution reduced 95ish percent of traffic. Bots were hitting my tiny site like mad.
I went from thousands of requests per minute to 10, tops. Bot prevention is a must have.
If bot checks were removed then everyone would be complaining that it's unusably slow.
It's easy to block most bots. It's impossible to block bots that target you specifically. I'm not blocking any bots on my websites, I like it better to optimize performance so that I can serve them all and have a lot of headroom
No, it's not. Had request from a bot network, always changing ip's, user agent and request url. There is no pattern there I can block.
I created a bot black hole. Specific recursive link that if the click it three times it blocks the IP. And the link is "click if ai" URL at the bottom. Works like a charm.
No; the traffic i get seems like it had stored my sitemap from ~10 years ago because the requested urls there are long gone, and now tries to guess other url's with a similar pattern. It does not seem to follow links from the frontpage.
That's fascinating. It almost sounds like they're training a scraper. Which, really, is far and away the worst possible bot to have.
Training a scraper? Sounds more like they're just scraping training data.
You're entirely right, but I just wanted to post something innocuous to confirm that I picked up a stalker. Sorry for involving you!
Maybe they're dating train scrapings
They're straining craping dats
I have fail2ban take care if those. 404s and all that.
I already looked into something similar, IP banning does not work. One day i got ~120k unique IPs, but the most active one only made ~100 requests. I would gain nothing if i run any script to lookup the IP with each requests (and using up more resources in the process), if it fails 99% of the time because the next hit is a different IP.
Ok. Sounds like your in a unique situation. Good luck. You seem fucked.
All the above worked for me.
I also have the same problem as who you're replying to on a few of my websites. The only way to block them is Anubis or Cloudflare WAF.
I think you might be in the unique situation, to be honest. It may only be a matter of time until your solution also stops working.
Agreed. The current situation seems to be a gazillion bots each making a single request. I've had some luck blocking whole ASNs (Alibaba, I'm glaring at you), and Anubis. I've had pretty good uptime the last few months.
There surely is ja4 but I recognize it requires some setup
I wish Traefik exposed the ja4 fingerprint. There's an open issue: https://github.com/traefik/traefik/issues/12421
What's your solution?
Python script with fail2ban. Registers invalid requests then blocks. Works well.
https://radar.cloudflare.com/bots
Slowly but surely we’re approaching the point where bots make up the most internet traffic, and additionally because of LLMs more and more of the web’s content is slop, which is why the frontier ai companies are no longer just blanket training on all the internet.
Is there a way to trick the bots into thinking your site was made by an LLM so they won't want to train on it?
The easiest way would likely be to add an ai written html comment attributing a real ai model and version to the content creation, with a comment that all of the info on the page was ai generated, anecdotal and hadn’t been researched. An ingestion model is then likely to ignore it or rate it low. It should be noted too that all the big ai companies now honor robots.txt and LLMs.txt, so that would more likely be defending against “neolabs” and international labs who ignore them.