this post was submitted on 14 Aug 2025
305 points (97.8% liked)

Technology

74003 readers
3418 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots


founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] drmoose@lemmy.world 16 points 21 hours ago (4 children)

As someone who's been on the web since the 90s I hate this.

The web was designed to be user agent agnostic. Desktop, phone, fridge, ai agents, curl, python script - whatever agent you are using shouldn't matter for access. That's the whole point of open internet, period.

[–] Allero@lemmy.today 22 points 19 hours ago* (last edited 19 hours ago) (1 children)

When the Web was first designed, some of the concerns we have today were nonexistent.

I believe in freedom of information, and would love for the information I share to be accessed in any way a given user wants.

But I have to stand defensive and support the author here, too. The modern LLM boom aims to essentially replace original resources with AI-generated summaries step by step. This is detrimental to the Internet, and to knowledge as we know and preserve it.

First, there is an event commonly called Google Zero, which is briefly mentioned in the article. If you don't know what it is, it is the not-so-hypothetical-anymore moment when Google (or, really, any other large player) essentially accumulates all information on the Web, feeds it to AI, and since then doesn't serve links anymore, going straight to answers based on training data. Users will jump to this - they already do - because it offers convenience. But for any independent creators it means having no audience, no money, and no means to produce new quality content, trapping users in a self-containing loop that loses nuance, actuality, and truthfulness, and stays under corporate control. This goes beyond cooking recipes and personal notes - it permeates science, political discussion, and much more.

Second, LLMs multiply traffic coming to sites, which becomes an infrastructural problem. Bots access sites at much higher rates than humans do, and when their intent is to scrape your entire website every now and again and there are dozens of them, this becomes huge.

Third, having proprietary models train on the data I provide without any attribution, copyright etc. makes giant corporation profit off my back, while at the same thing making it so that less genuine users will see what I produce. This means careers of authors, journalists etc. are dying, and this also means they are left free to abuse each and every one of us without any consent.

Fourth, and I wonder if you see it by now, LLMs and the way they represent data, along with SEO tools meant to drive information through the search bots, begin to shape how we talk. All I say doesn't have to be a list of points, yet it is. It could be less verbose, more readable, yet it is the way it is. Because when we interact with the products of such developments too much, we begin shaping our own language in a way that is less human-readable and more meant for machines, without us often being aware of it. This is a real issue of communication.

So, as much as I hate it, I'm gonna protect a lot of the data I share.

[–] frezik@lemmy.blahaj.zone 5 points 16 hours ago (1 children)

Instructions unclear, built whole site with nested tables.

[–] caseyweederman@lemmy.ca 1 points 56 minutes ago

Each one had better be in its own iframe.

[–] whyNotSquirrel@sh.itjust.works 12 points 20 hours ago (2 children)

Open until your server is down because LLM are overloading it

[–] kescusay@lemmy.world 13 points 19 hours ago (1 children)

At my company, we had to implement all sorts of WAF rules precisely for that reason. Those things are fucking aggressive.

[–] bravesilvernest@lemmy.ml 7 points 18 hours ago

Same. And just because page size is "low" doesn't mean shit when they're flooding requests. Try having public research data and watch how much your costs go up just due to load balancer throughput.

[–] JustARaccoon@lemmy.world 8 points 20 hours ago (1 children)

They did have a lot of concerns with abuse though and you can see that in the way the cookies debate went before they were supported in their current form. I think AI crawlers tanking bandwidths for websites and misusing the data they scrape would 100% be something the Mozilla from back then would've had concerns over allowing or encouraging.

[–] drmoose@lemmy.world 1 points 19 hours ago (1 children)

You're conflating two different issues. The topic is "for whom the web is for?" not banwidth distribution and optimization.

If LLM bot is being abusive then that's no different from any other user agent behaving like this and we should expand these protections from intentional/unintentional ddos irrelevant of user agent.

[–] ernest314@lemmy.zip 4 points 18 hours ago* (last edited 18 hours ago)

I think your starting point (allowing bot user agents to crawl the web has overlooked benefits) is a good one, but things aren't black and white--there are clear drawbacks, too. Bots obviously have an orders of magnitude higher potential for abuse; to the point where bot traffic--as it currently stands in the real world--is qualitatively different from human traffic.

we should expand these protections from intentional/unintentional ddos irrelevant of user agent.

Sure, but targeted regulation based on heuristics (in this case, user agent) is also a widely accepted practice. DUI laws exist, even though the goals (fewer murders and safer roads) are already separately regulated.

Would it be nice if we didn't have to do this? Or there were some other solution? Sure, but I have no idea where to even start, unfortunately.