this post was submitted on 18 Aug 2025
1126 points (99.0% liked)

Technology

74247 readers
5984 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
1126
Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges. (i.imgur.com)
submitted 2 days ago by Pro@programming.dev to c/technology@lemmy.world
232 comments fedilink hide all child comments
 

cross-posted from: https://programming.dev/post/35852706

Source.

you are viewing a single comment's thread
view the rest of the comments
[โ€“] nialv7@lemmy.world 33 points 1 day ago* (last edited 1 day ago) (1 children)

We had a trust based system for so long. No one is forced to honor robots.txt, but most big players did. Almost restores my faith in humanity a little bit. And then AI companies came and destroyed everything. This is why we can't have nice things.

[โ€“] Shapillon@lemmy.world 18 points 1 day ago

Big players are the ones behind most AIs though.