this post was submitted on 16 Sep 2026
70 points (97.3% liked)

Explain Like I'm Five

22330 readers
21 users here now

Simplifying Complexity, One Answer at a Time!

Rules

  1. Be respectful and inclusive.
  2. No harassment, hate speech, or trolling.
  3. Engage in constructive discussions.
  4. Share relevant content.
  5. Follow guidelines and moderators' instructions.
  6. Use appropriate language and tone.
  7. Report violations.
  8. Foster a continuous learning environment.

founded 3 years ago
MODERATORS
i_stole_ur_taco@lemmy.ca
schnurrito@discuss.tchncs.de
70
ELI5: Why do AI web scrapers take down sites? (lemmy.world)
submitted 5 days ago* (last edited 3 days ago) by IndigoGollum@lemmy.world to c/explainlikeimfive@lemmy.world
50 comments fedilink hide all child comments
 

I'd link to some blog posts about this an example, but the site they're from went down a while ago.

At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host's resources beyond being an โŠ› to webmasters?

Edit: Let me elaborate. A lot of the answers i'm seeing are just restating the problem without explaining why LLM scrapers are apparently all either coded by idiots or assholes. I know that they're hitting sites with unreasonable numbers of requests, wasting bandwidth, and making tools like Anubis too important. I know that's what's happening. I'm asking why. What they gain from not being even a little intelligent about this.

With all the money and effort (and maybe even brainpower) going into this, surely there's some explanation beyond incompetence.

you are viewing a single comment's thread
view the rest of the comments
[โ€“] Dran_Arcana@lemmy.world 4 points 4 days ago* (last edited 4 days ago)

The unfortunate answer is because it's cheaper to not give a shit. Sending a request and waiting for a timeout costs next to nothing, and scales linearly in terms of compute cost. The overwhelmed server on the other end slows exponentially with each concurrent request. The crawlers are set to maximize the efficiency of local resources, which include both wall-clock time and developer time. Why send one request at a time when your server can handle tens of thousands?

Try x; wait 60 seconds, if fail: put on a list to try again later.

Costs nothing to write and nothing to run. And if you own the hardware, and are paying for power already, may as well extract maximum dollar per watt.