this post was submitted on 16 Sep 2026
70 points (97.3% liked)

Explain Like I'm Five

22330 readers
21 users here now

Simplifying Complexity, One Answer at a Time!

Rules

  1. Be respectful and inclusive.
  2. No harassment, hate speech, or trolling.
  3. Engage in constructive discussions.
  4. Share relevant content.
  5. Follow guidelines and moderators' instructions.
  6. Use appropriate language and tone.
  7. Report violations.
  8. Foster a continuous learning environment.

founded 3 years ago
MODERATORS
i_stole_ur_taco@lemmy.ca
schnurrito@discuss.tchncs.de
70
ELI5: Why do AI web scrapers take down sites? (lemmy.world)
submitted 5 days ago* (last edited 3 days ago) by IndigoGollum@lemmy.world to c/explainlikeimfive@lemmy.world
50 comments fedilink hide all child comments
 

I'd link to some blog posts about this an example, but the site they're from went down a while ago.

At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host's resources beyond being an ⊛ to webmasters?

Edit: Let me elaborate. A lot of the answers i'm seeing are just restating the problem without explaining why LLM scrapers are apparently all either coded by idiots or assholes. I know that they're hitting sites with unreasonable numbers of requests, wasting bandwidth, and making tools like Anubis too important. I know that's what's happening. I'm asking why. What they gain from not being even a little intelligent about this.

With all the money and effort (and maybe even brainpower) going into this, surely there's some explanation beyond incompetence.

you are viewing a single comment's thread
view the rest of the comments
[–] drmoose@lemmy.world 1 points 4 days ago (1 children)

This is incorrect as you don't need vertical scaling for broad crawling which by definition is a horizontal operation. So you can slurp the web with a per domain throttle that has a respectful limit. The only pressure is browser cache time as if you're crawling a domain you want to take advantage of as much cache as possible but modern websites have very long cache times, like days.

So it's mostly incompetence. Crawling is actually really hard because web is just very complex.

[–] DecorativeTarp@lemmy.zip 1 points 3 days ago* (last edited 3 days ago) (1 children)

The entire point here is that the bots are being disrespectful.

[–] drmoose@lemmy.world 1 points 3 days ago (1 children)

I'm just correcting you architectural definition. No crawler needs to vertically scale as you define it here and in fact it would be a worse tech for "slurp all data" goal.

The real answer is much more simple - incompetence. The inability to write a good cralwer by leaders in software tech is much damning than anything else. It's like leading car manufacturers not being able to make the wheels round - it's just sad.

However this actually shows a much more important problem with the web - we have no way to defer or distribute negative externalities and people have been trying to solve this since the inception of the internet itself. It'll never be solved but minimized to the point where the impact is just something you tolerate because it's simply impossible to solve and anyone who tells you otherwise is selling you something.

Source: I've been working on similar problems for the last 20 years at least.

[–] DecorativeTarp@lemmy.zip 0 points 3 days ago (1 children)

You’re not correcting anything, dude.

[–] drmoose@lemmy.world 1 points 3 days ago (1 children)

Whatever, stew in your ignorance then. Bye.

[–] DecorativeTarp@lemmy.zip 1 points 3 days ago* (last edited 3 days ago)

Been stewing at it for longer than you have, buddy.