this post was submitted on 16 Sep 2026
70 points (97.3% liked)

Explain Like I'm Five

22330 readers
21 users here now

Simplifying Complexity, One Answer at a Time!

Rules

  1. Be respectful and inclusive.
  2. No harassment, hate speech, or trolling.
  3. Engage in constructive discussions.
  4. Share relevant content.
  5. Follow guidelines and moderators' instructions.
  6. Use appropriate language and tone.
  7. Report violations.
  8. Foster a continuous learning environment.

founded 3 years ago
MODERATORS
 

I'd link to some blog posts about this an example, but the site they're from went down a while ago.

At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host's resources beyond being an ⊛ to webmasters?

Edit: Let me elaborate. A lot of the answers i'm seeing are just restating the problem without explaining why LLM scrapers are apparently all either coded by idiots or assholes. I know that they're hitting sites with unreasonable numbers of requests, wasting bandwidth, and making tools like Anubis too important. I know that's what's happening. I'm asking why. What they gain from not being even a little intelligent about this.

With all the money and effort (and maybe even brainpower) going into this, surely there's some explanation beyond incompetence.

you are viewing a single comment's thread
view the rest of the comments
[–] leds@feddit.dk 6 points 4 days ago (1 children)

To lock all knowledge in their models as the onky source, destroy independent websites , destroy physical books after scanning , destroy peoples brains

[–] grue@lemmy.world 4 points 4 days ago (1 children)

They're not destroying the books "after" scanning 'for the evulz' to deprive the world of them. They're destroying them before scanning because cutting the spines off with a bandsaw lets them drop the stack of pages in a sheet-feed scanner and get the job done a lot faster than using a book scanner.

Similarly, I'm sure they've got some actual rationale for their crawler behavior that makes sense (at least to them) and isn't a conspiracy theory.

Nobody is a mustache-twirling villain in their own mind, even if they turn out to be so from everybody else's perspective.

[–] Natanael@infosec.pub 2 points 4 days ago

No they're just stupid when it comes to the crawlers. They have VC money and no expectations to behave well. They want to capture everything they can as fast as possible because the race to build up datasets yup use in training is the only real target they have