this post was submitted on 16 Sep 2026
70 points (97.3% liked)
Explain Like I'm Five
22330 readers
21 users here now
Simplifying Complexity, One Answer at a Time!
Rules
- Be respectful and inclusive.
- No harassment, hate speech, or trolling.
- Engage in constructive discussions.
- Share relevant content.
- Follow guidelines and moderators' instructions.
- Use appropriate language and tone.
- Report violations.
- Foster a continuous learning environment.
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Everybody in this thread is talking about what they're doing, but not a single reply has been able to explain why the bad behavior somehow benefits the companies doing it.
I don't believe the only reason is incompetence; there's got to be somehing else to it.
To lock all knowledge in their models as the onky source, destroy independent websites , destroy physical books after scanning , destroy peoples brains
They're not destroying the books "after" scanning 'for the evulz' to deprive the world of them. They're destroying them before scanning because cutting the spines off with a bandsaw lets them drop the stack of pages in a sheet-feed scanner and get the job done a lot faster than using a book scanner.
Similarly, I'm sure they've got some actual rationale for their crawler behavior that makes sense (at least to them) and isn't a conspiracy theory.
Nobody is a mustache-twirling villain in their own mind, even if they turn out to be so from everybody else's perspective.
No they're just stupid when it comes to the crawlers. They have VC money and no expectations to behave well. They want to capture everything they can as fast as possible because the race to build up datasets yup use in training is the only real target they have
The unfortunate answer is because it's cheaper to not give a shit. Sending a request and waiting for a timeout costs next to nothing, and scales linearly in terms of compute cost. The overwhelmed server on the other end slows exponentially with each concurrent request. The crawlers are set to maximize the efficiency of local resources, which include both wall-clock time and developer time. Why send one request at a time when your server can handle tens of thousands?
Try x; wait 60 seconds, if fail: put on a list to try again later.
Costs nothing to write and nothing to run. And if you own the hardware, and are paying for power already, may as well extract maximum dollar per watt.
I don't know, man. Almost everything in the last 10 years can be explained simply by incompetence.
No, quite a lot in the last 10 years has very much been caused by hate, malice, and greed.
It's 100% a mix of various incompetences. I develop anti fraud systems and deal with cralwers all the time. The web is actually really complex, probably the most complex technology in history of human kind and I'm not even kidding.
It's not incompetence. They just don't care, the same way they stole all the books from the internet.