this post was submitted on 16 Sep 2026
70 points (97.3% liked)
Explain Like I'm Five
22330 readers
21 users here now
Simplifying Complexity, One Answer at a Time!
Rules
- Be respectful and inclusive.
- No harassment, hate speech, or trolling.
- Engage in constructive discussions.
- Share relevant content.
- Follow guidelines and moderators' instructions.
- Use appropriate language and tone.
- Report violations.
- Foster a continuous learning environment.
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I'm just correcting you architectural definition. No crawler needs to vertically scale as you define it here and in fact it would be a worse tech for "slurp all data" goal.
The real answer is much more simple - incompetence. The inability to write a good cralwer by leaders in software tech is much damning than anything else. It's like leading car manufacturers not being able to make the wheels round - it's just sad.
However this actually shows a much more important problem with the web - we have no way to defer or distribute negative externalities and people have been trying to solve this since the inception of the internet itself. It'll never be solved but minimized to the point where the impact is just something you tolerate because it's simply impossible to solve and anyone who tells you otherwise is selling you something.
Source: I've been working on similar problems for the last 20 years at least.
You’re not correcting anything, dude.
Whatever, stew in your ignorance then. Bye.
Been stewing at it for longer than you have, buddy.