Technology

Humanity should be way more frightened of greedy psychopaths running the world than any technology.

In addition to what everyone else said, I'll tell you why I don't think that what we have currently can become some runaway superhuman intelligence:

All “AI” people are talking about are basically “just” LLM wrappers: tools built to run a LLM in a loop while giving it access to the ability to write and run code. This gets pretty powerful, since you can do a lot programmatically (including monetary transactions), but has a fundamental limit:

LLMs are trained ahead of time. And that's a very cost intensive, long process. Beyond that, the LLM has only a context window of finite length. So there are no long-term planning capabilities. While running after being trained, the LLM can only condense knowledge into a summary and then re-read it, a lossy process that might introduce mistakes in “understanding” and will make any prolonged task degrade over time.

And even if they figure out how to make the things actually self-modify and learn things, you can't give every session that ability, since that'll vastly increase the storage and performance requirements. Do sessions with that ability will be closely monitored and limited.

So I don't see any space where an evil AI could emerge, even when factoring in an essential possible future improvement to the technology.

Ed Zitron’s podcast “Better Offline” recently talked about this. There is a small tech community called the Rationalists who believe AI super intelligence is inevitable and it will either accelerate humanity to new heights or kill off the species. These rationalists are actually a very prominent faction within tech and the AI industry. Apparently the language used within top AI firms is largely about the immense danger posed by AI super intelligence if any other group or company were to achieve it first. This creates an echo chamber that makes the people in the industry kind of lose their mind and misinterpret the capability of LLMs.

Everyone outside that faction recognizes that LLMs are just unreliable cloud software with limited use cases and that there’s no evidence LLMs could actually lead to any kind of intelligence.

Edit: Please, someone stop me from commenting on an AI post next time I’m tempted. I’m so tired of people replying, arguing with opinions I don’t have, based entirely around a hypothetical technology that doesn’t exist yet. LLMs make slop. Thats it. They hacked a company after weeks of 100s of instances being run, using old exploits published online. San Francisco will be underwater from climate change before LLMs can do anything more.

I'm more worried about the people using AI than the tech itself.

Oh how I would love for AI to actually start rapidly self-improving.

I'm not pro-AI by any stretch for a huge number of reasons, but in my day job as a technical product manager I have to use a number of AI-powered tools and workflows. Whereas 18 months ago they were a novelty that the bosses were enamored of, today they get real work done and speed up a number of key areas, mainly like fancy automation workflows (which have been around forever). The rate of improvement and capability for certain use cases has been very, very rapid. We do seem to be reaching a plateau though. The tools and models from 6 months ago are just about as good as the ones from today. And China's models from today are better than the US's models from 6 months ago, but at a fraction of the cost.

So when I hear people talking about the "end of the world", I think a realistic case for that is that the AI tools that come out in the next few years will be good enough to do a fair amount of white-collar work, and we won't have a plan in place to deal with all of the now out-of-work people that leads to. Whole classes of jobs in areas like data entry, telemarketing, and customer support will be almost entirely automated out of existence. New classes of jobs will be created, of course, but it's hard to imagine them existing in sufficient numbers to make up for the losses. And this is just for LLM-workloads.

In other areas, other types of AI are already enabling robotic automation of factories and in the near-ish future might make self-driving truck fleets practical. This would be devastating because factory workers and truck drivers make up like 30% of the US workforce -- and again, there are no signs of new classes of jobs that could absorb the out-of-work workers, and no plans for a social safety net or wealth distribution.

So I'm not worried about a rogue AI unleashing some bioweapon or launching a nuclear attack. I'm worried about a bunch of CEOs deploying barely-acceptable AI to displace millions of workers as soon as it is safe(ish) to do so.

I feel like the AI hype has died and people are trying to prop it up with doomsday scaring.

These people are too high on their own supplies.