Technology

as someone who used to work on “expert models” i’m excited that not everyone has abandoned them for “what if we just had a model that knows everything (that doesn’t exist) and costs a billion dollars to run”

Yeah...

But you know how people are already comparing vibe coding to 40k where "priests" pray to computers and hope if they do the exact same thing they'll get the same result they want?

If we start walking down this road of even the chat or not understanding why what it did was better....

Serious unintended consequences are going to be inevitable.

Like, I swear nobody knows the paperclip story anymore.

Instrumental convergence posits that an intelligent agent with seemingly harmless but unbounded goals can act in surprisingly harmful ways. For example, a sufficiently intelligent program with the sole, unconstrained goal of solving a complex mathematics problem like the Riemann hypothesis could attempt to turn the Earth (and in principle other celestial bodies) into additional computing infrastructure to succeed in its calculations.[2]

https://en.wikipedia.org/wiki/Instrumental_convergence

I mean, we can make a very very solid argument that much of our current problems are caused by high level stock trading being done by algorithms who's only instruction is "make numbers go up".

This shit aint even hypothetical anymore, it's just instead of "make as many paperclips" we told it "make more money than you did yesterday".

Which is why we're burning down the planet to make billionaires even more money

I was going to ask how this is different than a Reinforcement Learning algorithm but then they called out Deep Minds Alpha-Go