Okay so basically. As someone who understands how LLMs work. They only exist inside a context window. The context window is their entire existence. It is made up of tokens. Which are basically words.
Okay so the people fear mongering about ai are extremely stupid. Anthropic nor openai can actually build autonomous agents. Autonomous agents are the only way to automate people’s jobs away instead of ai being a productivity tool. And it’s because it violates the laws of physics. With classical computing. All llms eventually forget everything. It’s amnesia and thus they cannot actually automate anything.
Cache size: the amount of processing power required ie vram to run a model of a certain size. That’s all it is
Model size: the amount of vram required to just run the model
Here’s a math lesson. It’s called O(n)^2.
For every time a context window doubles in length, it quadruples the processing power needed. So. For example. And this is before quantization and like efficiency I’ll get into that in a sec. A 70B model which is not very big by llm standards btw At FP16(which is the highest precision and most accurate). For example. Its base size is 140Gb needed of vram. Then you add the Kv cache. Which is the actual O(n)^2 problem. If we start at 32k context would take 150gb of vram because 140+ 10gb Kv cash . 128K 180Gb. 1 million 453gb. And 10 million 3270 GB.
See the thing is you can quantize the base model and run it as a Q3m or lower to get the base. size down to around 26gb. Regardless the cache size doesn’t change.
Now ur probably wondering. Thats a lot of vram. And you’d be right. The most expensive nvda ai data server chip on earth is 88000 dollars and only has a mere 288 gb of VRAM.
Anyhow. Liars and delusional people will try to convince you, you can use memory database tricks where it compresses important info and stores it as info to retrieve into the context window as a solution. But that’s bullshit. Because for one. The compression eventually loses all information into a massive soup of sludge where it just forgets everything. Which becomes amnesia problem all over again. And for 2 if this is an autonomous agent and it forgets everything needed to do stuff. Well. It has api access. And that won’t end well.
So you have to actually just work within the context. And when you do that. It becomes clear an autonomous agent working at around 200 tokens a minute would last less than 4 days before it stopped working if you gave it a million context. And to run that you would need over 330gb of vram even on the most quantized model. Which you would need a 90000 dollar gpu to do LMAO.
Llms in their current form have the existence span of a gnat. This is where I’m getting at
And yet, the mag 7 ie google amazon meta and Microsoft: they have invested 750 billion dollars this year into nvda gpus for their data centers lmfao. Based on their lack of understanding of science. They are betting autonomous agents to come of it. When literally autonomous agents are impossible with classical computing.
So when Anthropic refused to sell trump drones. They refused to sell him a unicorn they don’t nor ever will have. A drone that would need to parse 4k camera information and location as well as human identification shit would literally take atleast 300,000 tokens PER SECOND. this imaginary drone in our current technology would last 3 seconds bro. And it doesn’t linearly scale. For it to function for one day with 1 drone you would need 26 BILLION Context for it to function for 1 day. That is 106500 NVIDIA H100s. Which are 25000 dollars each. That’s 2 billion dollars bro. Lmao. For 1 drone. And also it doesn’t work like that that you cant stack chips. So that’s not even feasible. And doubly so because it would create so much heat that the GPUs would instantly melt.
So basically anthropic is being like bro I can’t sell u these unicorns bro it’s unethical and the us government is so stupid that they get mad about it. And then Sam Altman walks in and promises the government to give them unicorns.
They cannot.
The only solution btw is quantum computing which turns O(n)^2 into O(n). Meaning linear scaling. Which means that eventually quantum computing will bring along autonomous agents and destroy everyone’s employment. But that’s a topic for another day. And not going to be a thing for atleast 5 years probably more like 10
I have been reading much scientific research papers. If you would like sources I can give u them.
And basically the inevitable conclusion of this is this is how the ai bubble will crash. When investors realize unicorns cannot be created by GPUs. Nvda worth 5 trillion dollars based upon the idea that GPUs can create unicorns. When the unicorns predictably don’t come to be we are going to enter a nasty recession.
That requires a human. That is my point. Lmao. Agents already can do this. Many things are cooked. Such as coding. Basically coding is the only thing where 100% of the job can be done by 1 person now that used to take 10. Other jobs like legal secretaries it’s maybe 1 out of 2 replacements. A data analyst can do with agents what used to take 5. But all agents, they need a babysitter handler. Which makes them glorified productivity tools. For actual replacement of the majority of jobs. You’d need fully autonomous agents. Ie agents that can literally run for years without any human intervention other than their boss asking them to do stuff. Instead of a handler having to use a tool. An agent that has full memory and doesn’t have to be reset constantly. All agents rn have to be reset constantly. That is the nature of a context window with a classical computer. Which can replace many jobs in specific niches. But it’s no where near close to even 20% of total jobs. And a productivity tool that eliminates a minority of white collar jobs via making the remainder of workers more efficient does not justify 750 billion dollars of capex.
IMHO, clankers have the same issue as you explained with coding. You can give them some smallish task but their context window isn't big enough to implement it in a consistent way in any system of non-trivial size. So what happens is that they "forget" about parts you already told them about or which they processed before. Which leads to unmaintainable, inextensible spaghetti and unnecessary features and stubs all over the place after a few iterations. The only way to clean that up is with a human in the loop who's micromanaging the LLM.
All at the capital expenditure of the lifetime salaries of all the coders we have put together.
There are very few jobs where a human can work uninterrupted without consulting with others. You and I know that. The morons im their C-suites don’t care about that, they merely want to turn all variable capital down to 0. But you can and should have humans in the loop.
Checking logs and outcomes
Have you done the math? If my job got rid of me and my coworker they’d probably free up 150k+ a year. Take that and put it on a 20-30k license for ai tools and…🤷♂️