“A lot of people want local inference,” he said. “And somehow, they assume, that the problem is solved now. Because we are going to have open-source models. We’re going to be able to run them on our own computers. And so we’re not going to need all the stuff in the data center.”
“But from a global allocation perspective, actually, it’s quite terrible to have everybody have a bunch of the world’s limited supply of [dynamic RAM] sitting in their own computer,” he added.
Yeah, I've made that point here many times myself. Capacity utilization is going to be much higher in the cloud, so it's cheaper to do in the cloud than to have everyone with their own inference hardware.
But...that depends on the constraint being limited on hardware. That's the case in 2026, but unless much more hardware-intensive models keep showing up and being much better than came before, so that someone like OpenAI can get value that increases tremendously out of more hardware for a model, that hardware constraint won't persist.
Local compute also provides some benefits. You don't rely on Internet connectivity, so you don't have things go down if your network connection does. While we don't have many latency-critical applications out there today, you can get better latency. Unless and until OpenAI is going to provide service with some pretty hard and convincing data privacy guarantees, you get data privacy. Whether it's a company that might be exposing their core IP or what their people are working on, an individual who doesn't want their sexy chatbot sessions showing up places they don't want them one day, a government who has national security concerns, or anyone who is just worried about a cloud LLM provider leveraging lock-in down the line to extract concessions or more money for them, local LLMs provide a lot of benefits.
I am confident that the world as a whole won't shift to local AI for at least four or five years, and probably more. It can't—most memory right now is going to cloud AI, and reduced capacity utilization means that getting the same compute capacity would require more memory than is out there, we don't have construction for many times existing memory capacity underway yet, and it would take four to five years for that to happen if we started today.
But I do think that it will tend to happen over time.
I think that in the long run, if cloud AI companies are going to persistently keep people on remote compute, it'll need to be because their models stay private and their performance is so much better than open-weight models that people choose them for quality, not for hardware scarcity.
