104
No datacenter needed now, just run it on your computer. Anything with 12gb+ of vram can run it comfortably at low quants. If they can come out with a good MOE model it will crush the competition. At this current rate of progress, China will out-compete foreign AI development within a year or two.
These data centers got to be part grift right? Like I'm a dolt when it comes to tech and even I know eventually powerful ai will be able to be launched off consumer hardware. So who are these datacenters for?
Basically the top tier datacenter models right now are extreme generalists and store quite a lot of humanity's total knowledge on them, even the ones that are capable of being run locally are 2-4tb in size and would require thousands of gigabytes of ram to run well, a cluster that could run that would cost 50k-100k usd. The local 27b models try to rip out the "intelligence" of these titanic large models. They then augment the model's intelligence with toolcalls (looking at say, github or a search engine) instead of having all the knowledge baked into the model. This approach results in less hallucinations and far less resource usage, but it also isn't as generally useful because you may not have access to some niche information, information you couldn't find with a google search.
The big benefit for local models is you can use them to crush through a lot of google searches in one go and find you a good source, its great for people doing scientific research because you can search all the journals you have access to very quickly for relevant info. Its also great for doing up small bits of code quickly and iterating on that code for your various experiments.
This is the only thing I’ve tried to really get an AI model working for, I want it to do some large scale searching and format the info with links to the sources for me to look through. None of the text it writes will end up in my final product but it should be able to do the searching a lot more efficiently than I can myself.
Unfortunately I’ve completely failed to get this to work
You might like Proton Mail's Lumo project, I find it useful for quicker searches without needing an account, they use open-weights models only. Unsloth with EXA or setting up searxng-mcp is your other best bet, Unsloth has a desktop app now.