104
No datacenter needed now, just run it on your computer. Anything with 12gb+ of vram can run it comfortably at low quants. If they can come out with a good MOE model it will crush the competition. At this current rate of progress, China will out-compete foreign AI development within a year or two.
The speed at which open weight models are progressing is crazy. They must be having sleepless nights in Silicon Valley lately.
27B dense models need like somewhere between 14 and 16 GB of VRAM at 4-bit quantization, and that's not even accounting for KV cache. Even on a 16 GB card that would be uncomfortably tight, and wouldn't give you any room for long context lengths. You definitely need a 24 GB card to run this properly. That's still damn impressive. I didn't expect frontier model performance on a single consumer GPU so soon.
I think when they release the 35B-A3B MoE model, it will be the final knockout punch. That one you can technically even run on an 8 or 6 GB card at decent speeds with some smart offloading if you have enough system RAM. Being able to run a frontier level model on a low power home server definitely changes the game.
Can you explain what the 35B-A3B MoE model is? I've seen so much jargon like this before that I mentioned in a previous thread on why people ignore self-hosted AI (it's like listening to aliens speak). I am a little familiar with some of the terminology, but don't really understand why a 35B model would be easier to run than a 27B one and so on. Or what the quantization is.
I've got a 16GB vram card myself and find most models like the recent Qwen to respond far too slow or cause out of memory issues and crash.
35b-a3b means it only actively calls 3b relevant parameters while generating, it offers similar performance as a base model like 27b but for less resources (you can hold some of the model in ram, but at the cost of storing more of the model on your drive). if youre able to run it entirely in vram somehow it can run blazingly fast in comparison to other models as well. I personally hope they're more ambitious than a 35b a3b, something that works more like a6b, a9b, a13b, etc would be better for most lower end computers.