104
No datacenter needed now, just run it on your computer. Anything with 12gb+ of vram can run it comfortably at low quants. If they can come out with a good MOE model it will crush the competition. At this current rate of progress, China will out-compete foreign AI development within a year or two.
Can you explain what the 35B-A3B MoE model is? I've seen so much jargon like this before that I mentioned in a previous thread on why people ignore self-hosted AI (it's like listening to aliens speak). I am a little familiar with some of the terminology, but don't really understand why a 35B model would be easier to run than a 27B one and so on. Or what the quantization is.
I've got a 16GB vram card myself and find most models like the recent Qwen to respond far too slow or cause out of memory issues and crash.
35b-a3b means it only actively calls 3b relevant parameters while generating, it offers similar performance as a base model like 27b but for less resources (you can hold some of the model in ram, but at the cost of storing more of the model on your drive). if youre able to run it entirely in vram somehow it can run blazingly fast in comparison to other models as well. I personally hope they're more ambitious than a 35b a3b, something that works more like a6b, a9b, a13b, etc would be better for most lower end computers.