Technology

So basically PuzzleMoE is a new way to compress Mixture of Experts models without degrading their performance. The big problem with MoE is that it takes up a ton of memory because you have to store all the expert weights even though only a few are used at a time. Old methods like dropping or merging experts would just nuke the model's accuracy, but PuzzleMoE's trick is to do fine grained element wise merging by figuring out which individual weights are important and which ones are redundant across experts to merge them smartly.

Then a bit packing scheme stuffs the mask and sign data into the unused exponent bits of the Bfloat16 format since MoE weights don't use the full exponent range anyway. This lets inference run without any extra memory overhead making the whole thing fast.

This approach beats other compression methods by up to 16.7% on MMLU at 50% compression while having up to 1.8x faster inference. Pretty neat stuff.