this post was submitted on 26 Jun 2026
148 points (92.0% liked)
Technology
85873 readers
3994 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Distillation allows you to make a smaller model that can produce the same outputs as a larger model. Basically they're pirating all of the hard work anthropic did pirating the entire internet.
Alibaba gets a model that produces basically the same output for a tiny fraction of the cost to operate the model once it's finished training. Distillation training also uses basically all of its data from the big model (afaik it's all of it sourced from the parent model)
It's like if you took a lump of metal and showed it Porsche 911s until it turned into a 911 shaped chunk of metal that had 95% of the performance, but it only cost you $3000 for the ingot, and also cost ⅕ the amount in fuel and maintenance.
Ok, thanks for the detailed explanation. I guess if your goal is to make your model sound like another model that makes perfect sense.