this post was submitted on 02 Nov 2026
709 points (99.4% liked)
Technology
89409 readers
4286 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Except the people I hear running “local models” report a lot of hardware, like a half dozen graphics cards. The common person is screwed, they have a smartphone.
Two or three years ago sure. Now...not even close. Tons of models run on basically shit now. Especially with MOEs. I run new lightweight models on an i9 and 64gb of ddr4 ram with a 4gb quadro card...in a laptop from 19'.
I think this is why a lot of these tech bros are saying silly shit about suggesting people need to rent from them. That's their entire shtick. Otherwise they got nothing else.
Got some references and keywords for research?
If you've got at least 8 gigs of VRAM and at least 32 gigs of RAM you can run some pretty beefy Qwen models using Strata, far more than you normally could. Upgrade that to 12 or 16 and 64 gigs respectively and you can run a pretty decent quant of Qwen 3.8 flash next. It's a LOT bigger than the usual 27 and 35 billion models people run locally that themselves require pretty beefy hardware. Strata is doing some magic by deciding which experts stay in VRAM vs which ones are in RAM vs which ones are so rarely used that they can just be read from the SSD as needed (though it's best if you don't have to result to the SSD as a backup option at all)
I saw the video, you still need a fast NVMe, I'm still in data SSD...
https://youtu.be/8F_5pdcD3HY
This guy does a great brain dump on how little is needed to run LLMs on basically old junk.
Kudos