this post was submitted on 18 Oct 2026
1158 points (97.9% liked)
Technology
89443 readers
4292 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Thank you! That's the best answer I've seen on any platform!
What would you recommend as a local model? Want to dip my toes in but I'm woefully ignorant. And again, I've done my time in IT, so you don't have to dumb it down, uh, too much.
The last time I ran one locally was a few months ago, so this might be somewhat out of date, but I'll say Qwen.
Generally there are two numbers to look at first, the number of parameters (number of nodes + much larger number of connections between nodes) and the quantisation (number of bits that define each parameter)
So an 8 billion parameter model at 8-bit quantisation will require 8GB of working RAM, probably VRAM for fast results at home (at 8 bits to the Byte)
and also a 16 billion parameter model at 4 bits quantisation will also fill the same 8GB.
in my experience, high bit depth enables the thing to respond in more complex terms, and more parameters means more "knowledge" is there to be accessed. I have better results from phrasing questions that'll have short answers and leaning more towards more parameters than bit depth.
The next number to consider is "context window", which is basically how far back into your chat the thing can "remember" and have previously given information influence the next answer.
A year to two ago, when I was trying a lot of LLMs for coding tips, Google's seemed the best for larger projects, simply because it had a much bigger context window than any other, and so could "consider" longer and more interconnected scripts as a whole, even if perhaps Claude was "smarter" with a small problem. I don't know if this is still true in autumn 2026.
They've (commercial models) started keeping a per-account or per-chat log of important facts that exists alongside the context window, but doesn't carry as much detail. Good for seeming to get to know you, not as good for writing new code that won't conflict with a long-established bit of old code.
It's not really practical to have million+ token context windows on home/gaming hardware.
(usually a token is a word, or a couple of tokens are a long or rare word, or a token is a punctuation mark)
LM Studio is good software with generally sensible defaults for downloading pre-trained models and running them on your own PC.