this post was submitted on 03 Apr 2026
23 points (62.9% liked)
Technology
83990 readers
3447 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Also, for any interested, desktop inference and quantization is my autistic interest. Ask my anything.
I don't like Gemma 4 much so far, but if you want to try it anyway:
On Nvidia with no CPU offloading, watch this PR and run it with TabbyAPI: https://github.com/turboderp-org/exllamav3/pull/185
With CPU offloading, watch this PR and the mainline llama.cpp issues they link. Once Gemma4 inference isn't busted, run it in IK or mainline llama.cpp: https://github.com/ikawrakow/ik_llama.cpp/issues/1572
If you're on an AMD APU, like a Mini PC server, look at: https://github.com/lemonade-sdk/lemonade
On an AMD or Intel GPU, either use llama.cpp or kobold.cpp with the vulkan backend.
Avoid ollama like it's the plague.
Learn chat templating and play with it in mikupad before you use a "easy" frontend, so you understand what its doing internally (and know when/how it goes wrong): https://github.com/lmg-anon/mikupad
But TBH I'd point most people to Qwen 3.5/3.6 or Step 3.5 instead. They seem big, but being sparse MoEs, they can run quite quickly on single-GPU desktops: https://huggingface.co/models?other=ik_llama.cpp&sort=modified
What's wrong with ollama?
Ughhh, I could go on forever, but to keep it short:
Tech bro enshittification: https://old.reddit.com/r/LocalLLaMA/comments/1p0u8hd/ollamas_enshitification_has_begun_opensource_is/
Hiding attribution to the actual open source project it's based on: https://old.reddit.com/r/LocalLLaMA/comments/1jgh0kd/opinion_ollama_is_overhyped_and_its_unethical/
A huge support drain on llama.cpp, without a single cent, nor a notable contribution, given back.
Constant bugs and broken models from "quick and dirty" model support updates, just for hype.
Breaking standard GGUFs.
Deliberately misnaming models (like the Deepseek Qwen distills and "Deepseek") for hype.
Horrible defaults (like ancient default models, 4096 context, really bad/lazy quantizations).
A bunch of spam, drama, and abuse on Linkedin, Twitter, Reddit and such.
Basically, the devs are Tech Bros. They're scammer-adjacent. I've been in local inference for years, and wouldn't touch ollama if you paid me to. I'd trust Gemini API over them any day.
I'd recommend base llama.cpp or ik_llama.cpp or kobold.cpp, but if you must use an "turnkey" and popular UI, LMStudio is way better.
But the problem is, if you want a performant local LLM, nothing about local inference is really turnkey. It's just too hardware sensitive, and moves too fast.