This is a most excellent place for technology news and articles.
This is true. But VLLM's quantization support (as outlined in the article) is quite primitive.
Thats not what its built for. Its made for maximum speed with many concurrent users, not cramming LLMs into a desktop rig.
If you are running an LLM for personal use, you are usually better off with a backend optimized for higher fidelity quantization, like MLX DWQs, exl3, ik llama GGUFs and stuff like that.
If you want raw speed on a desktop, use one of the ninfer project branches.
inari@piefed.zip: Are you a bot? You are posting multiple times every hour of the day.
Nope. I post things I find on Reddit or Hacker News sometimes, though.
Do you know the recipe of the apple pie?
Nah, I dont think so. There are comments mixed with posts, there arent that many posts, and they seem to be reasonably curated.
Having used these extensively, I can say with confidence that they feel exactly as dumb as they are.
Do you mind sharing what models you used and what your experience was? In my opinion the Qwen 3.6 models or maybe the 3.5 were the first local models that were actually useful, but I don’t have that much experience.
Qwen is really good. For coding, it's the best local model I've found.
If you ascribe any level of sentience or intelligence to a computer, you definitely feel much less dumb than you actually are.
Nice strawman you got there. Is this relevant to the post, or do you just feel like calling random unrelated people dumb?
