129
Which one of them made these overlapping bar labels?
this post was submitted on 28 Jul 2026
129 points (91.6% liked)
Technology
43101 readers
177 users here now
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
founded 7 years ago
MODERATORS
Hey, a lot of people sucked making matplotlib plots before LLMs existed.
load more comments
(1 replies)
It’s a bit misleading.
Qwen 27B has way less “world knowledge” than GPT-5. Ask it random trivia without internet search access, and GPT would know waaay more.
This is generally true of small vs large models.
…But honestly, Qwen 27B is better at tool use or agentic stuff. It’s hyper optimized for just that and coding assistance, basically.
This is often true of old vs new. Most newer models have hyper focused on agents/coding, often to the detriment of other use cases.
Quantization for practically running Qwen 27B also has an impact. A off-the-shelf Q4_K_M is not the same as the unquantized weights in real-world use, or even an “optimized” quantization like a custom exl3.
I mean baking knowledge into a model isn't really all that useful to begin with. Just download wikipedia locally and have it access it through tool use, it's way more efficient and more accurate. And yeah, I find Q6 tends to be the sweet spot where it's close enough to full 16 bit in performance, but doesn't chew up too much memory.
This is a pretty good idea I hadn't thought about. The whole wikipedia is about 37Gb.
yeah, it's not a completely insane amount of data, and a db like postgres can do fast text search on that too with fuzzy matching
If you run it with RAG (retrieval augmented generation), where the model includes web search, you get pretty decent results. I often include the phrase "search the web extensively for up to date information" in my prompts.
What version do you use and how do you run Qwen3.6? I've played around a bit with the Q4 version in LM studio +Zed, but I was not happy with the results. It looses track very often and often enters infinite loops or just stops...
I use an exl3, with 4 bit MLPs but higher bit depth attention layers. And I force some custom sampling so I can lower the temperature a bit while keeping it out of loops.
This won’t work in LM Studio though. You have to run such a thing in TabbyAPI or some other backend that supports exllamav3.
Ok, thanks! Thaf sounds quite advanced, but I'll have a read afterwards :)
That's why you couple it with your own, self-hosted yacy or searxng instance. Embedded world knowledge does not help a model if it becomes outdated. I just let my agent research, embed that knowledge to a little Qdrant server, so other servers are not bothered again and pull the information from there when needed again. With a little RAG you can have GPT at home.
I sort of get it because of you know you're only using the model when you have Internet and want it to search then just let it.
Facts! Thank you!
This seems like a great time to mention what I built and hosted months ago. https://masland.tech/ai-efficiency-index/
The AI Efficiency Index tracks the cost/intelligence mix. Low intelligence is useless. Capability scales exponentially. Expensive ≠ better value. MiMo-V2.5 is the leader.
If you have any questions feel free to ask. This uses the Artificial Analysis leaderboard. They have "cost per task" which I think is weird.
Those models are for general use. If you have business use case and data related to it you can finetune model for specific use that will outperform all of frontier models and run at fraction of cost.
that's the other huge advantage of open models you can run locally
How can you run a local model for better coding support than fable/opus?
You can't right now because there is no dataset available for fine tuning. I'm just saying that it's possible to fine tune 8B-35B parameter model in int4 that will outperform those models ex. for single programming language and developer specific problems. You can read example of fine tuned 8B model here https://fermisense.com/when-machines-take-the-wheel/
load more comments
(3 replies)
[citation needed]
This is for qwen 3.5, not 3.6, but the vram requirements are a little out there for most:
https://willitrunai.com/blog/qwen-3-5-27b-vram-requirements
16 to 55GB...
I mean that's high but not unreasonable, especially with unified ram machines
Little outside my budget personally, but I've definitely been itching to pull the trigger on a nVidia P40
I bought two RTX 3060 with 12 Gb each for a total of 24Gb VRAM. Sionce the cards are not top shelf anymore for gaming, they can be had used in the 200€ range, so my setup was 400€ over the cost of my existing PC. My PC, while not last gen was already pretty beefy, with a Ryzen 9 and 64 Gb RAM. I think 400€ to have a pretty good local AI setup is not bad at all.
load more comments
(3 replies)
I can run both the 27B and 35B version of Qwen3.6 on my 10GB VRAM 3080, and they run ok. You don't need to put the entire model in VRAM, even if it's probably beneficial.
Full quantisation? I've only got a 8GB 3070, but I'll give it a go
Edit: Tried the unsloth/qwen3.6 with llama.CPP, and it failed to allocate a 26GB Vulcan buffer and died. Dunno what magic your using, no luck for me though :(
I run it using LM Studio, which defaults to Q4 quantization, I think. I was able to put about 10 layers on the GPU with 64k token context. That put me at about 9.1 GB VRAM usage, leaving some room for Video playback xD
I'll give LM studio a go, thanks.
what kinda hardware do you need to "run this on your desktop"?
You need a GPU with around 16gb vram at a minimum to run qunatized version.
I run Qwen 3.6 27B quantized down to Q4_K_M or even a variant Q5_K_S on my 8gb VRAM entry level AMD GPU, with support of my CPU and 32gb system RAM. Yes, I also limit the Context Length heavily to something like 18k. It's slow. But the point is, you don't need necessarily 16gb VRAM.
But it's better to use faster models for this type of hardware anyway. The MoE type of models (such as 26B A4B, or 35B A3B) are vastly, vastly faster for normal usage, but they are a bit worse in some cases. Also Googles QAT trained model versions also have less RAM requirements without losing much quality.
I'm just saying that, so others are not discouraged too much. It's not the same experience as the full version off course, but you can use them with some tricks on weak hardware too.
I'm fairly optimistic that people will figure out how to optimize the models a lot further going forward. One obvious path is to try and separate the reasoning network from the trivia that gets baked into the model, and some work is being done in this area. If you could have a context free reasoning engine and then feed the facts it needs to know on the fly based on the context you're running it in, then you could likely have a much smaller model that's very capable.
You can run it on systems with as little as 32 GBs of RAM.
You don't want to run these models in RAM. I started using them on an RTX 3060-12Gb VRAM, and quickly added another, for 24Gb RAM. I also have 64 Gb RAM. Now it works well. Not lightning fast, but quite useable. VRAM is the key.
Sure, but with a model of that size you are unloading layers in any case. I'm just saying you need 32 GBs of RAM at the minimum to be able to run it.
load more comments
(3 replies)
according to performance on standard benchmark. somewhat covered by the controversy surrounding the term: benchmaxing.
if you see all benefit as a linear one dimensional height on a bar graph..
its almost like you assume that the previous model gave the same exact answer(same style) and the new mode gave the same exact answer PLUS additional useful information.
it might be convenient if measuring progress was so simple. but unfortunately/fortunately , it is not so simple .
the most important benchmark are the comparison of outcomes on the problems that YOU have & prompts that YOU can(will) write. nothing else matters for YOU.
- i admit benchmarks are well designed to objectively measure competence on challenging problems that require skill and really need only ONE correct answer.
Sure, a benchmark doesn't capture all the subtleties and different use cases, but it does give a general idea of the capabilities of a model. Obviously, you have to run the model and see if it does what you need. But the chart isn't really about the nuance, it's showing how drastically the efficiency of the models has improved in just a year. The fact that we can even reasonably compare a model you can run on a desktop to one that needed a data center just a year ago is phenomenal.
load more comments
(2 replies)
🙄
Is it still telling people to put glue on their pizza and convincing teenagers to kill themselves?
So which one tried to cheat its way into the test?
Llama is one for sure
Kimi is better. Waiting for it to appear on Ollama Cloud.
The difference is that you can run Qwen completely local though.
Ok, but all of them are far too inefficient for what they deliver.
Ok, but that's a completely nonsensical statement. If you ever used Qwen in an agentic loop, you'd know that it delivers working code, and it takes about same resources as playing a modern game, and I don't see anybody whinging that game are too inefficient for what they deliver.
load more comments
(2 replies)
view more: next ›