this post was submitted on 01 Sep 2026
207 points (92.6% liked)

Technology

87680 readers
3379 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots


founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] bl4ckp1xx13@lemmy.dbzer0.com 31 points 12 hours ago* (last edited 5 hours ago) (2 children)

If the model is adequately FOSS, i.e. open weights, and can run on a single consumer GPU (or NPU), and the "author" (quotations because I'm personally undecided if one can claim code generated by an aforementioned model is theirs) understands it, then I really don't see what the problem is...

Personally I have nothing against SLMs/LLMs as a technology, to me my grievances against ChatGPT or Claude are mostly about their environmental impacts and selling us back our own art, also keeping knowledge behind a for-profit black-box - if those aren't appropriate for a specific model, then I say using that model is fair, and good for productivity.

[–] Balinares@pawb.social 16 points 12 hours ago (1 children)

I'd take a model whose training data is open source and legitimately obtained. The only ones I know about are Apertus and OLMo, and they aren't really competitive.

[–] bl4ckp1xx13@lemmy.dbzer0.com 5 points 7 hours ago

Unfortunately not yet, no, true FOSS models are likely many years away, but I would argue that that follows typical FOSS lifecycles. Emerging technology is typically outperformed by proprietary endeavors, which creates an audience, and then that audience undertakes a FOSS implementation that initially underperforms, then just about competes, then eventually overtakes (i.e, GNU/Linux).

[–] themachinestops@lemmy.dbzer0.com 3 points 10 hours ago (2 children)

The problem with open weight models is that they are still hard to deploy. For DeepSeek for example you require NVIDIA HGX B200, these things are expensive.

[–] Wildmimic@anarchist.nexus 1 points 1 hour ago

If you are really using those things for work, a 32GB VRAM Blackwell currently costs around 3800-4000€. My own experience with local models on my 5070/12GB with 64GB DDR4 has shown me:

  • I can run a Qwen 3.8 35b Q_8 quant (BF16 is a tad too large) using CPU offloading, while still reaching acceptable speeds for private use.
  • Ternary Models are a game changer. Built to be used in edge computing, Bonsai-27B-Ternary from Prism ML runs completely on my GPU (using ca. 10GB of my 12GB VRAM incl. MMProj, 9 when disabling vision) and even leaves enough VRAM to run subagents (when killing my plasmashell lol -but i tend to offload these to the CPU in favor of increasing the context of the main model). currently you have to install a costum fork of llama.cpp to run ternary models, but the needed changes will be applied upstream soon. (There is a Bonsai quant which aims to run on iPhones with below 6GB VRAM footprint - the limit of what Ios currently allows)

If this scales linear, it would mean that the 32 GB of a smallish Blackwell would be enough to run Ternary models with around 70B parameters plus a ton of context window without breaking a sweat.

These are strange times. I absolutely love the technology and the advances, and i fucking hate the companies pusting these technologies.

Altman, Amodei, Musk, Nadella, Huang and all of the othe AI evangelists can go choke on a bag of dicks in my opinion. This technology never should have been this divisive, it should have just become another tool in humanities toolbox. I fear that these people have tainted the general populations outlook on this tech for a very long time.

[–] bl4ckp1xx13@lemmy.dbzer0.com 1 points 7 hours ago

Yes, unfortunately they are, and I do think efficiency is going to be a significant research front for open-weight models. The nature of this topic is highly speculative as our compute capabilities have only recently reached what is required to consider running generative AI models, what we have today are very crude first implementations of what I personally believe will become an everyday tool for developers, and more.

And we have seen this, there are models now capable of running on an individual's hardware (and not particularly expensive hardware either) that can outperform what ChatGPT initially launched with.