Ask Lemmy

From that perspective local LLMs sound more like classical piracy. Not ethical, not FOSS, but out of the hands of greedy corporates.

They are all trained on copyrighted material without permission, no LLM is ethical

I'll take a crack at it.

I run AI on an old Nvidia p40 purchased online for less than $200. The models I run on it came predominantly from Chinese companies and groups. On balance, they:

• use much less fossil fuel in producing these LLMs than their Western counterparts

• produce models that run much better on lower end consumer hardware

As for training data, the inputs used to create these models, it varies greatly but the most popular line, Qwen, Came from the company's own data from operating such huge networks and systems for so long

I really fail to see how any of that is worse than playing a video game.

None of this takes away from the very real issues around data center build out and Western companies using the systems to scare people and continue an economic bubble. That's all true and bad. But there's nuance. Not all AI is created equal

There's a lot of info that you need to know to explore this space, so I'll take my own shot at answering. Let me know if anything needs further explaining!

An "open weight" model is an AI model where you can download the data needed to run the model on your own hardware for free. Contrast this with proprietary models-as-a-service like ChatGPT and Claude where you have no access to the data needed to run the model yourself -- you can only use it through the services provided, usually for a fee, and which can be taken away from you or changed at any time with no recourse.

The mapping to traditional open source terms does not work well since what you get is a binary artifact.

Those artifacts are released with a license -- and many of the models are licensed permissively (e.g. MIT or Apache license terms). You can take those weights, modify them, and then release them as new models -- and people do actually do this in practice!

Is it really feasible to run it 100% locally?

Yes. I run models on my own computers and have tried a number of configurations to figure out what works well. The Qwen family of open weight models (from Alibaba) are the ones I've found most useful so far. Gemma4 models (from Google) are also useful.

I prefer models that have been modified by the community to remove corporate censorship -- i.e. stripping that "As a large language model..." cover-your-ass crap and evasiveness on topics like Tiananmen Square. If that means the model is technically capable of telling me to go kill myself too, so be it; I've spent 25+ years dealing with assholes on the internet and can handle abuse from a stupid robot if I have to. (In practice though, they're usually pretty nice still unless I deliberately tell them to act like an asshole -- and then Qwen, at least, starts to sound like a snarky redditor; it's quite funny most of the time, actually.)

If yes to the previous: the software doesn't come from nowhere and ultimately still relies on gas-turbine-powered datacenters and stolen IP and stolen personal data, no?

Models do require training to create, yes. It's generally not clear where the training physically happened IRL -- so, yes, some of them probably used power from gas turbines, but others may be drawing power from the Three Gorges Dam in China or solar plants or nuclear plants or whatever else is hooked up to the electric grid where the training happened. Most of them are also not very open about the data sets they were trained on. (There are exceptions to this though!) The Chinese models in particular are almost certainly trained heavily on logs extracted from Western models in addition to using whatever other data they could get ahold of. Whether you think that's ethical or not is a matter of perspective; how do you feel about Robin Hood?

Once a model has been trained though, it can run on a normal GPU. The power requirements to run an LLM are basically the same as running a video game, or, equivalently, about the same as turning on a few incandescent lightbulbs. (The iGPU in one of my systems uses 100W; the discrete GPU in another system I've tried uses 215W under load with appropriate tuning -- or 300W if you run it naively.)

If you want to run a model yourself, I recommend using llama.cpp -- there are instructions on how to get started with it here: https://llama.app/

These are the models I've found most useful:

• Qwen3.8-27B (official version): https://huggingface.co/Qwen/Qwen3.8-27B
• Qwen3.8-27B (uncensored): https://huggingface.co/MuXodious/Qwen3.8-27B-absolute-heresy
• Qwen3.6-35B-A3B (official version): https://huggingface.co/Qwen/Qwen3.6-35B-A3B
• Qwen3.6-35B-A3B (uncensored -- ⚠️ mildly NSFW graphics on page): https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic
• Gemma4 (official collection -- many variants in different sizes): https://huggingface.co/collections/google/gemma-4

If you have an iGPU only, I recommend using one of the so called "Mixture of Experts" (MoE) releases. These are typically named like 35B-A3B or similar; the first number indicates the total number of weights (35 billion) and the second indicates how many are "active" (i.e. actually used during computation) at one time while the model is running (3 billion in the example). These models need less computation to run and stay fast on weaker GPUs. Qwen3.6-35B-A3B is very good in this space and was my go-to model for a long time.

If you have a discrete GPU and enough VRAM, I recommend using a dense model (i.e. one that activates all its weights while answering) like Qwen3.8-27B.

It's worth noting that people don't usually use the full quality weights (which are typically ~2 bytes per weight); they use a "quantized" version -- compressed in a lossy fashion like a JPEG. Going down to 4-bits (half a byte) on average per weight is about as low as most people like to go -- you will see this indicated in names like Q4_K_M. (Quantized to ~4 bits with the K quantizaation scheme, medium variant.) Usually a bigger number is better in the sense of "closer to the original quality" -- at the cost of needing more RAM.

Full quality weights are often found as safetensor files on HuggingFace. Quantized weights intended for use with llama.cpp are usually in GGUF file format.

Does that help?

Is it really feasible to run it 100% locally?

Yes. My Macbook Air (M2) released in 2022 can run many publicly available LLM models. The ouput is not as fast as using a large powerful datacenter, but for my local needs, I'm not in a hurry. I get about 17 to 30 tokens per second speed running 100% locally.

If yes to the previous: the software doesn’t come from nowhere

For Mac users, the interface comes from here . For the specific LLM models that is a separate question for each.

and ultimately still relies on gas-turbine-powered datacenters

DCs generating power on-site is a relatively new phenomon because existing grids are at capacity, so the only way to bring new DCs online is locally generating power at that DC, usually using gas turbines or even worse, diesel generators. Most if not all of the publicly available models for running on your own hardware were built before those gas-turnbine-generating DCs were a thing.

The public models people are running now have existed for a number of years are likely made on regular utility grid power which is whatever that nation and region uses.

and stolen IP and stolen personal data, no?

The Llama LLM is made by Meta, so probably yes for that one. Deepseek is from an AI research lab in China. QWEN is from Chinese company Alibaba. We don't know for sure the inputs that created the Chinese models. US AI companies claim a number of the Chinese models are derived from American LLMs, but I haven't seen (or looked for) proof of these claims.

If what I wrote above is true, what exactly are people arguing when they say it’s still possible to use LLMs ethically

Likely they mean because you don't have to pay a large LLM owner in a rent-seeking model to run LLMs, nor does a person's use contribute to further development by those companies.

or true to FOSS philosophy, because … ???

With the open-weight models it doesn't rely on a commercial license to use, and the interfaces can be truly open source.

Is it really feasible to run it 100% locally? I know there’s plenty of people with very powerful rigs indeed, but still. Or are 99% of these people really saying “it would, in theory, be possible to run that locally, therefore your concerns are invalid”?

Very useful 35B models need (more or less) 8GB of VRAM and 32GB+ CPU RAM to be usable locally. 16GB RAM might work on a lean system with an RTX Nvidia card and an exl3 quantized model.

I know we are in a RAM apocalypse, but pre-apocalypse, that’s a quite reasonable requirement, IMO.

Personally, I run Deepseek V4 07-31 Flash at 19 tokens/second on a desktop with a single RTX 3090 and 128GB CPU RAM, and that’s an extremely capable model. Again, that’s expensive these days, but pre-ram apocalypse, that is not an unreasonable workstation/homelab.

If yes to the previous: the software doesn’t come from nowhere and ultimately still relies on gas-turbine-powered datacenters…

Most open weights LLMs are Chinese. And they are:

• Trained on pennies. They have to be, as they simply do not have a sea of GPUs like Big Tech. Training costs for their large models are in the millions or tens of millions; a single steel forge has used more energy than all of those training runs combined.

• China relies more on renewables, and I believe the datacenters aren’t so hastily constructed with gas turbines in the middle of cities.

• And as of now, they are transitioning away from Nvidia GPUs. Some labs already use Huawei accelerators.

…and stolen IP and stolen personal data?

Yep.

This is a huge caveat.

You can avoid this. Nvidia Nemotron models, for example, are trained on completely open datasets you can download and inspect yourself: https://huggingface.co/nvidia

They are very good, but just behind state of the art.

But in practice, the SOTA models most run use private datasets. Lord knows where the Chinese get it from, but given some common quirks between models, at least some data sources are shared (and possibly government provided?)

…However.

I would argue providing the result of the training as Apache licensed weights counts as “fair use,” in the same way non commercial fan works do.

They aren’t making a dime off releasing those weights. I’m not trying to sell anyone anything when I use them. Where is the IP theft if money isn’t changing hands?

Now, the Chinese LLM services they charge for? I have no excuse for that. Once money is on the table, it is definitely IP theft.

You can run model locally, a gaming PC is enough, may-be not for cuting edge models but if you want to generate clues for a RPG or rephrase a letter it's good enough. You can look for LM studio and Stability matrix for example

You can run them locally, yes. There are models that can even run on phones, but usecase is limited. But it can only be considered ethical, if the training data used is listed or ethically sourced IMO.

AI bros on Lemmy will disagree with me, but most open weight models are still trained unethically i.e, theft. Most proponents of LLMs (who I talked to on bsky), who say local models are ethical, don't fucking use it. They're larping on socials about how awesome it is, but none of the ones I talked to are using it in their projects. They mess around, realise it is not as good as the "unethical" options, go right back to Claude

Open weight models Qwen, deepseek, mistral, and the Ollama stuff etc are unethical in normal people's eyes, but "ethical" enough for AI bros.

From what I searched, there are very few that can be considered ethical - Olmo, Apertus, Starcoder(?). But idk anyone who uses these. My friend at IBM said they used Apertus, but it was nowhere near good as ChatGPT, so they no longer use Apertus now. And these models require minimum 6-8 GB VRAM for their lowest parameter model iirc.

Even the open-weight model bros are lobbying to redefine what 'open-source AI' means. That should give you a fair idea about people behind open-weight as well

It is absolutely possible to run 100% locally, but in practice, at the high end, only quantized models. The full size top end models require hundreds of GB of video ram, and while you can buy that, it's stupidly expensive. Quantized models can often perform nearly as well with a small fraction of the ram, but they do sacrifice a little in precision.

These models (well the good ones) ultimately all trace their origins to what you'd likely consider "stolen" data. Whether that's ethical is debatable. If you're in the "information should be free" camp, there may not be an issue here.

As for the power/environmental impact, for what they do LLMs are actually very low impact per-request. If you're concerned about your personal AI power use, then I hope you never fly in an airplane, and minimize your driving because those are much bigger issues.

It's the scale of use that makes AI an environmental problem, and that's a question about corporate use of AI, not personal use of AI.

If you can play modern games, you can probably run a local LLM. The more VRAM the better it'll go.

For ethics, there's two issues. The first is power consumption. Running a model locally isn't consuming any more power than running a game that fully pushes your system. It's not really a concern.

The second issue is content generation. It's all created through the theft of content others have made. There's no getting around that. As far as I'm aware there are no ethical models that specifically have not stolen content. If you're just using it to chat with or whatever, I think it's fine. You probably aren't taking work from anyone, at least to any degree more significant than piracy. If you're generating stuff to be distributed, then you're distributing stolen work. If that's ethical or not is up to you to decide.

I'll go one by one... although I do have to say, I can't answer the main question of whether it can be 100% ethical

Open-weight model

It's actually quite literal. LLMs may look like magic, but under the hood are basically very, very big mathematical/statistical functions that transform one type of input (for example, "User: hey ChatGPT what is an open-weight model?") to an output (example, "ChatGPT: To understand what an open model is...")... To simplify, these models function by piecing together a ridiculous amount of tiny, nonlinear mathematical functions, apply a weight to each of these tiny functions, and somehow get a coherent result out of it. The theory behind this has been known in math research for decades before OpenAI.

Because LLMs are mathematical models, if you know the weights of those tiny functions and how you chain theme together, you can run an exact replica of it. Open-weight models are just that, models where the entire weight and model architecture are released with open, permissive licenses. Many of these end up being modified by other users/hobbyists, and I do believe this sharing/modifying part of AI/LLM is quite true to FOSS philosophy.

As others have mentioned though, "open-weight" has no bearing on how the model is trained. It just means that the final product is free and open

Is it really feasible to run it 100% locally?

Theoretically yes, practically yes but depends on the model.

• For the really small models designed for mobile phones, absolutely. For example, LFM-2.5 and Ling-3.0-tiny run on just about any potato. Granted these models are... not great. They are probably functional as "search engine wrappers" though, methinks that's what they were built for.
• For the intermediate models, yes but with an asterisk. They require at least a decent GPU, so either you have a gaming rig, happened to have snagged one of the AI computers, or try to kneecap the model in some way (limit the amount it generates, make the mathematical operations less precise a.k.a. quantization, be okay with it running for a long time, ...). I have a local setup for Qwen3.8-27B with 4-bit quantization, but it runs at 2 tokens/second on my laptop (which has a ton of RAM but not a very good GPU)...
• For really large models, you can technically run them, but they are realistically designed for organizations with privacy in mind: hospitals, private companies that handle sensitive data, etc...

The software doesn’t come from nowhere

I mean IMO that's the heart of the whole debate. Making a modern LLM requires massive amounts of data and computation... both of these do have to come from somewhere. And my understanding is that, current AI/LLM research didn't create that much technological breakthrough, so building a better model often does indeed rely on either using more data or bigger architecture (which takes longer to compute)... which then leads to a truckload of ethics issues here and there

For private companies, we know that OpenAI is blatantly hoarding data from just about everywhere, see the recent Mathematics Millennium Prize scandal. A lot of the open-weight models are trained by Chinese training houses, and Chinese tech companies are not known for respecting intellectual property... and China uses lots of coal energy still.

Maybe Mistral is more ethical, given that 1) they are based in France and care about GDPR and data regulations a lot more, and 2) France uses lots of nuclear energy, but I can't say for sure. An European company operating an AI/LLM training house from Iceland using their excess geothermal energy is probably the most ethical we can get on this

In theory (and take this with a whole jug of salt) someone can run an off-grid farm and train a small LLM all on their own, which would be very ethical... but I doubt any serious organizations are doing it this way. Most ppl just want better performance and/or more optimized models for now

For a complete FOSS purist with an environmentally conscious angle, I don't know if there is an AI/LLM that can be considered 100% ethical... but then again Gentoo is a thing. So yeah, I don't know

I've come round to the idea that I will not use LLMs regardless of how "ethically" they can be sourced because their outputs are harmful in a way that builds up, like a very small dose of poison.

I'm not holding this as a hard stance, just a current analysis of the technology and how it might affect my life in my circumstances. I think we're well into a world where we need to be deciding if some technologies don't suit our lives, because there certainly are even more harmful technologies to come.

It might seem like a romantic or artist's approach, but with so much spiralling out of our control I'm trying to make the active choice to be more human.

In leav­ing progress to the machines, in letting technology go forward on its own terms and selecting from it, with what seems to us exces­sive caution, modesty, or restraint, the limited though completely ade­quate implements of their cul­tures, is it possible that in thus opting not to move “forward” or not only “forward,” these people did in fact succeed in living in hu­man history, with energy, liberty, and grace?

Always Coming Home - Stone Telling Part 3 by Ursula K. Le Guin

Ok seeing your edit, some basic explanations:

A Token is a small unit used by LLMs. Mostly not a full word, more of a word fragment of a few characters.

A Dataset is an accumulation of different data (mostly real-life data to avoid problems) which is used to train a Model.

A Model is basically a huge file, full of Tokens that are interconnected with each other.

Open Weights means that the numbers which describe the relationship between Tokens (this is all basically a ton of matrix multiplication, which is the reason this is using so much power) are available.

If you want to really understand, 3Blue1Brown has a complete series about neural networks!

In my book, not paying and supporting them financially - > ethical

My issue is when you pay for a subscription, when you buy a gpu specifically for ai, when you buy a model, etc. Because then you are funding the slop machine.

They call it open-weighted and not open-source because they don't have the training material (stolen books lol) but still want to pretend they are l33t hackers not bound to BigTech.

Yes, it's possible to use it locally with a $2000 computer (that's for the cheapest ones) but you'll only get a few words per second. It doesn't matter to the vibe coders who don't know how to code.

The software to run that is open-source but the training of the model (the "open-weight" black box) requires to destroy the environment at least once.

Last but not least the free models are obviously censored but people don't care about censorship anymore for some reason. "Tiananmen didn't happen? Not my problem" without understanding that more is hidden.

Anyway, no, there is nothing ethical about it.

Some companies build out more electrical capacity than they use. Substations are funded by the DC and built to twice the desired capacity - and half is delivered to the community. The generators are run only when utility power goes down, and run on diesel.

However, not all companies are this responsible and the grievously irresponsible ones get the press and make the responsible companies look bad.

I don't know if 5 year olds are allowed to watch 1.5h videos, but if so: this one has all the pro and con arguments explained nicely and makes a point why it's not a good idea to shame people for using LLMs.

https://youtu.be/y85nqc2zm7M

Not an expert but I've read up a little.

One can pay to license material for LLM's (Google at. al. are not fans of this ofc.), or simply only use stuff in the public domain or your own internal material.

Private/local models exist for various use cases like not wanting even the chance of your stuff hitting the cloud. Ex. https://ollama.com/

If you have a 4070 or so you can run a smaller model without a issue, which is well within hobbyist range, but larger models quickly balloon hardware requirements to the point where you're putting in some bucks.