Ask Lemmy

I run models locally on my 5070 i bought a year back for around 550€ (now costs around 900€ - insanity). Yes, it runs completely locally with pretty good results, and i am using an AM4 platform, which is now a few years old, and was also able to run smaller models with my 3070Ti with good speed.

My viewpoint regarding the models themselves is that since everything on the web has been used including everything i have put on the web in the last decades, there is also not much of an ethical problem here when using these things in a non-commercial setting - i am not making any money with it and i am not disseminating the output, and therefor i also do not cut into the profits of any creator (i mainly use it to automate tedious tasks or to get a complicated RegEx/sed/awk situation under control without breaking my brain - no cultural output like text or images). On a larger scale i would prefer it if models and datasets were under UNESCO stewardship - free for non-commercial use, with licenses being sold to corporations and organizations, where the income from the licenses get distributed to the people providing for the datasets (in this case in an opt-out basis - these models already represent a cultural snapshot of humanity, and the assumption is that artists WANT to add to human culture) or for financing young artists.

You need some decent hardware to do it, but yes one can absolutely run capable local models. The 7-8B models run fast and are probably decent for light coding, but they make piss poor chat bots. 27-36B models get much better outputs but take more hardware, and still not great in a chat interface (but far better than 8B). They run slow enough on my MacBook that it's just a toy for fun, despite unified memory. I aspire to having a local AI server then can run models in the 100B-350B range. It's a pricy bit of hardware, but I can leverage it to do all the things one might want to do but avoid for data privacy purposes. I try to avoid oversharing, but when I look at what ChatGPT knows about me, I'm giving way more away than I want to. I look forward to the day I don't have to use it at all.

Ethically is a slightly different story; there are a number of claims people like to make:

Plagiarism: not in any significant sense. Like any of us might borrow a turn of phrase, so might AI, but any significant body of text — even a paragraph — is very unlikely to match exactly to an ingested work. That's why they need to ingest so much different source material — so no one author or text dominates the output.

Stealing: Mostly yes. I think there is a significant difference between ingesting all of Reddit (where people including myself were sharing their words with the world) and feeding in copyrighted works of published authors without permission. I think when you turn around and sell access to a model built on material you didn't pay for, that's wrong. I do think open models available to everyone are far less evil because they take in information they don't own and give back a tool they don't own, having contributed significant money in the form of training.

Infrastructure: Yes but... that's how everyone does it from Walmart to sports stadiums. Don't single out data centers when the entire system is corrupt. Companies should pay for transmission lines and capacity they expect to draw. All companies, not just AI.

Environment: Yes but... the amount of water used sounds high on a global basis, but it's utterly dwarfed by agriculture and some other industries. Now some say, yeah but we have to eat, but do we have to eat almonds specifically? Beef? Do we have to grow food in deserts? Data center water usage looks big in raw numbers, but it's not in percentage terms. That being said, it can be a huge problem locally where a community might have a sustainable water supply but a big data center tips the scales.

The latter two are IT industry issues that are driven by AI but aren't going away when and if the bubble pops. Imagine the privacy nightmare that is coming. In fact, I'm not convinced that AI isn't just a cover for unprecedented data collection.

Is it really feasible to run it 100% locally? I know there's plenty of people with very powerful rigs indeed, but still.

It's absolutely possible, and done more and more. Smaller models exist, and run on sub 8GB VRAM without any issue. If you have a gaming setup, you can run the bigger models without any issue locally. You don't need Astra or Mythos or whatever. As a daily driver for "light" tasks (e.g. summarizing a document) small models perform without a significant difference to frontier cloud models. For bigger tasks (e.g. coding or agentic workflows) you need a bit of beef on your graphics card, but it still runs on regular consumer hardware. Anything above that is not needed for any normal use-cases.

If yes to the previous: the software doesn't come from nowhere and ultimately still relies on gas-turbine-powered datacenters and stolen IP and stolen personal data, no?

This has multiple layers:

• for running models locally, FOSS solutions exist and are no different to any other piece of software
• models themselves are usually trained on beefy data centers and questionable data, but alternatives exist, both for training and curated data sources. Second but - those usually underperform, because LLMs need those massive data sets. Limiting the dataset limits the output quality drastically.

If what I wrote above is true, what exactly are people arguing when they say it's still possible to use LLMs ethically or true to FOSS philosophy, because ... ???

Also multiple answers:

• ...because first there is a massive amount of hardliners both in the pro- and anti-LLM camp, and differentiated and objective views are rare. Ethical software and "true FOSS" are also very heated topics currently.
• ...because, while possible, objectively non-ethical use of LLMs does happen more (public vibecoding of products, weaponized use in cyber and conventional warfare, use in surveillance, etc.).
• ...because there are absurd amounts of money circulating in the AI bubble, it's too big to fail, and ethical and for-profit usually do not match.

I have a friend who runs his AI locally because he doesn't want to pay for tokens.

Is it really feasible to run it 100% locally? I know there's plenty of people with very powerful rigs indeed, but still.

There are small models you can run 100% locally on a mediocre computer or even a phone, and I mean you could be 100% offline and it will still work.

The better local models require a higher end gaming GPU or an ARM Mac with a decent amount of RAM, but nothing too extreme. You could get a computer to run them for about $2000 to $3000.

the software doesn't come from nowhere and ultimately still relies on gas-turbine-powered datacenters and stolen IP and stolen personal data, no?

The small local models typically start with a full infustrial-size model and then they “distill” it to smaller models. The concerns over training data are still valid. I’m not sure how much the concerns over power usage are about training vs running the full industrial size models commercially. Running a small local model uses a tiny fraction of the power it takes to run an industrial size model, and the training is a one-time cost (except these companies never stop training because they want to make next year’s model better and faster).

I do think it’s worth pointing out that running a model locally keeps your conversations with it private. Remember, it can be done 100% offline. So your data won’t be stolen.

There are ethically-sourced datasets used to train LLMs that can be powered by renewable energy and cooled by closed water systems. Maybe the question becomes "sure, but are those good for anything?" and the answer to that is "probably not right now".

Is it really feasible to run it 100% locally?

Yes. My mid 7-year-old rig featuring Ryzen 5 1600X, 16 GB DDR4 RAM, and GTX 1060 6gb can run fairly intelligent models that are actually practical to use.

The software doesn't come from nowhere

Correct. Training the model is the most resource-intensive and planet-churning part.

The datacenters would be built next to wind farms if it wasn't for oil industry bribes. It's just a scape goat.

Gas turbines need to be outlawed outright and we need proper regulation for all industries. They focus on getting everyone pissed about AI so you don't start talking about hanging oil execs.

"Open weighted" I take to mean that the user can see what sources the LLM/AI was trained upon. Usually this information is a closely guarded corporate secret.

As for "models" this is no different than the different models that a car company offers. Sentra from Nissan, F-150 from Ford, Civic from Honda

If locally run or through a data center, LLM/AIs have two major problems:

1. The results can't really be trusted due to their tendency to hallucinate.
2. Frequent LLM/AI usage has been proven to deskill the user as well as a reduction in cognitive ability. I'm dumb enough without getting help from a clanker.

Assuming "run it 100% locally" means on your own hardware with network access, I see no reason this can't happen given a powerful enough computer.