No Stupid Questions

You should try it out. Ollama is what you’re looking for and it’s easy to set up.

Yes, I don't have a very beefy graphics card but I can run LLMs locally. I use them for things like proof-reading. Look into llama.cpp. If you have an Nvidia card then look into using CUDA so your LLMs don't run slow as fuck using only your CPU.

Check how much VRAM you have and then do some research into the open-weight models. The best models require quite a bit of VRAM, but there are models that work well on cards with less VRAM.

It's energy-intensive, but on a level that most people already find acceptable. I'd say it's comparable to playing a AAA video game for a few minutes to generate an answer.

Local LLMs aren't slower if you pick a model well-suited for your GPU, but they are dumber. But that doesn't make a difference for me as I wouldn't use LLMs for any particularly cerebral tasks e.g. programming. They can work fine for things like bouncing off ideas, proof-reading, making suggestions for things like writing and debugging, etc, the kind of thing where you can read their response and just discard it if it's stupid. Whether or not that's "useful enough" for you is down to you. You can just give it a go and uninstall things if you don't find them useful.

Can I just slow it down

It will be slow because it is slow. And no, you cannot unless you like reading one word per second.

I've run local models on my work laptop, whose specs I don't know by heart, but it is a relatively ordinary laptop given out to software developers at a large company. It wasn't instant, but it did produce some useful results. Look into ollama.

Starve the beast by VPN hopping on opencode free models. It's become a little more difficult in recent weeks because they are now identifying people doing this via MAC address, but it's still fully possible.

Yeah, I'm running a mix of three 8B LLMs on an old MBP from 2018. Its not perfect, and it's slow, but when you know how to use it, it works well enough to keep it going.

There are models that will run on a smartphone. They are not as capable and have smaller context windows, but they work.

You can get models that will work on that hardware, like Qwen 3.5 9b model. What are you using the LLM for?

microsoft and google were already getting things from search. duck.ai claims to annonymize your queries.

Not really, no.

You can technically do this, but it’s going to be so expensive and so much work, it’s not going to be worth the effort, especially if you already think your interactions with LLMs haven’t been worth it.

To make a self-hosted LLM worth your time, money, and resources, you’ll need to make it basically a part time job.

Not really. Maybe the smallest models could work but they are severely limited in their capabilities. Depending on your use case it might be enough though, but not if you expect to be able to run general purpose tasks or code generation.

At the end of the day there is a reason the AI companies are building such massive datacenters. The cost of the highly capable models is massive compute and RAM requirements, and the commercial options are going to be quite far behind the proprietary models.

An option if the main reason is to avoid US tech companies could be to find a vendor that sells hosted open AI models. There are quite a few hosting providers that offer model as a service options. For instance Scaleway, StackIT, Evroc, OVH, and probably more. (It’s not like they magically don’t need datacenters though)

Why would you want to use them to begin with? Like what do they provide that you can't do yourself?

AirLLM GitHub repo