this post was submitted on 03 Nov 2026
257 points (92.4% liked)
Privacy
51833 readers
1185 users here now
A place to discuss privacy and freedom in the digital world.
Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.
In this community everyone is welcome to post links and discuss topics related to privacy.
Some Rules
- Posting a link to a website containing tracking isn't great, if contents of the website are behind a paywall maybe copy them into the post
- Don't promote proprietary software
- Try to keep things on topic
- If you have a question, please try searching for previous discussions, maybe it has already been answered
- Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
- Be nice :)
Related communities
much thanks to @gary_host_laptop for the logo design :)
founded 7 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
It's not realistic right now to recommend an offline LLM due to average hardware. Sounds nice though.
If you can selfhost but don't have the hardware for running the LLM itself. I currently have OpenWebUI setup linked to openrouter. Definitely not fully private but I've chosen only "Zero Data Retention" providers. Then on top of that be vary careful what you're asking.
I have a lower-end GPU (8GB VRAM) and I can still run smaller models on it with decent performance.
Dumb question: what do people use those local LLMs even for?
Not much, but you can use it for basically anything people use frontier models for, it'll just be somewhat dumber. I occasionally run LLMs for proof-reading important things that I've already hand-written and edited. Sometimes for bouncing off ideas. Sometimes for the kinds of questions I'd normally chuck at a forum but I can get a quicker answer if I ask an LLM. No, I don't "trust" the answers; anything where I can't just use my own judgement, the LLM will have given me enough to perform my own research to fact-check it.
What does "lower-end" mean in this case? 8GB VRAM could be anything from a 2014 Radeon R9 290X to a 2026 RTX 5060.
Radeon RX 7600, but yes, you're right, it's relative. It's good enough for any gaming I do, which is not an awful lot. I think it's considered mid-range for its generation, I guess I was just thinking of the GPUs I see people mention they have which seem much beefier than mine lol.
I'm no AI expert but when I looked into it, it seemed that VRAM was the biggest factor being mentioned in discussions of GPUs, hence why I mentioned it.
I could be wrong, but I think there are differences not just in FLOPS but perhaps more importantly in what kinds of functions are supported (with newer GPUs becoming gradually more general-purpose), so I'm not sure if the GPU from 2014 is capable of running AI models at all (slowly or otherwise).
I actually kinda want to find out, because my old GPU is a Vega 56 (which is also 8GB) and I'm not sure if it would be useful or not.
Strata makes it workable with low vram cards like 8 or 12gb.
I’ve been using Atomic Chat on my archaic iPhone 11 and it works well enough for my needs. Haven’t had much luck with several others, but that one works fine.