Selfhosted

I just got qwn3-4b-instruct working on an Intel A750 with 8 gigs of vram.

I use it to power Home Assostant/Voice Assistant on a few M5Stack echos around the house to replace cloud based smart speakers.

I wouldn’t say I ‘run’ it, but I mess around with it

I have a base level Mac Mini I got before the price increase a few years ago, also from when models were smaller and my ambitions were less. I’ve contemplated upgrading it, but I have other things to spend my money on.

M4 processor, 16 gb ram, 256 gb storage, and a 4tb external HDD drive (an old NAS drive).

It’s fine. It’s mostly a docker machine, but I do throw AI tasks at it a few times a week. Mostly stuff that I just let it chew on for 5-10 minutes and come back to.

I run openwebui, and have several models I bounce between. I mostly run Gemma4 (Google) and Granite4.2:8B (IBM).

Editing to add: the self hosted version of all these models, via llama, just in case that was unclear.

I've been using NixOS/nix containers to host some llama.cpp servers and Hermes Agent. Though I wrote a flake to get Project Maya working so I can serve GLM-5.3-Flash and in legit blown away by how well it works. It's a fork of strata/llama.cpp that focuses on where the relevant experts are loaded and has a neat dashboard to track their use. https://github.com/mw00/project-maya

My server is assembled into an openair mining rig with a Tesla P40, RTX 3080, and a few Tesla V100s. All of which I bought used off eBay.

I purchased my first domain to self host a matrix server that I can access via my phone and I also have tailscale set up so I can ssh into the server via my laptop while traveling

As a service? No, not really.

I use llms for specific tasks, usually small models, for things like identification and tagging (immich, paperless), translation or transcription (whisper), etc. I have a bunch of larger models on a Mac mini and my work laptop for various testing purposes, and have used them to make summaries and other such things.

They aren't great for search IMO, and search summaries can be quite bad.

I do find them useful to interrogate an unknown rs232 protocol or similar, though, which is where larger models come in handy.

Corporate LLMs are generally garbage to me not only because they are just data farming, but also because they really aren't good at everything. They might claim it, but the reality is a tailored model is often the better approach.

Using openwebui as a front end for llama.cpp is a decent strategy, combined with sub-agent routing (so generic model hands off to tailored models then returns the right info). This can be useful for some scenarios.

A caveat on coding - IMO, they are mostly bad at it. If you are using it to generate a small part of something that needs to be repeated they are handy, but at scale they aren't great, meaning for the entire project (or even an entire plugin or module, depending on complexity). Maintainability becomes a serious issue. While they have improved somewhat, there are limits to what they can do.

Macs with M series chips are good options for a hardware base since the memory is shared, and IMO is one of the most cost effective options over a dgpu with a ton of memory. A dgpu with a decent bit of memory (say 8-12GB) is a solid choice for a model thats larger (27b with 3 or 4b active as a mixture-of-experts model) especially if you have an iGPU for regular graphics tasks. Since its all shared on the M series macs, just get something that fits entirely in memory there.

My server's old GPU (8gb) can't run anything useful LLW wise I don't think.

Have run some stuff locally on my main computer, but nothing setup as a proper service.

Qwen 3.8 flash next on a 4090, using strata instead of llama.cpp, gives me about 45-55 tokens/sec. Mainly used for coding tasks. The model size is perfect. If the qwen 4.0 models can be a bit less wordy in reasoning but the same size and quality they’ll be really strong.

Played around with it on my desktop machine. Someone's distilled variant of Qwen 3.8-27B-Q4_K_M, on a intel12700k + RTX3090 24GB is what I use.

I haven't adjusted the reasoning down from very high but I mostly use it for trying tweaks to half-finished coding projects I've lost motivation to complete many years ago. I used it to develop a very simple JS-free personal website. I have secure shell which I can access from Termux or PuTTY, but sometimes I have the llama.cpp webui insecurely accessible online for one day at a time just for fun.

Framework desktop

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

4 acronyms in this thread; the most compressed thread commented on today has 6 acronyms.

[Thread #135 for this comm, first seen 17th Oct 2026, 13:50] [FAQ] [Full list] [Contact] [Source code]

I use llama-swap with a couple of different models, one 8B as a general chat bot, with offloading to several MoE-models for specific workloads. It's all run on an old dell xps with 64GB ram.

Mine is used for code reviews, general questions, coding assistance and for anything else I can think of.

I have been running a very simple setup but it is more impressive than I ever expected a local model to be.

My PC:

Refurbished 6900XT (16gb vram)

"Cheap" 32GB DDR5 ram

Linux Fedora

Unsloth Desktop. You can connect over cloudfare if you want, or local network, etc

Qwen 3.8 27b GSQ-RCO IQ3_XXS with 65k context @ q8, mmproj not on the GPU, mtp and fa on. I get ~50 t/s max. This is about the maximum this old GPU can handle without unplugging the monitor or slowing down the model too much.

tbh, I've been messing around with it mostly. I have been interested in its coding ability, so I have had it create some code at a few different levels of complexity. Most of these tests were nothing crazy, like a simple harness to connect to unsloth localhost models, and some HTML design tests. I have it building a UFO exploring the solar system type game, it'll be done like a week after qwen4 27b comes out!