Selfhosted

No, I have taste.

I started running LLMs a couple months ago on my own hardware. I have a Framework Desktop that I ordered last year and also recently picked up a refurbished 24GB AMD RX 7900 XTX which I'm doing some performance testing against. The dGPU is much better for dense models, and slightly faster for MoE if I'm willing to run them at a lower quant -- but uses more power and has annoying coil whine. The Framework Desktop uses ~100W under load, is quieter, and for the MoE models already runs them fast enough for most of my needs -- so most of my LLM use happens on that system still.

For software: I'm using ollama on the Framework currently, but I want to replace it with just using llama.cpp directly eventually. I've been using llama-cli for testing the dGPU. I wrote my own chat client to interact with ollama as well as a few other programs for specific tasks.

I've been using the LLMs for a mix of research (both personal and professional), entertainment, practical coding tasks (mostly debugging and brainstorming, plus a bit of UI prototyping, automatic generation of sequence diagrams for documentation, and light scripting), as well as automation of tedious tasks.

As an example of the latter, people often send me requests to prepare data sets by email but don't specify the sources they want precisely so I have to go match the name against the real name in our archives; LLMs are great for mapping the imperfect name -- with typos, missing prefixes, incorrect addition of spaces, addition/removal of hyphens, etc. -- to the exact name I actually need to pull the data off disk when given a lookup table to compare against.

As far as models go, I'm mostly using various Qwen 3.6 and Gemma4 variants. I have multiple versions of each for different purposes. llmfan46's uncensored Qwen 3.6 35B-A3B @ Q6_K (from Hugging Face) is my default model currently.

i don't use it at all, i do want some selfhosted speech to text model (whisper?) but my computer is ancient so it would be awfully slow. i have some multi hour audio recordings from presentations, would be nice to have them in text and searchable..

I have a simple slow model running on CPU in my cluster for karakeep. I've tried running a variety of models on my 7900XT but even with 16GB their performance just isn't there. My new work m5 Mac book with 48GB of ram is the first time I've seen usable performance for local models and it has been pretty impressive.

Well, I don't exactly host AI. But some of my software uses AI and/or machine learning. My photo gallery does face detection, I've installed text to speech and speech to text. My Home Assistant has a voice satellite (which is a poor-man's Alexa because I lack the hardware to do voice recognition in realtime). And I also regularly try some large language models and chatbots. But I don't have any real application (yet). And it's slow without a proper GPU. So I'm more or less just messing around. Currently that's with Ministral 3.

I installed LM Studio just for fun on a 6800XT. But it was even less useful than the web-based ones.

I've played with it for Home Assistant integration, but I just dont have much interest in it, the whole thing is too inefficient at the moment, and the tiny models that can run in a few gigs of system ram on an ipgu or npu arent good enough in quality or speed to rely on.

Hopefully some future generation micro-models will be more useful for the way I want to use it (aka , ultra light, no dedicated hardware etc.), but for now it's a lot of compute resources, plus heat and energy for a gimmick.

Bought b70 with egpu enclosure and usb4 connection wasn't really planning to actually run anything but now ended up with llama.cpp with openwebui - kids/parents want to/have to use chat, might as well provide local solution than them using industry options. Also started with ollama and Gemma 4 26b a4b - asked it to write script to setup llama.cpp in container.

My server is way to weak for that unfortunately. I run some llms on my laptop with ollama but it's not particularly effective. I use it to run dolphin series models when k need an uncensored LLM I have tried running some of the coding models but they just aren't smart enough on my level of compute for any useful work so Ive ended up just paying api prices on open routers.

I'm running dwarfstar which is a 2 bit deepseek v4 flash. It's quite capable even at 2 bit.