What tools and projects do you use for your local LLM chats and agents?
I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
You must log in or register to comment.
php server and a local certificate authority.
Literally just having ollama installed and some models that work well with the amount of VRAM I have. I don’t auto-start the daemon because I rarely use it. Occasionally I use it for proof-reading/feedback on things like cover letters, or debugging. I don’t let it write anything. If it has something useful to say it can tell me in the chat and I can implement it if it’s good. Usually I don’t need it though so it just sits on my disk for the most part.
Currently, I’m using Hermes and OpenCode mostly, backed by Ollama running a couple of different models. Usually I’m running the Gemma4 or Gwen3.5 models.
Memory management is done through Obsidian as every harness seems to have really good skills for it.
Mind elaborating how you route them through obsidian?
My Hermes agent and OpenCode both can see the same Obsidian vault. I just started working with OpenCode and haven’t quite figured it out, however my Hermes agent has learned that everything I work on needs a file in Obsidian and that if it needs to look something up that it should look through Obsidian first. Prompts usually tell them to “look in the vault” or “write up what we’ve been doing in the vault, suitable for another agent to understand what is going on with no prior context”.
It’s not currently all that reliable, but I’ve only been working with Hermes for a few months and started playing with OpenCode a few weeks ago. I need to work a bit more on orginization, though. Last time I looked in there, I backed out quickly as it was a god awful mess.
vLLM to host and openwebui for a frontend that I really only use to demo models to friends.
harness
Opencode running with full access inside a rootless podman container with only the directories I want mounted (usually a single project, sometimes read only).
So far though, I’ve only used it to document and report on project repos because Qwen is not that good at actually coding (although it can give pretty decent conf file examples, but not always optimal).
Also useful for parsing pita data dump type files that I’m too lazy to construct multiline awk commands for.
It depends on how complex you want to make it.
The simplest way would be to use a GUI tool like LM Studio, it has the benefit of selecting the model quants that are likely to work on your hardware and having sane defaults for the various parameters that you can work from.
Typically you’ll want to increase the context window, the default is good for short one-off questions but if you’re going to do RAG or coding tasks you will need a much larger context window. You will typically find this through trial and error, increasing until you hit memory limits or notice a large slowdown on tokens/second.
You can also have the model respond to an OpenAI-like API interface on a designated port/interface if you want to run non-chat tools through the same model.
Then, depending on the task that you’re trying to solve, you may want to find an MCP server built for that task which can provide the ability for the AI to interact with tools in a way that’s more controlled (giving an AI raw terminal access on a user’s machine is typically a bad idea).
If you need something more flexible that you can incorporate into scripts and automation, llama.cpp has been popular for a while. Like Linux distros, everyone is going to have an opinion on the best and whatever you decide on will be the wrong choice. There’s also Ollama and vLLM if you want to run a datacenter.
Model-wise it completely depends on the task. Do you need a multi-modal model that supports vision, is it going to be coding, is it trained to use tools/MCP servers, does it use reasoning, etc. The Qwen models are open weight and available in many different quants. The version 3.5 models run well on consumer hardware and can use vision, reasoning and tools.
I’m running it through Llama.cpp but it will likely be available via LM Studio as well.
Ollama on an N150 mini PC, 16GB, no GPU, Open WebUI on top. Being honest, 1-3B models are fine for searching my offline Kiwix stuff and 8B works if you don’t mind waiting. I’m turning this into a small box (heyeddi.com/projects/doomsday-box) and I keep going back and forth on whether people even want AI on it, or just the media and offline archives. Would you?
This community won’t be much helpful but there is one specific for local LLMs. I think it was ai_ and some other…
Openwebui pointing to llamacpp running on an old ass GPU. It’s good enough for what it is.
I’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG.
Pi Coding Agent on the laptop, openwebui as a self hosted service.
I don’t host LLMs myself. My best hardware has 16GB of RAM, I could run a 4B model or so, but what’s the point? So I buy compute with ZDR turned on.
