I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.
If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…
ollama
Shittiest local inference engine. Use llama.cpp instead.
qwen 3:4b-instruct
Old version 3 model from Alibaba. 4 billion parameters total. It will struggle with anything other than natural language generation.
on 6vCPU and no GPU
Running on a 6 vcore VM with no GPU pass through.
TTFT
Time.To.First.Token. How long it takes the model to start responding after you hit enter. Dependent on not only processing speed but also context length.
prompt is uncached
Cold start = no cache = slow. But faster when mid conversation and the key-value cache is hot.
100tps
100 tokens per second. 1 token is roughly 1/4 word.
Use open weight models either small enough to run yourself or via your diverse online host of choice.
without supporting the data centers
online host of choice
I believe there are options to host open models on hardware you don’t own without having to give money to the “evil” data centers. Getting away from data centers entirely is impossible as that is how the internet has worked for a very long time now.
Not to mention that it’s training and not inference where most of the compute is going anyway.
Srsly, unless you grow all your food and make all your possessions, you’re constantly supporting billionaires. I mean sure, run your own LLM, but don’t let it create a delusion of ethical purity.
There are 3 or 4 billionaires in my country. Founders of Wise and Bolt. None of them own farms or grocery stores.
Matter of fact, the most popular grocery store chain is coop so most of the “owners” (members) of your local cooperative aren’t even millionaires.
Download Ollama. Download one of the smaller Qwen models. Have at it.
Try qwen3.5:9b (a 7.6Gb model) to start.
Friends don’t let friends use Ollama.
You can, but it’s not worth the effort. The technology itself is set up for people of power, such as governments and corporations. That’s why they steal our hardware and make it unaffordable. The only genuine use is accessibility, but it’s purely audio/text recognition. Everything else is for targeting people with drones and surveillance state.
Edit: how could I forget deepfakes? You practically can’t live without them nowadays /s
The only genuine use is accessibility, but it’s purely audio/text recognition.
On that note:
LocalTranslate is an offline translation application that uses Firefox’s neural translation models (from mozilla/firefox-translations-models) to perform high-quality translations locally on your device.
Wild conclusion
I run a “server” (old 9-year gamer laptop) with an 8 GB Geforce 1070 Max-Q. It’s currently running full GPU-offloaded stock llama.cpp, with Unsloth’s Qwen3.5 9B at UD Q4_K_XL (good enough quantization for coding assistance and short agentic tasks) and a 90k context at around 300 t/s prefill and 25 t/s decode, which are good enough speeds for daily use.
It can’t vibecode GTA6, but it can help with quick scripting, research and things like that.
You can run very small models with Ollama or Llama.cpp. It really depends on the type of work you want to do, but these small models won’t do anything super complex. In my opinion the models don’t start getting really good until around the 30GB size like Qwen 3.8 27b. You could probably still use a small model for some chat, search, rag, maybe simple script writing.
Lumo and some on duck.ai seem fine for the free usage I do. Here’s a fun one that appears accurate, enjoy! duck.ai/share/01cee404-a240-4c14-9a83-584d406c5a68#Ch22BaAoEAejJEeRNZYh-MfEYLFoDCsVZsPd23IGU5w
The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier
Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age
What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai
Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware
trying to code
copy pasting to and from duck
That’s, uh… The worst way to do any of this. And GPT-6 Luna is the weakest GPT-6, if you’re going to run something that’s on someone else’s hardware anyway, you can just use openrouter and choose ZDR hosters of open-weight models that are somewhere between GPT-6.1 Sol and Terra in accuracy.
OP said they didn’t want to support billionaires and data centers
The money used to make zdr requests on openrouter to good models like GLM 5.3 flows into large inference providers like fireworks and baseten
Yes, copy pasting is a terrible way to vibe code. But my point is it’s still going to be better than trying to ollama with a 1060 and 16 GB RAM
I don’t think I’ve seen this recommended yet, but maybe you can check out CoCore https://www.cocore.dev/. I saw a demo of this some time ago and it’s still pretty much an experimental project. Essentially this is more of a community shared way to run local LLMs, especially for those without necessary compute. You are basically using an LLM hosted on another person’s computer in an encrypted manner. There’s a lot of technical detail about it that goes over my head. It is slow, and of course, still requires trust, or some auditing to verify it. But I think it’s kinda a cool way to crowdsource resource to run. The network of users is on fediverse btw.
Unsloth Studio is one of the easiest and most approachable methods of doing what you want. There are some smaller models that will run fine on the hardware you described, but they won’t be particularly “smart”. Still, they might work for what you need.
It’s important to bear in mind that as we move forward, these open weight models will just get smaller, better, and more efficient. Data centers are absolutely the wrong approach for this tech, and the only reason it exists that way now is due to capital interests trying to steal all the value. It won’t last as the technology matures, so go ahead and at least familiarize yourself with some ways to run local because that is the future.
Totally: You can run stuff on an even worse specced potato computer! It’ll just be slower, possibly to the point where you don’t really find it to be worth it, but you could still run local models. You won’t really be slowing it down, it’ll be the hardware limitations, but there are plenty of models that can work. There are specific model suggestions in the other comments but maybe give folks some more info on what exactly you want to do and you might get some more useful info.
Maybe write some scripts for task automation, I am not really sure. As I said I haven’t really used it before and don’t really buy into how awesome it is. So I just kind of want to play around with it, without giving all my data away or spending money to upgrade my computer for something I likely won’t use often or at all after testing it out.
I’d put it like this: If you already know what you’re doing code-wise, “AI” (read: LLMs) can potentially be helpful but you should vet absolutely everything it does and assume (probably correctly) that it has fucked up in some way.
If you do not know how to code, it’s probably not a great idea to trust it with anything important, but if you’re just doing it as a learning experience, that’s OK. Just don’t fall into the trap of letting it think for you: It is not a replacement for learning how to write code.
It’s something that has been wildly oversold in certain ways because the people running it have a financial incentive to hype it up, but used correctly and with caution, it can be potentially useful in the hands of someone who already knows what they’re doing.
Yeah it would be 100% for fun and or learning. Coding is not my profession, and I have imposter syndrome in my actual profession… Lol. I certainly wouldn’t trust any code I or an LLM wrote to provide it to someone else or contribute to anything.
For those who wish to avoid big tech, and don’t mind potentially being on a data center: openrouter lets you use w/e model you wish (even free), paying per token, from whatever provider you wish (in particular, smaller companies, folks self hosting, etc.)
Some datacenters are built in terrible places, but not all.
OmniRoute is also pretty nice for that.
