I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.
If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…
Use open weight models either small enough to run yourself or via your diverse online host of choice.
Srsly, unless you grow all your food and make all your possessions, you’re constantly supporting billionaires. I mean sure, run your own LLM, but don’t let it create a delusion of ethical purity.
You can, but it’s not worth the effort. The technology itself is set up for people of power, such as governments and corporations. That’s why they steal our hardware and make it unaffordable. The only genuine use is accessibility, but it’s purely audio/text recognition. Everything else is for targeting people with drones and surveillance state.
Edit: how could I forget deepfakes? You practically can’t live without them nowadays /s
The only genuine use is accessibility, but it’s purely audio/text recognition.
On that note:
LocalTranslate is an offline translation application that uses Firefox’s neural translation models (from mozilla/firefox-translations-models) to perform high-quality translations locally on your device.
Wild conclusion
You can run very small models with Ollama or Llama.cpp. It really depends on the type of work you want to do, but these small models won’t do anything super complex. In my opinion the models don’t start getting really good until around the 30GB size like Qwen 3.8 27b. You could probably still use a small model for some chat, search, rag, maybe simple script writing.
You can get the TernaryQuench version of Qwen3.8:27b to run great on a GPU with just 16GB of VRAM. I started using it last week and it’s been amazing.
I have a 4060 Ti 16GB and was using the 14b version for a long while but that ternary conversion tech is just enough to squeeze the full 27b version into my card with plenty of room for context 👍
Lumo and some on duck.ai seem fine for the free usage I do. Here’s a fun one that appears accurate, enjoy! duck.ai/share/01cee404-a240-4c14-9a83-584d406c5a68#Ch22BaAoEAejJEeRNZYh-MfEYLFoDCsVZsPd23IGU5w
I run a “server” (old 9-year gamer laptop) with an 8 GB Geforce 1070 Max-Q. It’s currently running full GPU-offloaded stock llama.cpp, with Unsloth’s Qwen3.5 9B at UD Q4_K_XL (good enough quantization for coding assistance and short agentic tasks) and a 90k context at around 300 t/s prefill and 25 t/s decode, which are good enough speeds for daily use.
It can’t vibecode GTA6, but it can help with quick scripting, research and things like that.
Download Ollama. Download one of the smaller Qwen models. Have at it.
Try qwen3.5:9b (a 7.6Gb model) to start.
Friends don’t let friends use Ollama.
I don’t think I’ve seen this recommended yet, but maybe you can check out CoCore https://www.cocore.dev/. I saw a demo of this some time ago and it’s still pretty much an experimental project. Essentially this is more of a community shared way to run local LLMs, especially for those without necessary compute. You are basically using an LLM hosted on another person’s computer in an encrypted manner. There’s a lot of technical detail about it that goes over my head. It is slow, and of course, still requires trust, or some auditing to verify it. But I think it’s kinda a cool way to crowdsource resource to run. The network of users is on fediverse btw.
Totally: You can run stuff on an even worse specced potato computer! It’ll just be slower, possibly to the point where you don’t really find it to be worth it, but you could still run local models. You won’t really be slowing it down, it’ll be the hardware limitations, but there are plenty of models that can work. There are specific model suggestions in the other comments but maybe give folks some more info on what exactly you want to do and you might get some more useful info.
Maybe write some scripts for task automation, I am not really sure. As I said I haven’t really used it before and don’t really buy into how awesome it is. So I just kind of want to play around with it, without giving all my data away or spending money to upgrade my computer for something I likely won’t use often or at all after testing it out.
I’d put it like this: If you already know what you’re doing code-wise, “AI” (read: LLMs) can potentially be helpful but you should vet absolutely everything it does and assume (probably correctly) that it has fucked up in some way.
If you do not know how to code, it’s probably not a great idea to trust it with anything important, but if you’re just doing it as a learning experience, that’s OK. Just don’t fall into the trap of letting it think for you: It is not a replacement for learning how to write code.
It’s something that has been wildly oversold in certain ways because the people running it have a financial incentive to hype it up, but used correctly and with caution, it can be potentially useful in the hands of someone who already knows what they’re doing.
Yeah it would be 100% for fun and or learning. Coding is not my profession, and I have imposter syndrome in my actual profession… Lol. I certainly wouldn’t trust any code I or an LLM wrote to provide it to someone else or contribute to anything.
Yes. LMstudio runs on most recent systems with a 4-12B token model available to load (there’s lots to choose from) and can answer questions and do editing/scripting tasks without any changes.
I run it on a M1 laptop and it does what I want, fairly quickly. See if it does what you’re looking for.
The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier
Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age
What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai
Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware
For those who wish to avoid big tech, and don’t mind potentially being on a data center: openrouter lets you use w/e model you wish (even free), paying per token, from whatever provider you wish (in particular, smaller companies, folks self hosting, etc.)
Some datacenters are built in terrible places, but not all.
OmniRoute is also pretty nice for that.
You should try it out. Ollama is what you’re looking for and it’s easy to set up.
