I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier
Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age
What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai
Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware