Local AI on a Budget

One of my many side projects is using Local AI but like most, I’m on a budget. I’ve tried really small models, some as small as 1.5 billion parameters and in a nutshell, they act like a 1.5 billion parameter model. They work Ok, are somewhat performant but you lose a lot of ability, tooling may or may not work, forget image OCR, and kiss coding goodbye. MOE models are better, you can use a much larger model span it between GPU and CPU but all in all still weak sauce. Through my journey with local AI, I had an epiphany and started thinking through what I was trying to accomplish and since AI changes weekly there had to be a better way.

Let me start first with my hardware, I bought it on Amazon before RAM spiked for another project. It’s a BOSGAME P4 Mini PC with integrated Radeon graphics, 32GB of DDR4, AMD processor and a 1TB M2 disk. While its powerful for its size the GPU is its limiting factor.

It does allow you to give it 16GB UMA memory to the GPU but that’s it. I went through several Ubuntu builds that worked but not great. I tuned Ollama, tuned the hardware and still no luck. It would work but it was very slow and if I’d let it run it would eventually produce something but not quick at all. Then I started thinking about what MOE Models really do, they allow you to run a model and it pulls the pieces it needs when it needs it. This is great for loading one 14 billion parameter model and letting it do its thing, you have a little room for context but again slow. This led me to thinking about this differently, what if instead of running one MOE, what if I decided who the experts were and hired their little brother.

I had to think through this and a feature of Anything LLM is that you can do model routing, this allows you to use a specialized model for OCR, a different model for coding, research and planning or even general purpose. I had originally started with a restricted Ollama, allowing only 1 model at a time but found that if I wanted to get the performance I wanted I needed to run more than that. I updated my Ollama service to look like this:

This gave me the ability to run more models. Up to the 5, I also optimized my Ollama configuration to take into account my less than ideal video card. I then used Model Routing in Anything LLM, there are guides online or use Gemini on how to set that up. It takes a bit to get it to route properly but once its setup it works better than a single model for my use case. I run Anything LLM in a container using compose. The models I chose are as follows:

You will then want to set up the routing in Anything LLM and then test. Qwen2.5vl:3b is critical if you want OCR abilities to drop a file into the chat and have it parse it as well.

The smaller models are inadequate by themselves. MOE still needs decent hardware but if you focus on what you are trying to accomplish you can make it usable. I average 20 to 25 TPS now, and before I was barely able to hit 10.

I have a script that I can run against ollama to spit out the information I needed to determine the models that gave me the best bang for the buck and here are some of the results. I have stuck with 3 billion parameter models or less as that seems to be the sweet spot, but they are optimized for certain parts then I route to them.

On smaller 1.5 billion parameter models you can get these types of speeds.

Not too shabby for a MiniPC that has an old integrated GPU.