Local AI: Run LLMs on Your Own Hardware

Running models on your own hardware buys you privacy, a fixed cost, and no rate limits. It costs you speed, some output quality, and an evening. This hub is about deciding whether that trade is one you want, and then making it well.

Two questions run through everything here: will it run on my hardware, and what should I buy if it will not. The first is arithmetic and I can answer it exactly. The second usually depends on a measurement, and I am careful about which is which.

The position this pillar takes

A speed number gets published here only if it came off hardware I own. The benchmark database now has a first row, measured under the methodology. Every figure that is not in that database is still arithmetic, computed by the same code that runs the calculators, and every page says so where you will see it.

That is a slower way to build a site about hardware. It is the only version of it I would want to read.

Where to start

If you want a number you could publish, how I benchmark a local LLM is the protocol and the first Spark row. Chat is a different page.

If you have never run a model locally, start with running your first local LLM with Ollama. About fifteen minutes, no account, and it includes the step most guides skip: checking whether the model actually landed on your GPU, because one that quietly fell back to the processor still answers you and just takes far longer about it.

If you are working out what your machine can handle, the flagship is how much VRAM you need for every model size. Weights, KV cache and overhead, worked through in full, including the term most guides leave out entirely.

If you are choosing between models, which local model should you run answers the half that is arithmetic and refuses the half that is not. There is no ranking on it, deliberately.

If a filename like Q4_K_M.gguf looks like two things stuck together, it is, and quantization explained separates them. It also makes the argument nobody makes: quantizing harder buys you context, not just fit.

If a model keeps dying partway through a long session, that is the cache growing rather than a bug, and what a context window costs in memory has the per-token arithmetic.

The three facts this pillar keeps coming back to

Memory decides what loads; bandwidth decides whether you enjoy it. A machine can be excellent at one and poor at the other, and the two questions have completely different answers. Only the first is calculable from a specification sheet.

Context is not free and it is not fixed. The weights are allocated once. The cache grows with every token you exchange, out of the same pool, which is why the same card can run a 13B model or a 3B model depending on nothing but how much conversation you want it to hold.

Buying is frequently the wrong answer. For light use, a machine idling at 40 W costs more per month in electricity than the API calls it would replace. That case is worked through honestly in cloud GPU vs owning hardware, which sits in the compute pillar because it is a buying decision rather than a how-to.

The tools

Four calculators do the arithmetic on this page interactively, and share their code with the articles so the prose and the live result cannot disagree:

• Model fit: weights, cache and overhead against your memory, and the longest context that leaves you
• What tokens per second feels like: streams text at a rate you choose, which is the only honest way to understand a speed figure before you buy for it
• Power cost: what the machine costs to run, including energy per million tokens
• Local vs cloud break-even: says “never” in as many words when the arithmetic says never

What is not here yet

Hardware reviews and any comparison that turns on a table of measured speeds across machines. The benchmark database has a first Spark row; it is not yet a buying guide.

FAQ

Can I run a useful LLM without a GPU?

Yes, within limits, and the limit is speed rather than capability. Small models quantized to 4 bits fit comfortably in ordinary system memory, and a 3B model is genuinely useful for classification, extraction and summarising. What a processor cannot match is memory bandwidth, which is what sets generation speed, so the experience is slower rather than worse. How much slower is a measurement I have not made.

What does quantization do?

It stores each of the model’s weights in fewer bits, so 16 bits down to 4 cuts the model to roughly a quarter of its size. That is usually the difference between a model fitting your hardware and not. The quality cost is real, smaller than the size saving, and not linear: 8 bits is nearly free, 4 bits is a fair trade, and below 4 it rises quickly.

How much VRAM do I actually need?

It depends on three things people usually reduce to one. The weights are the easy part and the part that runs out last. The KV cache grows with your conversation and can exceed the weights entirely at long context. Then the runtime wants a gigabyte on top, and you should leave headroom on purpose. The model fit calculator runs all three against your numbers.

Is unified memory as good as VRAM?

For the question of whether a model fits, close enough: it is one pool and the model has to fit in it, and those pools are often far larger than a consumer graphics card’s. For the question of how fast it runs it is a different machine, usually slower than dedicated memory on a discrete card. A model can fit comfortably in unified memory and still generate more slowly than a smaller model would on a graphics card.

Why does this site not recommend specific models?

Because I have not measured output quality on any of them, and because any ranking ages badly: the models worth running change every few months. What does not change is the method, which is to write down ten prompts from your actual work, start smaller than you think, and escalate only when something specific fails. That beats any article, including mine.