This question gets answered badly almost everywhere, because it is two questions wearing one coat.
What can my machine run? Arithmetic. Exact, checkable, and worked out below in full.
Which of those should I choose? A judgment about output quality on your particular work, which no table settles and which I have not measured. I am going to give you the first honestly and then tell you how to settle the second yourself, rather than hand you a ranking I invented.
The ceiling, and how context moves it
Here is the largest model at 4 bits that fits, with a gibibyte of overhead, at three different context lengths:
| Memory | At 4k context | At 8k | At 32k |
|---|---|---|---|
| 6 GiB | 8B | 8B | 1B |
| 8 GiB | 13B | 8B | 3B |
| 12 GiB | 13B | 13B | 13B |
| 16 GiB | 13B | 13B | 13B |
| 24 GiB | 32B | 32B | 32B |
| 48 GiB | 70B | 70B | 70B |
Read the 8 GiB row twice. The same card runs a 13B model or a 3B model depending on nothing but how much conversation you want it to hold. Two size classes, decided by a setting, and no “what fits in 8 GB” list I have seen mentions it.
That is the practical lesson of the whole page. Decide your context first, because it moves the answer more than any other choice you make, and what a context window costs is worth reading before you settle on one.
Above 12 GiB the table goes flat, and that is not an error. Once the weights fit comfortably, the cache has room to grow and context stops being the binding constraint. The steps then happen at the size classes themselves.
If you are sizing a purchase rather than checking a card
Do not buy exactly enough. A model that precisely fills your memory will spill the moment the context does. Aim to sit at about eighty percent:
| Target model, 4 bits, 8k context | Needs | Buy at least |
|---|---|---|
| 3B | 3.40 GiB | 4.25 GiB |
| 7B | 5.26 GiB | 6.57 GiB |
| 8B | 5.73 GiB | 7.16 GiB |
| 13B | 8.05 GiB | 10.07 GiB |
| 32B | 16.90 GiB | 21.13 GiB |
| 70B | 34.60 GiB | 43.25 GiB |
Which maps onto real cards in an unforgiving way. A 70B model at 4 bits wants 48 GiB to be comfortable, and there is no consumer graphics card at that capacity, which is why people reach for unified memory or two cards for that tier.
Under the ceiling, the real trade
Fitting the largest thing you can is not automatically right. On 12 GiB at 8k of context, several very different choices all fit:
| Choice | Memory used | Headroom | Longest context |
|---|---|---|---|
| 8B at 4 bits | 5.73 GiB | 6.27 GiB | 59,594 |
| 7B at 8 bits | 8.52 GiB | 3.48 GiB | 36,706 |
| 13B at 4 bits | 8.05 GiB | 3.95 GiB | 40,520 |
| 13B at 5 bits | 9.57 GiB | 2.43 GiB | 28,123 |
Four real options on one card, and they differ by more than a benchmark score. The 8B at 4 bits holds twice the conversation of the 13B at 5 bits. The 13B at 4 bits is a bigger model than the 7B at 8 bits and holds more context, which is the direction people find surprising and is worked through in quantization explained.
Run your own numbers through the model fit calculator rather than reading a table built on a model shape that may not be yours.
The part I will not fake
Which of those four gives better answers on your work is not a memory question, and it is the question you actually asked.
I have not measured output quality on any model, so I am not going to rank them. What I can tell you is the shape of the trade-offs, which is stable enough to be useful:
- Bigger generally beats smaller at the same quantization, and the gap narrows as both get large enough for your task.
- Bigger tolerates quantization better, so a larger model squeezed harder frequently beats a smaller one at higher precision, within reason.
- Instruction-tuned beats base for anything conversational. A base model is a completion engine and will not behave the way you expect.
- Specialised beats general on its speciality and often loses elsewhere. A coding model is a real advantage for code and a liability for prose.
- Recency matters more than parameter count across generations. A recent 8B model can beat an older 13B outright, which is why a two-year-old comparison is worse than no comparison.
Notice that none of those is a number, and that is deliberate. Published benchmark scores measure benchmarks. Whether a model is good at your prompts, in your language, on your kind of problem, is a question about your work.
How to settle it in an evening
The method is duller than a ranking and it actually works.
- Write down ten prompts you genuinely use. Real ones, from real work, including the two that usually go wrong.
- Start smaller than you think. A 3B or an 8B at 4 bits. If it handles your ten, you are finished, and you have kept the memory for context.
- Escalate only on demonstrated failure. Move up a size class or a quantization level when something specific fails, not because a larger number feels safer.
- Compare the outputs side by side, not from memory. Same prompts, same settings, read them together. Impressions of model quality drift within minutes.
- Check the speed is liveable at the size you landed on. Fitting and being pleasant to use are different, and what a given tokens per second feels like is worth knowing before you commit.
Two downloads and an evening beats any article, including this one. The models change every few months and your ten prompts do not.
The other thing that changes the answer
Speed. A model that fits and generates slowly is a different product from one that fits and generates quickly, and the difference is set by memory bandwidth rather than by anything on this page.
I have one published speed row, on the DGX Spark, in the benchmark database. That is not a speed column for every model in this table, so this article still stops at what fits. When more runs land, the choice becomes a genuine three-way trade between size, context and how long you wait.
FAQ
What model can I run with 8 GB of VRAM?
Up to about 8B at 4 bits with 8k of context, or up to 13B if you are happy with 4k, or only around 3B if you want 32k. The context setting moves the answer by two size classes on that card, which is why a single figure for “what fits in 8 GB” is misleading without one attached.
Is a bigger model always better?
At the same quantization, generally yes, with diminishing returns once both are large enough for what you are doing. But bigger costs memory that context also wants, and the largest model that fits is frequently not the best choice: on 12 GiB, an 8B at 4 bits holds twice the conversation of a 13B at 5 bits. Decide what you need to hold before deciding how big to go.
Should I run a 13B at 4 bits or a 7B at 8 bits?
They are close in size and the larger model usually wins, since it has more capacity to begin with and tolerates quantization better. It also leaves room for more context on the same card. It is not a universal rule, and the way to settle it for your work is to run both on the same ten prompts and read the outputs together.
What is the best local model right now?
I am not going to answer that, because any answer ages badly and I have not measured output quality on any of them. The models worth running change every few months, and an article confidently ranking them is either very fresh or actively misleading you. What does not change is the method: your prompts, smallest first, escalate on failure.
Do I need an instruction-tuned model?
For anything conversational, yes. A base model completes text rather than following instructions, so it will continue your question instead of answering it, which reads as the model being broken when it is doing exactly what it was trained to do. Look for “instruct” or “chat” in the model name.
How do I know if a smaller model is good enough?
By testing it on the work, which is the only way. Write down ten real prompts, run the small one, and read the answers. People routinely discover a 3B model handles their classification and extraction perfectly well, and equally routinely discover it does not handle their reasoning task at all. Both are useful findings and neither is in a benchmark table.
Sources
- Every memory, headroom and context figure on this page is computed by the
same code that runs the model fit calculator, using the
reference model shape used site-wide: 32 layers, a width of 4096, 32
attention heads, 8 key-value heads, cache at 16 bits. Your model’s four
numbers are in its
config.jsonand are worth substituting. - The qualitative trade-offs are the consensus shape rather than results from a test, and are stated without numbers for that reason. Nothing on this page was measured on any hardware.
- No model is ranked or recommended here. When runs are published to the benchmark database, this article gains measured speed alongside the fit arithmetic and its updated date moves. Output quality on your own work remains something only you can measure.