Almost every answer to this question stops at the weights, which is the part that is easy to calculate and the part that runs out last. Three things compete for the memory on your card, and the one people forget is the one that grows while you are using it.
Here is the whole calculation:
weights = parameters x bits per weight / 8
KV cache = 2 x layers x head dim x kv heads x bytes x context length
overhead = the runtime, the driver and the activations
total = weights + KV cache + overheadEverything below is that formula applied. There is not a single measured number on this page: it is arithmetic, and you can check every line of it. What this page cannot tell you is how fast any of it runs, which is a measurement and belongs in the benchmark database.
Step 1: the weights
A parameter is a number, and quantization decides how many bits each one gets. Multiply, divide by eight for bytes, and you have the floor.
| Parameters | 4-bit | 5-bit | 8-bit | 16-bit |
|---|---|---|---|---|
| 3B | 1.4 GiB | 1.7 GiB | 2.8 GiB | 5.6 GiB |
| 7B | 3.3 GiB | 4.1 GiB | 6.5 GiB | 13.0 GiB |
| 8B | 3.7 GiB | 4.7 GiB | 7.5 GiB | 14.9 GiB |
| 13B | 6.1 GiB | 7.6 GiB | 12.1 GiB | 24.2 GiB |
| 32B | 14.9 GiB | 18.6 GiB | 29.8 GiB | 59.6 GiB |
| 70B | 32.6 GiB | 40.7 GiB | 65.2 GiB | 130.4 GiB |
Two things worth reading off that table. Dropping from 16 bits to 4 cuts a model to a quarter, which is routinely the difference between a model fitting your hardware and not. And a 70B model at 4 bits needs more memory than a 32B model at 8, so the choice between them is real rather than obvious.
The figures are nominal. A downloaded GGUF file runs above them, because a k-quant stores per-block scales alongside the weights and leaves some tensors at a higher precision than others. Treat the table as a floor and the file size on the model card as the truth. Which format to pick, and what the suffixes on those filenames actually mean, is its own question.
Step 2: the KV cache, which is where people get caught
The model keeps the keys and values it has already computed for every token in the conversation, so it does not recompute them for the next one. That cache costs a fixed number of bytes per token and grows linearly with context.
For a model with 32 layers, a width of 4096, 32 attention heads and 8 key-value heads, holding the cache at 16 bits:
| Context | KV cache |
|---|---|
| 4k | 0.5 GiB |
| 8k | 1.0 GiB |
| 16k | 2.0 GiB |
| 32k | 4.0 GiB |
| 128k | 16.0 GiB |
At 128k of context the cache costs more than the weights of the model holding it. This is why a model that loaded fine can run out of memory an hour into a long conversation: nothing changed except how much you had said to it. That failure mode, and what a context window actually costs as you use it, is worth its own read if you work with long documents.
Get the four numbers from your model’s config.json, which every repository
on Hugging Face publishes: num_hidden_layers, hidden_size,
num_attention_heads and num_key_value_heads.
Step 3: why that last number matters more than it looks
If num_key_value_heads equals num_attention_heads, the model has no
grouped-query attention and every head keeps its own cache. Same model shape
as above, one key-value head per attention head:
| Context | Without GQA | With 4:1 GQA |
|---|---|---|
| 4k | 2.0 GiB | 0.5 GiB |
| 8k | 4.0 GiB | 1.0 GiB |
| 16k | 8.0 GiB | 2.0 GiB |
| 32k | 16.0 GiB | 4.0 GiB |
| 128k | 64.0 GiB | 16.0 GiB |
Four times the memory, for the same model at the same context. This is the single biggest reason a recent model holds far more context than an older one of the same size, and it is invisible if you only look at parameter counts.
Step 4: overhead, and the headroom you should not spend
The runtime, the driver context and the activation buffers want something on top. A gigabyte is a reasonable starting allowance and it varies with the runtime, the batch size and the driver.
Then leave headroom on purpose. A model that exactly fills your card will spill the moment the context does, and a card with nothing spare is slower than one with room. If you are sizing a purchase rather than checking a download, aim to use about eighty percent of what you have.
Sizing a purchase raises a separate question this page does not answer: whether to buy at all. Memory decides what a machine can run, not whether owning one beats renting the same tokens, and that arithmetic frequently says never.
What fits on what
Putting the three together, at 4 bits with 8k of context and a gigabyte of overhead:
| Card | Comfortable | Tight | No |
|---|---|---|---|
| 8 GiB | 3B, 7B | 8B | 13B and up |
| 12 GiB | 7B, 8B | 13B | 32B and up |
| 16 GiB | 8B, 13B | 13B at long context | 32B and up |
| 24 GiB | 13B | 32B | 70B |
| 48 GiB | 32B | 70B at short context | 70B at long context |
“Tight” means it loads and leaves you little room to grow the context, which in practice means it will fail partway through a long session rather than refusing to start.
Rather than reading a table, put your own numbers in: the model fit calculator runs this exact arithmetic, including the KV cache and the longest context your memory allows, and shows its working.
What this arithmetic cannot tell you
Whether it will be fast enough. Fitting and running well are different questions. Memory bandwidth sets speed, capacity sets whether it loads at all, and a machine can be excellent at one and poor at the other. A number for speed has to be measured, which is what the bench is for, and what a given tokens per second feels like is worth knowing before you buy for it.
Whether the smaller model is good enough. Every gigabyte saved by quantising harder or dropping a size class costs some output quality. How much is not settled by any memory calculation, and it depends entirely on what you are asking the model to do.
What offloading will cost you. Runtimes that split a model between GPU and system memory will load something this page calls too big. It will also be slower, and how much slower is a measurement rather than a formula.
FAQ
How much VRAM do I need to run a 7B model?
About 3.3 GiB for the weights at 4 bits, plus context and overhead, so an 8 GiB card handles it comfortably with room for a long conversation. At 8 bits the weights are 6.5 GiB, which fits an 8 GiB card only barely and leaves almost nothing for context. The practical answer for most people is that 8 GiB runs 7B models well and 12 GiB runs them without thinking about it.
Can I run a 70B model at home?
At 4 bits the weights alone are 32.6 GiB, so you need 48 GiB of memory to do it comfortably, or a machine with a large pool of unified memory. It is achievable and it is not cheap. Before committing, work out whether a 32B model at 8 bits, which needs 29.8 GiB, answers your actual questions well enough, because that is a much easier machine to buy.
Does the context length really change how much memory I need?
Yes, linearly, and it is the most common cause of an out-of-memory error that appears mid-conversation rather than at load. For the model shape used above, every 8k of context costs a gibibyte. Going from 8k to 128k costs sixteen, which is more than the entire model at 4 bits.
Is unified memory the same as VRAM for this?
For the fitting question, close enough: it is one pool and the model has to fit in it. For the speed question it is a different machine, because unified memory is usually slower than the dedicated memory on a discrete card. A model can fit comfortably in unified memory and still generate more slowly than a smaller model would on a graphics card.
Why is my downloaded file bigger than this page says?
Because bits per weight is nominal and real quantised files are mixtures. A GGUF k-quant stores per-block scales next to the weights and quantises different tensors to different levels, so the file lands above the nominal figure. Embedding and output layers are often kept at higher precision than the rest. The file size on the model card is the number to trust.
Sources
- The KV cache formula follows from the transformer decoder in Vaswani et al. (2017), Attention Is All You Need.
- Grouped-query attention, and why the cache scales with key-value heads: Ainslie et al. (2023), GQA.
- Field names are the Hugging Face transformers configuration keys, published
per model in each repository’s
config.json.
Every figure on this page is arithmetic from those two formulas, computed by the same code that runs the model fit calculator. None of it was measured on any hardware. When runs are published to the benchmark database, this article gains measured memory use alongside the predicted figures and its updated date moves.