What Will Actually Fit on Your Hardware
Researched
Seeded with published specifications and cited sources rather than results measured here. Every seeded figure links to where it came from.
Fits, with 18.27 GiB to spare.
This calculator needs JavaScript. The formulas are written out below and the arithmetic is short enough to do by hand.
Runs in your browser. Nothing you type here is uploaded, logged or sent anywhere. The page is a static file and the arithmetic happens on your machine. Same principle as the rest of the site.
How this is calculated
Three terms, added together, then compared against what you have.
weights = parameters × bits per weight ÷ 8
head dim = hidden_size ÷ num_attention_heads
KV per token = 2 × layers × head dim × num_key_value_heads × bytes per element
KV cache = KV per token × context length
total = weights + KV cache + runtime overhead
The leading 2 in the KV line is the key and the value. The reason
num_key_value_heads appears rather than
num_attention_heads is grouped-query attention: several
attention heads share one set of keys and values, so a model with
32 attention heads and 8 KV heads pays
a quarter of what its width suggests.
| Constant | Value | Why |
|---|---|---|
| Bits in a byte | 8 | Turns a bits-per-weight figure into bytes |
| Bytes in a GiB | 1,073,741,824 | What a GPU reports. A card sold as 24 GB holds 24 GiB |
| Key and value tensors | 2 | Both are cached per token, per layer |
| Runtime overhead | Your input, 1 GiB by default | Driver context, activations and framework. A rule of thumb, not a measurement |
Worked example, the one above: 8e9 parameters at 4 bits is 4e9 bytes, or 3.73 GiB. Head dim is 4096 over 32, which is 128. KV per token is 2 × 32 × 128 × 8 × 2, which is 131,072 bytes, or 128 KiB. Times 8,192 tokens is exactly 1.00 GiB.
What this can't know
- Bits per weight is nominal, and real files are mixtures. A GGUF k-quant stores per-block scales next to the weights and leaves some tensors at a higher precision than others, so a downloaded file runs above its nominal figure. The ground truth for the weights is the size of the file on disk. Use that when you have it.
- It does not know your model. Layers, width and head
counts have to come from the model's own
config.json. The values loaded above are an example to replace, not a claim about any particular model. - Overhead varies more than the default suggests. A CUDA context, the framework and the activation buffers depend on the runtime, the batch size and the driver. Treat the default as a placeholder and raise it if you are running close to the edge.
- Fitting is not running well. A model that fits with nothing to spare will be slower than one with room, and on a shared GPU anything else on the card takes its share first. Nothing here predicts speed. For that you need a measurement, which is what the bench is for.
- Offloading is not modelled. Runtimes that split a model between GPU and system RAM will load something this page calls too big. It will also be much slower, and how much slower is a measurement, not a formula.
Sources
- The KV cache formula follows directly from the transformer decoder described in Vaswani et al. (2017), Attention Is All You Need. arxiv.org/abs/1706.03762
- Grouped-query attention, and why the cache scales with key/value heads rather than attention heads: Ainslie et al. (2023), GQA. arxiv.org/abs/2305.13245
-
Field names used above are the Hugging Face transformers configuration
keys, published per model in each repository's
config.json. huggingface.co/docs/transformers - No figure on this page came off the bench. It computes from what you type in. When runs are published, this tool gains a machine picker and is re-badged Measured; the method those runs will be held to is on the methodology page.
FAQ
How much VRAM do I need to run an LLM?
Enough for three things, and people usually forget the third. The weights are parameters times bits per weight divided by eight, so an 8 billion parameter model at 4 bits is about 3.7 GiB. The KV cache grows with context and is the term that catches people out. The runtime, the driver and the activations want something on top, typically a spare gigabyte or so. Add all three, then leave headroom, because a model that exactly fills your card will spill the moment the context does.
Why does the file I downloaded not match this estimate?
Because bits per weight is a nominal figure and a real quantised file is a mixture. GGUF k-quants store per-block scales and minimums alongside the weights, and they quantise different tensors to different levels, so the file lands above the nominal number. Embedding and output layers are often left at a higher precision than the rest. If you know the actual file size, type its size in gibibytes into the bits-per-weight box as a check: the ground truth for the weights is the file on disk, not this estimate.
What is the KV cache and why does it grow?
It is the keys and values the model has already computed for every token in the conversation, kept so it does not have to recompute them for the next one. It costs a fixed number of bytes per token, 128 KiB per token for the configuration loaded above, and it grows linearly with context length. Doubling your context doubles it. This is why a model that loads happily can still run out of memory an hour into a long conversation.
What is grouped-query attention and why does it matter here?
It lets several attention heads share one set of keys and values instead of each keeping its own. The KV cache then scales with the number of key/value heads rather than the number of attention heads, which on a model with four attention heads per KV head cuts the cache to a quarter. Look for num_key_value_heads in config.json. When it equals num_attention_heads the model has no grouped-query attention and its context is four to eight times more expensive than a modern model of the same width.
Should I use a bigger model at lower precision, or a smaller one at higher precision?
The memory arithmetic on this page answers only half of that. It tells you what fits; it says nothing about which one answers your question better, and that trade is not settled by any number here. What this tool can tell you is that the choice is usually real: a larger model at four bits and a smaller one at eight often land within a gigabyte of each other, so it is worth checking both rather than assuming the bigger one is out of reach.
Where to go next
- A size that fits still has to be fast enough to use. What tokens per second actually feels like makes that concrete.
- Running a model for the first time: Run your first local LLM with Ollama.
- What a machine costs to keep running once the model fits: the power cost calculator.