Weights are a fixed cost. You load them once and they sit there.
Context is not. The model caches the keys and values it computed for every token in the conversation so it does not recompute them for the next one, and that cache grows linearly for as long as you keep talking. It comes out of the same memory the weights are in.
Which is why the failure mode is so distinctive. A model loads, works fine, and then dies partway through a session, with nothing having changed except how much you had said to it.
The per-token figure
For a model with 32 layers, a width of 4096, 32 attention heads and 8 key-value heads, holding the cache at 16 bits:
KV cache = 2 x layers x head dim x kv heads x bytes x context lengthThat works out at 128 KiB per token. Every token, both the ones you typed and the ones it generated.
| Context | Cache |
|---|---|
| 2,048 | 0.25 GiB |
| 4,096 | 0.50 GiB |
| 8,192 | 1.00 GiB |
| 16,384 | 2.00 GiB |
| 32,768 | 4.00 GiB |
| 65,536 | 8.00 GiB |
| 131,072 | 16.00 GiB |
Put that next to the model. An 8B model at 4 bits is 3.73 GiB of weights. So a 32k window costs slightly more than the model itself, and a full 128k window costs 16 GiB, more than four times the weights. The thing people size their card around is the smaller number.
Add it up for the full window: 3.73 GiB of weights, 16 GiB of cache, a gibibyte of overhead, and you need 20.73 GiB to hold an 8B model with its context full. That is a 24 GiB card doing the work of what everyone describes as an 8 GiB model.
How much conversation that actually is
Tokens are a bad unit for planning, because nobody thinks in them. English runs at roughly 0.75 words per token, so here is the same arithmetic in words, for an 8B model at 4 bits with a gibibyte of overhead:
| Memory | Tokens | Words | Roughly |
|---|---|---|---|
| 8 GiB | 26,826 | 20,120 | 85 minutes of reading |
| 12 GiB | 59,594 | 44,696 | 3 hours |
| 16 GiB | 92,362 | 69,272 | 5 hours |
| 24 GiB | 157,898 | 118,424 | 8 hours |
Reading times at 238 words per minute, which is the figure the tokens per second simulator uses for the same reason: it turns an abstract number into something you can feel.
That 8 GiB row is the useful one. Twenty thousand words is a long conversation but not an absurd one, and it is roughly a novella. If your work involves pasting in documents rather than chatting, you will reach it in one message.
The two levers that change it
Grouped-query attention, which you do not control but should check. The cache scales with key-value heads, not attention heads. The shape above has four attention heads per key-value head. A model without that sharing caches four times as much:
| Context | With 4:1 GQA | Without GQA |
|---|---|---|
| 8,192 | 1.00 GiB | 4.00 GiB |
| 32,768 | 4.00 GiB | 16.00 GiB |
| 131,072 | 16.00 GiB | 64.00 GiB |
Sixty-four gibibytes of cache for one model’s context. This is the single
biggest reason a recent model holds far more context than an older one of the
same parameter count, and you cannot see it in the parameter count at all. It
is in num_key_value_heads in the model’s config.json.
Cache quantization, which you do control. Holding the cache at 8 bits instead of 16 halves it, and roughly doubles how far you get:
| Memory | 16-bit cache | 8-bit cache |
|---|---|---|
| 8 GiB | 26,826 tokens | 53,652 tokens |
| 12 GiB | 59,594 | 119,188 |
| 16 GiB | 92,362 | 184,724 |
| 24 GiB | 157,898 | 315,796 |
Most runtimes expose this, and it is usually the cheapest context you will ever buy. It is not free: the cache is what the model attends to, so quantizing it has a quality cost of its own, and how much depends on the model and the length. Worth testing on your own work rather than assuming.
There is a third lever, which is to quantize the weights harder and spend the freed memory on cache. That trade is worked through in quantization explained, and the direction surprises people: a larger model quantized harder can hold more conversation than a smaller one at higher precision on the same card.
What the arithmetic does not cover
Advertised context is not free context. A model card saying 128k is telling you what the model was trained to handle, not what your hardware will hold. Those are independent numbers and the smaller one wins.
Prefill is a separate cost, in time rather than memory. Everything above is about holding the cache. Filling it, when you paste in a long document, is compute the model has to do before it says anything, and it is why a long prompt has a long pause in front of it. How long is a measurement rather than a formula, so it belongs in the benchmark database rather than here.
Runtimes differ in the details. Paged and block-based cache implementations allocate differently, some reserve the full window up front while others grow as needed, and batching several conversations multiplies the cache by the number of them. The formula gives you the size of the thing being allocated, not every runtime’s allocation policy.
Quality over long context is its own subject. Fitting 128k of context and using 128k of context well are different claims, and the second one is not a memory question.
What to do with this
Decide the context you actually need before choosing a card, rather than after. Most conversational use never approaches the advertised window, and most document work exceeds it immediately, so the honest answer depends entirely on which one you do.
Then put your own model shape into the model fit calculator. It runs this exact arithmetic, reports the longest context your memory allows, and tells you whether weights or context is the thing limiting you, which decides whether the fix is a smaller quantization or a shorter window. The full memory picture, weights and overhead included, is in how much VRAM you need for every model size.
FAQ
Why does my model run out of memory mid-conversation?
Because the cache grew. Weights are allocated once at load, but every token you exchange adds to the KV cache, and at 128 KiB per token for a typical 8B shape, a long session adds gibibytes. Nothing changed except the length of the conversation, which is exactly why it looks like a random failure rather than a sizing problem.
How much VRAM does a 32k context window need?
For the model shape above, 4 GiB on top of the weights and overhead. For a
model without grouped-query attention it is 16 GiB. The per-token figure comes
from four numbers in the model’s config.json, so it varies by model and it
is worth computing rather than assuming.
Does a longer context slow the model down?
Yes, in two ways worth separating. Filling the cache in the first place is compute the model does before it responds, which is the pause in front of a long pasted document. After that, attending over a longer cache is more work per token generated. How much of each is a measurement rather than a formula, which is what the bench is for.
Should I use an 8-bit KV cache?
If context is what is limiting you, it is usually the first thing to try, since it halves the cache and roughly doubles how far you get. It has a quality cost, because the cache is what the model attends to, and the size of that cost depends on the model and the length. Run your actual work at both settings before deciding.
Is unified memory different for this?
Not for the arithmetic: it is one pool and the cache has to fit in it, same as anything else. The difference is that unified memory pools are often much larger than a discrete card’s, which makes long context achievable on machines that would not fit the same model on a graphics card. Speed is a separate question and usually goes the other way.
Can I just set a smaller context window?
Yes, and it is the direct fix when the cache is what is over your budget. Runtimes let you cap it, and the cap sets the allocation. The cost is that the model forgets the earlier part of a long conversation, which is a trade rather than a solution, and the model fit calculator will tell you the largest cap your memory supports.
Sources
- The KV cache follows from the transformer decoder in Vaswani et al. (2017), Attention Is All You Need.
- Grouped-query attention, and why the cache scales with key-value heads rather than attention heads: Ainslie et al. (2023), GQA.
- The four shape numbers are the Hugging Face transformers configuration keys,
published per model in each repository’s
config.json:num_hidden_layers,hidden_size,num_attention_headsandnum_key_value_heads. - Roughly 0.75 words per token is a rule of thumb for English and varies with the tokenizer and the text. Reading times use 238 words per minute, the silent reading rate from Brysbaert (2019).
- Every figure on this page is computed by the same code that runs the model fit calculator. None of it was measured on any hardware. When runs are published to the benchmark database, the two speed questions above gain measured answers and this article’s updated date moves.