A model is a large pile of numbers. Quantization is the decision to store each of those numbers in fewer bits than it was trained in, and everything else follows from that one choice.
The size arithmetic is the easy half:
weights = parameters x bits per weight / 8At 16 bits a 13B model is 24.21 GiB of weights. At 4 bits it is 6.05 GiB. That is routinely the difference between a model loading on your card and not.
| Bits per weight | 7B | 8B | 13B | 32B | 70B |
|---|---|---|---|---|---|
| 16 | 13.04 | 14.90 | 24.21 | 59.60 | 130.39 |
| 8 | 6.52 | 7.45 | 12.11 | 29.80 | 65.19 |
| 6 | 4.89 | 5.59 | 9.08 | 22.35 | 48.89 |
| 5 | 4.07 | 4.66 | 7.57 | 18.63 | 40.75 |
| 4 | 3.26 | 3.73 | 6.05 | 14.90 | 32.60 |
| 3 | 2.44 | 2.79 | 4.54 | 11.18 | 24.45 |
| 2 | 1.63 | 1.86 | 3.03 | 7.45 | 16.30 |
All figures in GiB, all of it arithmetic. There is not a measured number on this page.
The names are two things stacked
Most of the confusion here is that a filename like llama3.1-8b-Q4_K_M.gguf
is describing two separate decisions, and people argue about them as if they
were one.
GGUF is a container. It is the single-file format used by llama.cpp and everything built on it, holding the weights, the tokenizer and the metadata together, which is why a local runtime can load a download with no further setup. GGUF is not a quantization method. It is the box.
Q4_K_M is the scheme inside the box. A k-quant, at a nominal four bits,
in the medium variant. That is what actually decides how the numbers are
stored.
The main schemes you will meet, and what distinguishes them:
- k-quants (
Q2_KthroughQ8_0, inside GGUF). Weights are stored in blocks with a scale per block, and different tensors get different precision. Runs on CPU and GPU, which is why it dominates local use. - GPTQ. Post-training quantization that adjusts the remaining weights to compensate for the error introduced as each one is rounded, using a small calibration set. GPU-oriented.
- AWQ. Starts from the observation that a small fraction of weights matter disproportionately to output quality, identifies them from activation statistics, and protects them while quantizing the rest harder.
- bitsandbytes and NF4. Quantizes on load rather than shipping a pre-quantized file, which is convenient for experimentation and is the route most fine-tuning at reduced precision takes.
The useful distinction is not which is best in the abstract. It is that k-quants inside GGUF are what runs anywhere, and the others assume a GPU and a Python stack.
A four-bit file is never four bits
This is the single most common surprise, and it is not a bug.
A k-quant does not store four bits per weight and stop. It stores weights in blocks, and each block needs a scale so those four bits can be decoded back into something useful. That scale is stored alongside. On top of that, a k-quant deliberately keeps some tensors at higher precision than others, because the embedding and output layers cost a lot of quality when squeezed and relatively little size when spared.
So the real average lands above the nominal figure the name advertises, and by
different amounts for different variants of the same nominal width. The _S,
_M and _L suffixes are exactly that: how much extra precision the mixture
spends.
Treat the table above as a floor and the file size on the model card as the truth. The model fit calculator takes bits per weight as an input for this reason, so you can put in the effective figure rather than the label.
The thing quantization buys that nobody mentions
Everyone frames this as “will the model fit”. That is the smaller half of it.
Weights are fixed once the model loads. The KV cache is not: it grows with every token in the conversation, and it competes for the same memory. So every gibibyte you save on weights becomes context you can actually hold.
An 8B model on an 8 GiB card, with a gibibyte of overhead. The model shape is 32 layers, a width of 4096, 32 attention heads and 8 key-value heads, holding the cache at 16 bits, which works out at 128 KiB of cache per token:
| Bits per weight | Weights | Longest context that fits |
|---|---|---|
| 8 | 7.45 GiB | does not load |
| 6 | 5.59 GiB | 11,567 tokens |
| 5 | 4.66 GiB | 19,197 tokens |
| 4 | 3.73 GiB | 26,826 tokens |
| 3 | 2.79 GiB | 34,455 tokens |
| 2 | 1.86 GiB | 42,085 tokens |
Every step down the table frees weights and buys roughly 7,600 more tokens of conversation, because that is what the freed gibibytes convert into. Whether the trade is worth it depends entirely on what you are doing, and it is a real trade rather than a free win, but it is invisible if you only ask whether the model loads.
The same effect makes a counterintuitive comparison come out the other way. Holding that cache geometry fixed so only the weights change, neither a 13B at 4 bits nor a 7B at 8 bits fits an 8 GiB card at 8k of context. Push both to their limit and the 13B holds 7,752 tokens where the 7B at 8 bits holds 3,938. The larger model quantized harder wins on both counts here, which is not what “quantize less, quality is better” would predict.
I have worked the memory arithmetic through in full, including the KV cache and grouped-query attention, in how much VRAM you need for every model size.
What it costs
Everything above is the benefit. The cost is output quality, and it is real.
Three things about that cost are worth holding onto:
It is not linear. The drop from 16 bits to 8 is very small. From 8 to 4 is noticeable but usually acceptable. Below 4 it starts to bite, and at 2 bits a model can be coherent and still be meaningfully worse at the thing you wanted it for. This is why 4 bits became the default and not 2.
Bigger models tolerate it better. A 70B model at 4 bits typically holds up better than a 7B at 4 bits, because there is more redundancy in the representation to lose. This is the arithmetic behind the common advice to prefer a larger model quantized harder over a smaller one at higher precision, within reason.
How much it costs your task is not something I can tell you. The published measurements are on benchmarks. Whether a given quantization degrades your particular prompts, in your language, on your kind of problem, is a question about your work, and the only honest way to answer it is to run both on the same inputs and read the outputs. It is a couple of downloads and an evening.
I have not measured quantization quality on any hardware and I am not going to quote a number I did not produce. The sources at the bottom are where the published figures live.
Picking one
- Running locally, want it to just work: GGUF with a k-quant.
Q4_K_Mis the common default for a reason, andQ5_K_Mif you have the room. - Nvidia GPU, Python stack, serving throughput: AWQ or GPTQ, and check what your serving runtime supports before downloading.
- Fine-tuning at home: bitsandbytes and NF4, which is the route QLoRA takes.
- Enough memory for 8 bits: use it, and stop reading. The quality cost at 8 bits is small enough that the decision is not interesting.
- Considering 2 or 3 bits: check whether a smaller model at 4 bits does your job first. It often does, and it is a less strange machine to reason about.
Then check the result against your actual memory with the model fit calculator, which will also tell you the longest context the choice leaves you. If you have not settled on a model yet, the size and quantization decisions are the same decision, and choosing between them is worth doing in one pass rather than two.
FAQ
What does Q4_K_M mean?
Four nominal bits per weight, a k-quant, medium mixture. The K marks it as a
k-quant, which stores weights in blocks with a scale per block rather than one
scale for a whole tensor. The S, M and L suffixes say how much of the
model is kept at higher precision than the nominal figure: more precision, a
larger file, slightly better output. Q4_K_M is the usual default because it
sits at a reasonable point on that curve.
Is GGUF better than AWQ?
They are not the same kind of thing, which is why the question does not resolve. GGUF is a file format and AWQ is a quantization method. The real choice is between a k-quant inside GGUF, which runs on CPU and GPU through llama.cpp and everything built on it, and a GPU-oriented scheme like AWQ or GPTQ that expects a Python serving stack. Pick by what you are running it on.
How much quality do I lose at 4 bits?
Less than the size saving would suggest, and more than nothing. The published comparisons live in the papers linked below, they are measured on benchmarks rather than on your prompts, and the honest answer for your own use is to run two quantizations of the same model on the same inputs and read the results. The general shape is that 8 bits is nearly free, 4 bits is a fair trade, and below 4 the cost rises quickly.
Is a 13B at 4 bits better than a 7B at 8 bits?
Frequently, and the memory arithmetic favours it too. The two are close in file size, 6.05 GiB against 6.52, but the larger model has more capacity to begin with and tolerates quantization better. On an 8 GiB card the 13B at 4 bits also leaves room for roughly twice the context. It is not a universal rule, and the way to settle it for your work is to run both.
Why is my downloaded file bigger than the calculator said?
Because the nominal bits per weight is a label, not a measurement of the file. A k-quant stores per-block scales next to the weights, and keeps some tensors at higher precision than the name implies. The file size on the model card is the number to trust, and it is the number to put into the calculator.
Does quantization make the model faster?
Usually yes, though for a reason people often get backwards. Generating a token requires reading the weights, so a smaller model moves less data through memory and memory bandwidth is what limits generation on most machines. That is a speed argument rather than a compute argument, and how much faster is a measurement rather than a formula, so it belongs in the benchmark database rather than on this page.
Sources
- GPTQ, the error-compensating post-training approach: Frantar et al. (2022), GPTQ.
- AWQ, and the finding that a small fraction of weights carry a disproportionate share of the quality: Lin et al. (2023), AWQ.
- 8-bit inference and the outlier features that make naive 8-bit fail: Dettmers et al. (2022), LLM.int8().
- NF4 and quantized fine-tuning: Dettmers et al. (2023), QLoRA.
- The GGUF format and the k-quant mixtures are defined by llama.cpp, which is the reference for what each suffix actually keeps at higher precision.
- Every size and context figure on this page is computed by the same code that runs the model fit calculator. None of it was measured on any hardware. When runs are published to the benchmark database, the speed question above gains measured answers and this article’s updated date moves.