Quantization Explained: INT8, INT4, GPTQ and AWQ for LLMs

A seventy billion parameter model stored in sixteen-bit numbers needs about one hundred forty gigabytes just for its weights. That does not fit on a single eighty gigabyte GPU. But store each weight in four bits instead, and the same model needs roughly thirty-five gigabytes. This lesson is about how that trick works, and what it costs you.

• 70B weights = 140 GB in FP16
• Fewer bits per weight

In the previous episode we did latency and cost math for LLM applications, and we saw that decoding speed is dominated by how many bytes we must read from memory per token. Quantization attacks exactly that number. Today we derive the math, quantize weights by hand, understand GPTQ and AWQ, and then write real Python.

• Last time: latency and cost math
• Today: shrink the weights

Here is the plan. First the intuition and the formula for uniform quantization. Then a worked example with real numbers. Then why outliers are the villain, followed by GPTQ and AWQ. Finally the memory math, the speed story, and Python code you can run yourself.

• Math, then methods
• Memory math
• Python code

A weight is a real number, and in sixteen-bit format it can take tens of thousands of distinct values. Quantization says: we do not need that much precision. Pick a small set of allowed values, for example sixteen levels for four bits, and snap every weight to the nearest one. You store the small integer, plus a little information to map it back.

• FP16: 65,536 patterns
• INT4: only 16 values

Why does this not destroy the model? Trained weights are clustered near zero in a bell shape, and a network has billions of them, so rounding errors are small and partly cancel out in the big sums inside each matrix multiplication. The model was also trained with noise, so it is somewhat tolerant. But tolerant is not invincible, and we will see where it breaks.

• Weights are noisy and redundant
• Small errors average out

Here is the core formula. The quantized integer x q equals round of x divided by s, plus z. The symbol x is the original float, s is the scale, a positive float, and z is the zero-point, an integer. To recover an approximation, dequantize: x hat equals s times the quantity x q minus z. The difference between x and x hat is the quantization error.

• x_q = round(x / s) + z
• x_hat = s · (x_q − z)

How do we choose s and z? We want the float range from min to max to land on the integer range zero to two to the b minus one. So s equals max minus min, divided by two to the b minus one. The zero-point z equals round of minus min over s, which guarantees that the float value zero maps exactly to an integer. That matters because zero appears everywhere in padding and sparse structure.

• s = (max − min) / (2^b − 1)
• z = round(−min / s)

There are two flavors. Symmetric quantization forces z to zero and uses s equals the largest absolute value divided by the largest integer, say seven for signed four-bit. It is simple and fast. Asymmetric quantization uses the true minimum and maximum, so if the data is skewed, it wastes fewer integer codes. The price is an extra zero-point to store and to handle in the kernel.

• Symmetric: z = 0, s = absmax / qmax
• Asymmetric: uses min and max

Inside the range, the rounding error for any weight is at most half a step, so s divided by two. Notice what that implies. Fewer bits means fewer levels, which means a bigger step s, which means a bigger worst-case error. And the step is proportional to the range, so anything that stretches the range, like a single huge value, makes every weight coarser.

• Rounding error ≤ s / 2
• Fewer bits → bigger s

Let us quantize six weights to asymmetric unsigned INT4, which has codes zero through fifteen. The weights are minus zero point eight two, minus zero point three one, zero point zero five, zero point two seven, zero point six four, and one point one zero. The minimum is minus zero point eight two, and the maximum is one point one zero.

• −0.82, −0.31, 0.05
• 0.27, 0.64, 1.10

The range is one point one zero minus minus zero point eight two, which is one point nine two. Divide by fifteen and the scale s is zero point one two eight. For the zero-point, zero point eight two divided by zero point one two eight is six point four one, which rounds to six. So float zero will map to integer code six.

• s = 1.92 / 15 = 0.128
• z = round(0.82 / 0.128) = 6

Now each weight. Minus zero point eight two over zero point one two eight is minus six point four one, rounds to minus six, plus six gives code zero. Minus zero point three one gives minus two, code four. Zero point zero five rounds to zero, code six. Zero point two seven gives two, code eight. Zero point six four is exactly five, code eleven. And one point one zero is eight point five nine, rounds to nine, code fifteen.

• Codes: 0, 4, 6, 8, 11, 15
• Divide, round, add 6

Dequantize with s times code minus six. Code zero gives minus zero point seven six eight, an error of zero point zero five two. Code four gives minus zero point two five six, error zero point zero five four. Code six gives exactly zero, error zero point zero five. Then zero point two five six, zero point six four, and one point one five two, with errors zero point zero one four, zero, and zero point zero five two. Every error is under half a step, zero point zero six four, as promised.

• Recovered: −0.768, −0.256, 0, 0.256, 0.64, 1.152
• Max error 0.054 ≤ s/2 = 0.064

Six weights in sixteen-bit format took ninety-six bits. Now they take twenty-four bits of codes, plus one shared scale and one shared zero-point. With six weights the overhead looks large, but with thousands of weights sharing one scale, it nearly vanishes. That brings up the key design question: how many weights should share one scale?

• 6 × 16 bits = 96 bits
• 6 × 4 bits + s, z overhead

A weight matrix can share scales at different granularities. Per-tensor uses one scale for the whole matrix, which is cheap but coarse. Per-channel uses one scale for each row. Per-group, the standard for four-bit LLMs, uses one scale for each block of typically sixty-four or one hundred twenty-eight consecutive weights. Smaller groups track local ranges better, at the cost of storing more scales.

• Per-tensor: one s for all
• Per-channel: one s per row
• Per-group: one s per 128 weights

Smaller groups cost memory. With a group size of one hundred twenty-eight and a sixteen-bit scale, the scale adds sixteen divided by one hundred twenty-eight, which is one eighth of a bit per weight. Add a four-bit zero-point and you land near four point one six bits per weight. So a so-called four-bit model really averages a little over four bits.

• Group 128, FP16 scale: +0.125 bits
• Plus zero-point: about 4.25 bits

Now the villain. Suppose a group holds small weights around zero point one, and one weight equal to ten. Symmetric INT4 gives s equal to ten divided by seven, about one point four three. Every small weight divided by one point four three is below zero point one, and rounds to zero. One big value has erased all the information in its neighbors.

• Weights near 0.1, one value 10
• s = 10 / 7 ≈ 1.43
• Small weights round to 0

In large language models this problem is worse for activations than for weights. Researchers observed that in big transformers a handful of hidden channels carry values far larger than the rest, across almost all tokens. If you quantize activations with one scale, those few channels dictate the step size and crush everything else. This is why quantizing activations is much harder than quantizing weights.

• A few channels get huge values
• Weights are mild, activations are not

The simplest method is round-to-nearest: quantize each weight independently with the formula we just used. It works acceptably at eight bits, but at four bits and below it often hurts. The reason is that what we actually care about is not the error in each weight, it is the error in the layer output, W times X, and weight errors combine through the inputs in ways that matter.

• RTN: round each weight alone
• Ignores how errors combine

GPTQ is a post-training method that does exactly that. For each layer it feeds in a small calibration set of real text, and chooses quantized weights that minimize the squared difference between the original output and the quantized output. The curvature of this error with respect to the weights, the Hessian, is determined by the inputs, which is the second-order information that GPTQ exploits.

• Minimize ‖WX − ŴX‖²
• Uses calibration data
• Second-order information

The clever part is error compensation. GPTQ quantizes weights one after another, and after rounding each one, it uses the Hessian to adjust the not-yet-quantized weights so they absorb the damage. Weights that are correlated through the inputs can cancel each other's errors. It processes whole columns in blocks for speed, and can quantize very large models in hours on a single GPU, without any gradient training.

• Quantize one weight at a time
• Push its error onto remaining weights

AWQ, activation-aware weight quantization, starts from an observation: a small fraction of weight channels matter far more, and you can find them by looking at activations, not at weights. Channels that see large activations amplify their weight errors. AWQ multiplies those weight channels by a factor, and divides the matching input by the same factor, so the math is unchanged, but the important weights now sit on a finer relative grid.

• Salience depends on activations
• Scale up important channels
• Fold inverse scale into input

A tiny example. Say the grid step is zero point two. A weight of zero point three lands halfway between grid points, error up to zero point one. Multiply that channel by two and the weight becomes zero point six, exactly on the grid. The input is halved, so the product is unchanged. In practice AWQ searches for the scaling using calibration data, balancing gains on salient channels against harm to the others.

• Weight 0.30 on a loud channel
• Scale by 2, step stays the same

Compare the two. GPTQ minimizes layer output error with Hessian-based compensation. AWQ rescales salient channels, then rounds. Both use a small calibration set, neither needs full training, and both typically produce weight-only four-bit models. Which is better varies by model and task, so measure on your own workload instead of trusting a leaderboard.

• GPTQ: error compensation
• AWQ: activation-aware scaling
• Both: calibration, weight-only

Weight-only schemes, written W four A sixteen, keep activations in sixteen-bit and dequantize weights on the fly. They save memory and bandwidth. Weight-and-activation schemes like W eight A eight also run the matrix multiply itself in eight-bit integer math, which can raise compute throughput, but they must tame activation outliers. FP8 is an eight-bit floating-point format on newer hardware, with dynamic range that suits outliers better than INT8.

• W4A16: weights int4, activations FP16
• W8A8: both int8
• FP8: floating 8-bit

You will also meet GGUF, the file format used by llama.cpp. It packs the weights, tokenizer and metadata into one file, and offers many block-wise quantization types at different bit widths, so you can trade size against quality. This ecosystem makes it practical to run large models on laptops and consumer hardware, including partly on CPU.

• GGUF: single-file model format
• Block-wise quants, many bit widths
• Runs on CPU and consumer GPUs

Memory math is simple. Parameters times bytes per parameter. A seven billion parameter model takes fourteen gigabytes in sixteen-bit, seven in INT8, and about three and a half in INT4. For seventy billion, that is one hundred forty, seventy, and thirty-five gigabytes. Add a little for scales, and remember the KV cache and activations still need room on top of the weights.

• 7B: 14 GB → 7 GB → 3.5 GB
• 70B: 140 GB → 70 GB → 35 GB
• Add scales, KV cache, activations

Why is it faster? As we saw with the KV cache lesson, generating one token means streaming essentially all the weights from memory, while the arithmetic units wait. If the weights are four times smaller, you move roughly four times fewer bytes. The dequantization costs a little compute, but compute was idle anyway. Real speedups depend on the kernel and the batch size, and they shrink as batches grow compute-bound.

• Decoding is memory-bound
• Bytes read per token drop 4x
• Dequantize cost is small

How much quality do you lose? It depends on the model, the method, the bit width, and the task. In general, eight-bit is usually very close to the original, four-bit is often acceptable, and below that the risk grows quickly. Tasks like math, code and long reasoning chains tend to be more sensitive than casual chat. I will not quote a number, because any number is only true for one setup.

• Eight-bit: usually very close
• Four-bit: often fine, not always
• Lower bits: risk grows

Time for code. In PyTorch, reshape the weight tensor to rows of group size, say one hundred twenty-eight. For each group compute the minimum and maximum, then s equals max minus min divided by fifteen, with a small floor to avoid dividing by zero, and z equals round of minus min over s. Then q equals clamp of round of w over s plus z, between zero and fifteen. Dequantize with s times q minus z.

• Reshape into groups
• Per-group min, max
• Quantize and dequantize

Now test it. Make a random normal weight matrix, quantize it, dequantize, and compute the mean squared error. Sweep the group size from thirty-two up to a thousand and watch the error rise as groups get bigger. Then inject one huge value into a matrix and see the error jump. You are reproducing the outlier problem with your own hands.

• Mean squared error per tensor
• Try group sizes 32, 128, 1024
• Inject one outlier

With Hugging Face you can load a model in sixteen-bit, call get memory footprint, then load the same model again passing a quantization config for four-bit, for example through bitsandbytes, or load a pre-quantized GPTQ or AWQ checkpoint. Print the footprint both times. For the weights you should see a drop approaching four times, though embeddings and some layers may stay in higher precision.

• from_pretrained with quantization config
• Compare memory footprint
• Compare sample outputs

Then generate. Use the same prompts and greedy decoding, so any difference comes from quantization and not sampling randomness. Compare the answers side by side. Often they are nearly identical, with occasional divergences in wording or in harder reasoning steps. Use prompts from your actual application, because that is the only distribution you care about.

• Same prompts, greedy decoding
• Diff the answers
• Check your real task

The classic mistake is judging quality by perplexity alone. Perplexity averages next-token loss over text, so it can look almost unchanged while specific abilities, like exact arithmetic, code, long context, structured output or tool calls, quietly degrade. Evaluate quantized models on your real tasks, with your own test set, like we built in the evaluation lessons.

• Perplexity is an average
• Misses rare failures
• Evaluate on real tasks

A few more traps. Calibrating on data that looks nothing like your use case. Budgeting only for weights and forgetting that the KV cache and activations also consume memory. And assuming a four-times smaller model is four times faster at every batch size, when large batches become compute-bound. Finally, always check the license and the exact quantization settings of any checkpoint you download.

• Tiny calibration data mismatch
• Forgetting KV cache memory
• Assuming speedup at any batch size

To recap. Quantization stores weights as small integers with a scale and zero-point, using x q equals round of x over s plus z. Smaller groups and careful scales limit the damage from outliers. GPTQ minimizes layer output error with second-order compensation, and AWQ protects salient channels using activation statistics. The payoff is mostly less memory traffic, and quality must be tested on your own tasks.

• x_q = round(x/s) + z
• Groups tame outliers
• GPTQ compensates, AWQ protects
• Memory traffic is the win

Now that your model fits in memory, how do you serve many users at once? In the next episode we cover vLLM, with continuous batching and PagedAttention.

• Continuous batching
• PagedAttention
• Subscribe for the next lesson