What is quantization of a model?

Quantization is the technique that lowers the numeric precision used to represent a model's weights, typically from 16 or 32 bits down to 8 or even 4 bits per parameter. The model takes up less memory, needs less bandwidth to load and responds faster, at the cost of an accuracy loss that in most tasks is small. It can be applied after training, the more common route, or simulated already during training so the model adapts to the reduced precision from the start. In practice it is the lever that makes a large model usable on more modest hardware: a 4-bit model takes up roughly a quarter of the memory of its 16-bit version, meaning it can run on a company GPU instead of a dedicated cloud cluster, with noticeably lower inference costs, a trade-off that is best measured on a case-by-case basis.

See also: SLM · A small, specialized language model: a fraction of an LLM's cost, runs even on-premise, and for focused tasks it is plenty.

How it works

Each weight, normally a floating-point number with many digits of precision, is approximated with a number represented in fewer bits, using a scheme that minimizes the introduced error. Some approaches quantize after training (post-training quantization), others simulate the reduced precision already during training to adapt better. The typical outcome: a 4-bit model takes up roughly a quarter of the memory of a 16-bit one, with a quality loss ranging from negligible to noticeable depending on the task and how far the compression is pushed. At equal bit width the formats are not equivalent: NVFP4, NVIDIA's 4-bit format supported by Blackwell GPUs, keeps one scaling factor per sixteen values rather than one per tensor, so the error stays local, and NVIDIA reports under 1% degradation versus FP8, the 8-bit format, in its own benchmarks (NVIDIA Developer). Compressing the weights and compressing the KV cache remain two separate decisions with their own constraints, though: the second one is covered in TurboQuant.

Why it matters for your business

Quantization is what makes it practical to run an SLM, or even a medium-sized model, on a company server instead of a dedicated cloud cluster: less GPU memory needed, lower inference costs, and a real option to keep both data and model inside the company perimeter. The trade-off has to be measured case by case: for classification or extraction tasks the loss is often imperceptible, for tasks requiring fine-grained reasoning it can show, which is exactly why an evaluation suite is needed before deciding how far to compress. And not all quants are equal: on real agentic workloads, an 8-bit build published by a community member held up on tool calls better than the official 8-bit release of the same model, while a 4-bit variant failed them repeatedly (Level1Techs, August 2026). Choosing the quantized checkpoint is a quality decision, not only a memory decision.