Quantization

The Tokens Per Second (TPS) article closed on a promise: the technique most directly responsible for improving both speed and memory footprint at once. This article covers exactly that: quantization.

The simple definition

Quantization is the process of reducing the numerical precision used to store a model’s parameters, representing each weight with fewer bits, in exchange for a smaller memory footprint and faster computation. Recall from the VRAM article’s hard constraint — a model’s entire set of parameters has to fit in a GPU’s memory to run at all. Quantization directly attacks the size of that requirement, storing each individual weight using a coarser, less precise number, so the same model takes up dramatically less space.

Why fewer bits per number actually works without breaking the model

Recall from the Weights article that a neural network’s parameters are just numeric values, learned through training. It turns out these values don’t need to be stored with extreme numerical precision to still produce genuinely useful, coherent output — recall from the TPU article’s own design insight, that Google’s chip was built specifically to tolerate reduced computational precision, since neural networks are naturally somewhat robust to small, consistent rounding.

Quantization exploits this same real, structural tolerance directly: converting weights from a high-precision format down to a lower-precision one, covered in specific detail throughout the rest of this phase, shrinks the model substantially while typically preserving most of its real, practical capability.

flowchart LR
    A[Original model: high-precision weights, e.g., FP32] --> B[Quantization: convert to lower precision, e.g., INT4]
    B --> C[Same model, dramatically smaller memory footprint]
    C --> D[Fits on hardware that couldn't run the original]

The real story: one developer’s explicit goal was a MacBook, not a data center

This deserves to be told directly, because it’s a genuine, well-documented case of quantization’s real-world impact being stated as an explicit goal from day one. In March 2023, developer Georgi Gerganov released llama.cpp, an open-source library for running Meta’s LLaMA models — and its own published documentation stated its main goal with total clarity: to run a large language model using 4-bit integer quantization on an ordinary MacBook.

This wasn’t a minor technical footnote; it was the entire point. By converting a model’s weights down to 4-bit precision, covered fully in the INT4 article later in this phase, llama.cpp made it genuinely possible for enthusiasts with consumer hardware, no data center required, to run billion-parameter models locally.

The project has since become, in its own words, “the de facto standard as the core of almost all local inference tools,” directly powering real, widely used applications including Ollama and LM Studio.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of the difference between describing a person’s height as “5 feet, 9 and three-quarter inches” versus simply “about 5 foot 10” — the second description loses some genuine precision, but for almost every practical purpose, that lost precision doesn’t actually matter, and the shorter, rounder number is far easier to work with and remember.

Where this breaks down: Rounding a height is a simple, one-time human approximation. Quantizing a model’s weights involves genuine, careful mathematical technique — converting values between number formats in a way that minimizes the accumulated error across potentially billions of individual weights working together, a precise engineering problem, not casual rounding, since naive, careless quantization really can noticeably degrade a model’s output quality.

Why this technique matters for genuinely every article covered throughout this phase

It’s worth naming this connection explicitly, since quantization is really the technique that ties this entire phase’s concerns together. A quantized model requires less VRAM, covered in its own article, meaning it fits on cheaper, more widely available hardware. It typically runs with a higher TPS, covered in the previous article, since moving and computing with smaller numbers is genuinely faster. And it directly improves both latency and throughput, covered in their own articles, by reducing the raw computational burden every single request places on the underlying hardware.

A concrete example, layered

For a simple beginner example: a 7-billion-parameter model that might require around 28 gigabytes of VRAM at full precision can shrink to roughly 4 gigabytes after aggressive quantization, turning a model that needed a specialized, expensive GPU into one that runs comfortably on an ordinary laptop. For a production example: TheBloke, a real, widely known figure in the open-source AI community, became known specifically for publishing pre-quantized versions of popular models on Hugging Face, letting developers download models already converted to formats like the GGUF format llama.cpp uses, rather than needing to perform the quantization process themselves.

The exact memory savings, laid out precisely

This is worth spelling out directly with real numbers, since the format-by-format articles later in this phase each mention their own savings individually — it’s genuinely useful to see all five side by side. Each parameter costs 4 bytes in FP32, 2 bytes in FP16 or BF16, 1 byte in INT8, and roughly 0.5 bytes in INT4.

Applied to a 7-billion-parameter model, that’s 28 GB (FP32), 14 GB (FP16/BF16), 7 GB (INT8), and 3.5 GB (INT4) — just for the weights, before any additional memory needed during computation. Scaled up to a 70-billion-parameter model, the same math produces 280 GB, 140 GB, 70 GB, and 35 GB respectively — the exact difference between needing a multi-GPU server rack and needing a single consumer GPU.

flowchart LR
    A["FP32: 4 bytes/param — 7B model = 28 GB"] --> E[Baseline]
    B["FP16/BF16: 2 bytes/param — 7B model = 14 GB"] --> E
    C["INT8: 1 byte/param — 7B model = 7 GB"] --> E
    D["INT4: ~0.5 bytes/param — 7B model = 3.5 GB"] --> E

Why quantization has a real, honest limit

It’s worth being direct about a genuine trade-off, not presenting quantization as a costless technique. Reducing precision too aggressively does measurably degrade a model’s output quality — genuine, published benchmarks show accuracy gradually declining as bit-width drops, with the specific trade-offs at each precision level covered directly in the FP16, INT8, and INT4 articles later in this phase. Quantization is a real, powerful lever, but like every other technique covered throughout this phase, it’s a deliberate trade-off, not a pure, unconditional improvement.

Common misconception

Where one scale is shared

Quantization methods differ partly in how many values share a scale:

Per-tensor: one scale covers a whole tensor. It uses little metadata but must fit many different values into one range.
Per-channel: each output channel gets its own scale. It usually represents varied channels more accurately.
Group-wise: small groups of weights share scales. This is common in aggressive LLM weight quantization.

one large group -> fewer scales, less metadata, rougher fit
smaller groups  -> more scales, more metadata, closer fit

The best choice depends on model quality, file size, supported kernels, and measured speed. More detailed scaling can protect accuracy without being the fastest representation on every device.

Calibration in plain language

Calibration sends representative examples through the model and observes typical value ranges. If the examples are unrepresentative, the chosen ranges may work well in the laboratory but clip important values in production.

The quantized model must be evaluated on the same real tasks and populations for which it will be used.

What gets quantized?

A model contains several kinds of numbers. A serving setup may use different formats for each one:

weights       -> often quantized to INT8 or 4-bit
activations   -> may stay FP16/BF16 or use a supported lower format
KV cache      -> may use its own reduced format
accumulation  -> may use a wider format to protect accuracy

Therefore, “an INT4 model” usually does not mean that every calculation everywhere uses four-bit integers.

Two main times to quantize

Post-training quantization (PTQ): train first, then convert and calibrate the model.
Quantization-aware training (QAT): simulate low precision during training so the model learns to tolerate it.

PTQ is simpler and cheaper. QAT can preserve more quality when aggressive quantization would otherwise cause too much damage.

One memory example

For 7 billion weights, the theoretical weight storage changes from about 28 GB at FP32 to 14 GB at FP16/BF16, 7 GB at INT8, or 3.5 GB at four bits. Actual files and VRAM usage differ because scales, metadata, KV cache, and runtime workspace also take space.

Hugging Face Transformers supports bitsandbytes loading for 8-bit and 4-bit models. NVIDIA TensorRT-LLM supports several reduced formats, including INT8 and INT4, as part of production inference optimization.

Verified sources

A frequent beginner assumption: that quantization is a simplified, lower-quality version of a model, similar to a blurry, compressed copy of a photograph. As the MacBook example demonstrated, well-executed quantization typically preserves the vast majority of a model’s real, practical capability — the goal isn’t a degraded shortcut, but a genuinely careful engineering technique specifically designed to minimize quality loss while maximizing accessibility.

Where this fits in what comes next

You now understand the general technique of reducing numerical precision to shrink and speed up a model. The next several articles — FP32, FP16, BF16, INT8, and INT4 — cover the specific, named number formats quantization actually converts between, each with its own real, distinct trade-offs.

In one sentence

Quantization reduces the numerical precision used to store a model’s weights, shrinking its memory footprint and increasing its speed, and llama.cpp’s explicit, published goal — running a large language model via 4-bit quantization on an ordinary MacBook — remains one of the clearest, most consequential real demonstrations of this technique turning inaccessible, data-center-scale models into something anyone could run locally.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026