The takeaway: Quantization is the trick that lets a model far too big for your laptop run on it anyway. It works by storing the model's numbers at lower precision — swapping 16-bit values for 8-bit or 4-bit ones — which shrinks the model's memory footprint several-fold and speeds it up, at the cost of a little accuracy. It is the single biggest reason you can now run a capable model on a phone, a gaming GPU, or a single server card instead of a rack of them.

What "precision" actually means here

A neural network is, at heart, a very large pile of numbers — its weights. In training, those weights are usually stored as 16-bit floating-point values (formats called FP16 or BF16). Sixteen bits per number buys a lot of fine gradation, which training needs. But once a model is trained, that much precision is often overkill just to run it.

Quantization reduces the number of bits used to store each weight — commonly from 16 bits down to 8 (INT8) or 4 (INT4). The saving is close to linear: a 70-billion-parameter model needs roughly 140 GB of memory for its weights in 16-bit, about 70 GB in 8-bit, and roughly 35 GB in 4-bit (the running model needs a bit more on top for its working memory). That is the difference between "needs several data-centre GPUs" and "fits on one card" — or between "won't load on your laptop" and "runs fine."

How you squeeze a number into fewer bits

The core operation is a mapping. You take the range of values a group of weights actually spans, and you divide it into a smaller number of evenly spaced levels — 256 levels for 8-bit, 16 for 4-bit — recording a scale factor (and sometimes a zero-point) so the low-precision integers can be turned back into approximate real numbers at run time. Do this per-tensor (one scale for a whole layer) and it's crude; do it per-channel or per-small-group of weights and you keep much more of the original detail.

There are two broad ways to apply it:

• Post-training quantization (PTQ): take a finished model and quantize it directly, with no retraining. It's fast and cheap, and it's what most downloadable "quantized" models use. Well-known PTQ methods include GPTQ and AWQ, and the GGUF format used by the llama.cpp ecosystem.
• Quantization-aware training (QAT): simulate the rounding during training or fine-tuning so the model learns to compensate. It costs more compute but generally preserves more quality, especially at very low bit-widths.

A well-known complication is outliers: in large language models a handful of activation values are enormous and, if you're not careful, they stretch the range so far that everything else loses precision. Techniques like LLM.int8() and SmoothQuant handle those outliers directly, while AWQ uses them the other way round — reading the activation pattern to decide which weights are too important to quantize crudely — so the outliers inform the quantization rather than wreck it.

What it costs you

Quantization is not free — it trades memory and speed for a little accuracy. Roughly:

• 8-bit is usually close to lossless for inference; most users can't tell the difference.
• 4-bit typically costs a small, often-acceptable drop in quality, and it's the sweet spot most local models ship at today.
• Below 4-bit (3-bit, 2-bit, and experimental ~1.58-bit schemes like BitNet) the losses grow and the engineering gets harder — this is an active research frontier, not a solved problem.

The degradation isn't uniform, either: quantization tends to bite hardest on the tasks that need the most precision, like long chains of reasoning or exact arithmetic, and least on casual conversation.

Why it matters

Two reasons. First, access: quantization is the main thing standing between "frontier-scale open models exist" and "you can actually run one." The open-weights models that made headlines this year are usable on ordinary hardware largely because the community quantizes them the day they drop.

Second, cost and efficiency: lower precision means less memory bandwidth and cheaper maths, so quantized models are faster and cheaper to serve — a close cousin of the efficiency gains from mixture-of-experts architectures. It even feeds back into training: QLoRA fine-tunes a 4-bit-quantized base model, which is how people tune very large models on a single GPU.

Quantization won't turn a small model into a big one — it makes a big model portable, not smarter. But portability is most of what has put capable AI onto everyday hardware, and it's why "can I run this locally?" is, more and more often, a yes.

Sources: QLoRA (Dettmers et al., 2023), GPTQ (Frantar et al., 2022), AWQ (Lin et al., 2023), LLM.int8() (Dettmers et al., 2022), Hugging Face — Quantization.