Quantization vs Distillation: Cheap Fix or Real Rebuild

Quantization compresses the model you have in hours. Distillation trains a new one that can match or beat it, at a much higher cost.

Quantization vs distillation is a choice between two different kinds of cheap. Quantization takes a model you already trust and stores its weights at lower numeric precision, INT8 or INT4, so it takes less memory and runs faster, for a few hours of GPU time and little else. Distillation takes a larger model's behavior and trains a new, smaller model to reproduce it, which can take weeks of real training and, in return, can match or beat the original on your specific task. One shrinks a model. The other builds a different one.

I get asked to make this call almost as often as the fine-tuning question, usually by a team staring at an inference bill that needs to shrink by next quarter, not next year. Both techniques promise "smaller and cheaper." They get there through different mechanisms, on different timelines, with a different quality ceiling waiting on the other side. Picking the one that matches your actual constraint, not the one that sounds more sophisticated in a planning doc, is most of the decision.

This sits under my broader framework for choosing an LLM, because quantization and distillation are both downstream of a model you already picked. Get the base model right first. Then decide how to shrink it.

Key takeaways

• Quantization compresses; distillation rebuilds. Quantization stores the same model's weights at lower precision in hours. Distillation trains a new, smaller model from a teacher's outputs, which takes days to weeks.
• Quantization can only approach the original's quality; distillation can occasionally beat it. A well-trained student, using the teacher's outputs as a richer training signal, has outperformed models of the same size trained conventionally from scratch.
• The cost gap is enormous and well documented. GPTQ can quantize a 175-billion-parameter model to 4-bit in roughly four GPU-hours. Distillation-based model families measure their training in days on large GPU clusters.
• The quality ceiling is different by design. Quantization's floor is your original model minus a small, measurable precision cost. Distillation's ceiling is the teacher's own judgment, closed by a good training recipe but rarely exceeded.
• The strongest production pattern uses both. Distill first to set the right capability floor, then quantize the distilled model to shrink it further. Shipped models do exactly this today.

What Quantization Actually Does to a Model You Already Have

Quantization starts from a model that already works and asks one question: how few bits can each weight use before the output changes? A model trained at 16-bit precision gets its weights re-stored as 8-bit or 4-bit integers, sometimes 8-bit floats. Fewer bits per weight means less memory and less data to move per token, and moving data is most of what inference actually spends time on. Nothing about the model's training changes. You are not teaching it anything new. You are storing what it already learned more coarsely.

The practical cost is remarkably small. Post-training quantization (PTQ) takes a finished model and a small calibration dataset and produces a compressed version in minutes to hours, with no retraining. GPTQ, one of the standard methods, quantized a 175-billion-parameter model down to 3 or 4 bits in roughly four GPU-hours, with accuracy loss the original paper called negligible against the full-precision baseline (Frantar et al., ICLR 2023). That is an afternoon of compute for a model that took months to train.

The quality trade is real but narrow. Eight-bit precision is close to lossless on almost everything. Four-bit is close to free on summarization, classification, and extraction, and measurably worse on multi-step reasoning, code generation, and long-context tasks, where small per-step errors compound. I go deep on the specific methods, GPTQ, AWQ, GGUF, and FP8, and exactly where each one holds or breaks, in my guide to LLM quantization. The short version for this comparison: quantization changes how a model is stored, not what it learned, and that ceiling is both its strength and its limit.

What Distillation Actually Does: A New Model, Not a Compressed One

Distillation starts from a different premise. A large "teacher" model already does the task well, and you train a separate, smaller "student" model to imitate it. You run the teacher on a set of inputs, capture its outputs or its reasoning traces, and train the student on that captured signal instead of on your own labeled data. The student is not the teacher with fewer bits. It is a different set of weights, trained from a different starting point, shaped by a different loss function.

That difference is why distillation costs so much more than quantization, and why it can pay off so much more. Training a student model is a real training run: you need the teacher's outputs across a representative set of inputs, a training loop, and a validation pass, measured in days to weeks rather than hours. NVIDIA's Minitron approach, which derives smaller models from an already-trained 15-billion-parameter Nemotron model, still needed a genuine distillation-based retraining pass, one that used up to 40 times fewer tokens than training from scratch, for a 1.8x compute-cost saving across the full model family (Sreenivas et al., 2024). Cheaper than training from scratch, yes. Nowhere near quantization's four-GPU-hour price tag.

What that extra cost buys is a model that is not merely smaller. It is genuinely a different, independently capable model. I lay out the full teacher-student mechanics, and how distillation compares specifically to fine-tuning, in distillation vs fine-tuning. This comparison is narrower: distillation against quantization, two techniques that solve different problems even though both end with a smaller model on disk.

Quantization vs Distillation: The Core Trade-off

Line the two up on the same axes and the difference stops being abstract.

Quantization is a storage decision with a bounded, well-understood cost. Distillation is a training decision with a real chance of falling short, and a real chance of a better cost-quality trade than the teacher itself. Quantization is worth trying almost by default, because the downside is small and measurable. Distillation is worth it once sustained inference volume justifies a real training investment.

Both bets sit inside a bigger pattern: scale keeps winning on raw capability, and a distilled or quantized copy rarely closes the gap to the frontier model it came from. I make the full argument in my book The Bitter Lesson, Revisited. Winning on capability and winning on your unit economics are different contests. Quantization and distillation are both bets that you only need to win the second one, for this task, at this volume.

Real Models That Prove the Ceiling: Minitron and Llama 3.2

Two shipped model families make the distillation side concrete. NVIDIA's Minitron models, derived from a 15-billion-parameter Nemotron model through pruning combined with distillation-based retraining, perform comparably to Mistral 7B, Gemma 7B, and Llama-3 8B, models trained on significantly more tokens from scratch, and post up to a 16% MMLU improvement over training an equivalent model from scratch (Sreenivas et al., 2024). That is distillation doing something quantization structurally cannot: producing a smaller model that beats what the same parameter count would achieve trained conventionally.

Meta's Llama 3.2 1B and 3B models tell the same story from a different provider. Meta built them through structured pruning from Llama 3.1 8B, then distilled them using logits from the 8B and 70B models as token-level targets during pretraining. The resulting 3B model outperforms Gemma 2 2.6B and Phi-3.5-mini on instruction-following, summarization, and tool use (Meta AI, 2024). Quantization can only approach the original model's quality from below. Distillation, done well, can beat what training the small model alone would have produced.

When Quantization Is the Right Call

Reach for quantization first in almost every case, because trying it costs so little relative to the payoff:

• You already trust the model's task quality. Quantization does not change what the model can do, only what it costs to run it. If accuracy is already where you want it, quantization is the fastest path to a lower bill.
• You need results this week, not this quarter. A PTQ pass finishes in hours. There is no faster lever for cutting memory and inference cost that still ships this sprint.
• Your task tolerates a small precision loss. Summarization, classification, extraction, and retrieval-grounded answering hold up well under 4-bit quantization. Watch multi-step reasoning and code generation closely, or leave those paths at 8-bit.

The failure mode to avoid is treating quantization as free. It is cheap, not free, and the honest version of "cheap" still means you evaluate the quantized model on the same eval set you used for the full-precision one, not the other way around.

When Distillation Is the Right Call

Distillation earns its cost in a narrower set of conditions:

• Quantization alone doesn't get you small enough. Four-bit quantization buys roughly 4x. If your target device or your target unit economics need 10x or more, only a genuinely smaller model, meaning fewer parameters, closes that gap.
• Sustained volume justifies the training cost. Distillation's training run, days to weeks, only pays off once query volume is high enough that the smaller student's per-query savings recoup it. Low-volume workloads rarely clear that bar.
• You want a model built for your task, not just a compressed copy of a general one. A student trained on a teacher's outputs for your actual task distribution can specialize further than a quantized general model ever will, because it is a real training pass, not a storage change.

Name the honest trade-off: a distillation run can fail. You capture a teacher's outputs, train a student, and validate it, and if the student does not clear your eval bar, you have spent the training budget and still need another plan. Quantization almost never fails this completely; it degrades a known amount on a known set of tasks. Distillation is a bet with a real training cost attached.

The Strongest Pattern: Distill First, Then Quantize the Student

The most common mistake I see is treating this as either/or. In production, the two techniques stack, and stacking them is usually the right call once you need real cost reduction. Distillation gets you a smaller model with the right capability floor. Quantization then shrinks that already-smaller model further, for the same few-hours cost it would take on any model.

DeepSeek's own distilled models show this stack in production, not in a paper. DeepSeek-R1-Distill-Qwen-32B is already a distilled model, trained on reasoning traces from the much larger DeepSeek-R1. Publicly shipped GGUF builds of that same distilled model quantize it further, from 65.54GB at full BF16 precision down to 19.85GB at a common 4-bit setting, and as low as 9.03GB at the most aggressive setting (bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF). Distillation did the capability shrink. Quantization did the storage shrink on top of it. Neither step alone gets a model that fits on a single consumer GPU at usable speed.

The order matters. Distill first, evaluate the student against ground truth, and only then quantize, because quantizing before you know the student's quality baseline makes it impossible to tell which step caused a regression if one shows up later. Quantize last, and you are compressing a model whose quality you have already verified.

What Compression Quietly Breaks (and What It Doesn't)

Both techniques share a failure mode that is easy to miss because the aggregate benchmark number looks fine. A 2026 unified evaluation of pruning, quantization, and distillation across more than 40 datasets found that factual recall is largely preserved under compression, while multi-step reasoning, multilingual ability, and instruction-following degrade disproportionately (UniComp, 2026). Your aggregate score can hold steady while the exact capability your product depends on quietly erodes underneath it.

The same research found that retained benchmark performance does not indicate preserved safety, fairness, or privacy behavior. A compressed model that scores the same on your accuracy eval can behave differently on the evals you did not think to run. That is not a reason to avoid either technique. It is a reason to run both through the same evaluation gate you would run any other model change through, on tasks that actually matter to your product, not just the tasks that are easy to benchmark.

Is quantization or distillation better for reducing LLM costs?

It depends on how much reduction you need and how much time you have. Quantization gets you roughly 2 to 4x smaller in hours, with a small, measurable quality cost. Distillation can get you 10x smaller or more, but the training run takes days to weeks and needs sustained query volume to pay off. Most teams should try quantization first and reach for distillation only when it alone doesn't hit the target.

Can you quantize a distilled model?

Yes, and it is one of the strongest patterns in production today. Distill a large teacher's capability into a smaller student first, evaluate that student against ground truth, then quantize the student the same way you would quantize any other model. DeepSeek's distilled Qwen and Llama variants ship as further-quantized GGUF files precisely because this stack works.

Does quantization or distillation hurt model quality more?

Quantization's quality cost is smaller and more predictable, usually within a few percentage points on hard tasks and close to unmeasurable on easy ones. Distillation's quality outcome is more variable. A well-executed distillation can match or even beat a conventionally trained model of the same size, as NVIDIA's Minitron and Meta's Llama 3.2 models show. A poorly executed one can underperform badly. Both need to be measured against your own eval set, not assumed.

How much does distillation cost compared to quantization?

Quantization is a hardware-time cost measured in hours. GPTQ can quantize a 175-billion-parameter model to 4-bit in roughly four GPU-hours. Distillation is a training-run cost measured in days to weeks, plus the infrastructure to generate and store a teacher's outputs. That cost only makes sense once query volume is high enough to recoup it through cheaper inference over time.

Quantization vs distillation is not a debate to win in the abstract. It is a decision about how much time you have and how far you need to shrink the bill. Quantize the model you already trust and measure the cost. Distill when quantization alone will not get you there, and you have the volume to justify the training run. If you want a team that has built both pipelines, the calibration harness and the teacher-student loop, and can tell you which one your traffic needs, that is the kind of work ViitorCloud's technology consulting practice does under a real deadline.