Quantization and pruning can cut model size and speed up inference—without destroying quality. This guide shows you what to try first, what usually breaks, and the exact sanity checks that prevent “it looked good on my laptop.”
Quickstart: make a model smaller (the safe order)
If you only do one thing: optimize with measurements, not vibes. Here’s the fastest “high chance of success” path.
- Baseline: measure accuracy + latency + memory on your real target device.
- INT8 quantization: try post-training quantization (PTQ) first.
- Quantization-aware training (QAT): use if PTQ drops quality too much.
- Structured pruning: prune channels/heads only if you can retrain + validate.
- Distillation: if you need a “smaller-but-still-smart” model reliably.
What “success” looks like
- Quality: within your acceptable drop (often 0–2% relative for many tasks)
- Latency: faster on the real runtime (TFLite / ONNX / TensorRT)
- Memory: lower peak RAM / VRAM, fewer cache misses
- Stability: no weird edge-case failures (calibration + stress tests)
The #1 trap
A model can get smaller but not faster, because the runtime/hardware can’t accelerate your chosen format (or you pruned in a way that doesn’t map to faster kernels).
- Measure on-device
- Confirm the backend uses INT8 kernels
- Validate with real input distributions
In one sentence
Quantization makes numbers smaller (e.g., FP32 → INT8). Pruning removes parts of the network (e.g., channels/neurons/heads). Quantization is often safer; pruning is often more “engineering heavy.”
Overview: quantization vs pruning (and why they break)
People often think optimization is “make the model smaller → it’s faster.” In reality, speed depends on hardware kernels, runtime support, and how your model’s layers map to accelerated ops.
Quantization (usually the first win)
Quantization stores/uses lower-precision numbers. The most common jump is FP32 → INT8.
- Biggest benefit: smaller model size, often faster inference
- Typical risk: accuracy drop if activations/weights are sensitive
- Best for: CNNs, many transformer parts, edge devices
Pruning (bigger engineering, can be huge)
Pruning removes parameters. The key is structured pruning if you want real speedups.
- Unstructured pruning: zeros weights (often smaller on disk, not always faster)
- Structured pruning: removes channels/filters/heads (more likely faster)
- Typical risk: quality drop + needs fine-tuning
- Quantization is “smaller math.”
- Pruning is “less model.”
- Distillation is “teach a small model to behave like a big one.”
What to measure (the only metrics that matter)
| Metric | Why it matters | How to measure |
|---|---|---|
| Quality (task metric) | Optimization is pointless if the model fails users. | Use the same eval set + same metric (mAP, F1, BLEU, etc.). |
| Latency (p50/p95) | Users feel tail latency, not “average speed.” | Benchmark on-device; report p50 + p95. |
| Memory (peak) | Edge devices crash or throttle when memory spikes. | Track peak RAM/VRAM and runtime allocations. |
| Throughput | Batching and streaming workloads need sustained FPS. | Measure steady-state after warmup. |
| Correctness on edge cases | Quantization can “move” decision boundaries. | Run a small “hard set” + adversarial-ish samples. |
Core concepts (clear definitions, no fluff)
1) What is quantization?
Quantization represents weights/activations using fewer bits. That reduces model size and can unlock faster integer kernels on mobile/edge hardware.
Common quantization types
| Type | What changes | Pros | Cons |
|---|---|---|---|
| FP16 | FP32 → FP16 | Often tiny quality loss, easy | Speedup depends on hardware |
| INT8 (PTQ) | Post-training INT8 | Fast to try, big size win | Can drop accuracy if calibration is weak |
| INT8 (QAT) | Train with fake-quant | Best accuracy for INT8 | Requires training pipeline |
| Per-channel | Different scales per channel | Usually higher quality | Not supported everywhere |
| Dynamic | Quantize weights (and maybe activations at runtime) | Simple for some models | Less control; not always fastest |
Post-training INT8 works best when your calibration data matches production inputs. If calibration data is “cleaner” or narrower than reality, activations can saturate and quality can collapse on real users.
2) What is pruning?
Pruning removes parameters or structure. There are two very different worlds here:
Unstructured pruning (often not faster)
Sets many weights to zero. Great for research, sometimes smaller on disk—often not faster unless you have sparse kernels.
- May compress well
- Speedup requires runtime sparse support
- Can be fragile if you prune too aggressively
Structured pruning (more likely faster)
Removes entire channels/filters/heads so the model becomes physically smaller (fewer ops).
- Often real latency wins
- Needs fine-tuning
- Better alignment with hardware kernels
3) Why optimizations fail in production
Most failures fall into the same buckets. If you avoid these, you’ll look like a wizard.
Failure mode: “Accuracy is fine… until real traffic”
- Calibration dataset doesn’t match reality
- Edge cases weren’t tested
- Different preprocessing at runtime
Failure mode: “Smaller but not faster”
- Runtime doesn’t use accelerated INT8 kernels
- Operator fell back to FP32/FP16
- Unstructured pruning created sparse weights without sparse kernels
Rule of thumb
Quantize for size + speed (when the runtime supports it). Prune for speed (when you remove structure). Distill for accuracy at small sizes (when you can afford training).
Step-by-step: a practical optimization workflow
This is a “do this, then this” path you can reuse across projects. It’s designed to prevent wasted weeks.
Step 0 — Build a baseline (non-negotiable)
You need one table of truth before touching anything. Keep it in your repo.
| Model | Format | Quality | Latency p50 | Latency p95 | Peak memory | Notes |
|---|---|---|---|---|---|---|
| baseline | FP32 | (your metric) | (ms) | (ms) | (MB) | Target device + runtime version |
- Warm up first (discard first N runs).
- Measure p50 + p95 (not just average).
- Use the same preprocessing pipeline for all variants.
- Measure on the target runtime (TFLite / ONNX Runtime / TensorRT), not only in PyTorch.
Step 1 — Try post-training INT8 quantization (PTQ)
PTQ is the fastest win because it doesn’t require retraining. The core requirement is representative calibration data.
Calibration checklist
- At least a few hundred real samples (more for diverse domains)
- Matches real preprocessing + ranges
- Includes “hard” inputs (dark images, noisy audio, long text, etc.)
- Same shapes as production (dynamic shapes can be tricky)
Sanity tests after PTQ
- Compare outputs on a small gold set (before/after)
- Run edge-case suite (your top failure cases)
- Check for saturation / clipped activations if possible
- Re-benchmark on device (don’t assume)
Step 2 — If quality drops: use quantization-aware training (QAT)
QAT simulates quantization during training so the model learns to be robust to lower precision. This is often the fix when PTQ “almost works” but loses too much quality.
When QAT is worth it
- PTQ drops quality beyond your threshold
- Your model has sensitive layers (attention, depthwise conv, small activations)
- You can retrain/fine-tune (even briefly)
- You need consistent behavior across devices
Step 3 — Prune for real speed (structured pruning)
If you need latency wins beyond quantization, pruning can help—especially structured pruning that reduces FLOPs.
Structured pruning targets (practical)
- CNNs: prune channels/filters in earlier layers carefully
- Transformers: prune attention heads or MLP width (then fine-tune)
- Anything: prune “expensive blocks” you can measure as hotspots
Rule: if pruning doesn’t reduce real compute (or your runtime can’t exploit it), you’ll get smaller files but not faster inference.
If you prune and accuracy collapses, it’s often because you removed capacity from layers that encode essential features. Start small (5–20%), then fine-tune, then repeat.
Step 4 — When you must go much smaller: distillation
If you need a model that’s dramatically smaller but still “feels” like the big one, distillation is often more reliable than extreme pruning.
When distillation shines
- Large accuracy drop with PTQ/pruning
- You want a compact model architecture (mobile-first)
- You can train/fine-tune with a teacher model
Easy distillation mindset
The student learns not only labels, but also the teacher’s “soft” outputs—capturing useful dark knowledge about alternatives.
Step 5 — Ship safely (monitor, validate, rollback)
Deployment checklist
- Version your model + runtime + preprocessing together
- Keep a fallback (previous model) for quick rollback
- Monitor quality proxies (drift signals, error spikes)
- Track device-specific issues (some accelerators behave differently)
Common mistakes (and the fixes that save you days)
Mistake 1 — Measuring only model size
A smaller file doesn’t guarantee faster inference.
- Fix: measure latency p50/p95 on the target runtime + device.
- Fix: confirm INT8 kernels are actually used (no hidden fallback).
Mistake 2 — Weak calibration data
PTQ can look okay on a tidy test set and break on real traffic.
- Fix: calibrate on representative, messy, real samples.
- Fix: include your “hard set” in the checks.
Mistake 3 — Pruning in a way hardware can’t exploit
Unstructured pruning often creates sparse weights without speed.
- Fix: use structured pruning for real latency wins.
- Fix: profile hotspots and prune the expensive parts first.
Mistake 4 — Changing preprocessing accidentally
Different normalization/tokenization can dominate results more than quantization does.
- Fix: lock preprocessing (same code path) across variants.
- Fix: keep a small gold set and compare outputs after export.
If PTQ breaks quality: try better calibration → then QAT. If speed doesn’t improve: verify kernel support → then consider structured pruning or a smaller architecture.
FAQ (what people search for)
Which is better: quantization or pruning?
Most teams should try quantization first because it’s faster to attempt and often has a great size/speed payoff. Use structured pruning when you need extra latency gains and you can fine-tune after pruning.
PTQ vs QAT: when do I need quantization-aware training?
Use PTQ if you want quick results and can tolerate a small quality drop. Use QAT when PTQ drops quality too much or you need more consistent behavior across devices.
Why is my INT8 model not faster?
This usually happens when your runtime falls back to higher precision ops, or when your target hardware doesn’t accelerate the model’s operators in INT8. Always benchmark on the real device and verify accelerated kernels are enabled.
How much can I prune without ruining accuracy?
There’s no universal number. A safe starting point is 5–20% structured pruning, followed by fine-tuning and evaluation. Repeat in small steps and stop when quality drops beyond your threshold.
What should I measure to avoid being fooled?
Measure task quality (your real metric), latency p50/p95 on-device, peak memory, and performance on a small hard set of edge cases.
Cheatsheet: the “do this / avoid that” checklist
Fast wins
- Try FP16 first if supported (easy, low risk)
- Then try INT8 PTQ with good calibration data
- Benchmark on-device (p50/p95), not only in Python
- Keep preprocessing identical across exports
When things go wrong
- Accuracy dropped? Improve calibration → then try QAT
- Not faster? Check kernel support / operator fallback
- Need more speed? Use structured pruning + fine-tune
- Need much smaller? Consider distillation
One rule you can tattoo on your brain
If you didn’t measure on the target device, you didn’t measure.
Wrap-up
Quantization and pruning are two of the most effective ways to ship models on real devices. The winning approach is boring (in a good way): measure a baseline, apply one optimization, re-measure, and only then decide what to do next.
- Pick one model and record the baseline table (quality + p50/p95 + memory).
- Try INT8 PTQ with a representative calibration set.
- If PTQ is too lossy, do a short QAT fine-tune and re-test.
Quiz
Quick self-check. Aim for understanding you can use when you actually ship a model.