Models are typically trained in 32-bit floating point (FP32), because training benefits from the numerical precision and dynamic range that gives. Most edge hardware — MCUs, low-power SoCs, many embedded NPUs — either can't run FP32 efficiently or can't run it at all within a realistic power and memory budget. Quantization is the process of converting a trained model to a lower-precision numeric format so it actually fits and runs efficiently on that hardware, and the choice of target format is a real engineering decision, not a checkbox.
What quantization actually changes
Quantization maps a model's weights (and often its activations) from a high-precision format to a lower-precision one — commonly FP32 down to FP16, INT8, or in more aggressive cases even lower bit-widths. Fewer bits per value means less memory to store the model, less memory bandwidth to move values during inference, and — critically for edge hardware — access to integer arithmetic units that are often faster and far more power-efficient than floating-point units on constrained silicon. The cost is precision: representing values with fewer bits necessarily loses some numerical detail, which can translate into accuracy loss if not managed carefully.
Common target formats and their tradeoffs
| Format | Size vs. FP32 | Typical accuracy impact | Hardware fit |
|---|---|---|---|
| FP16 | 50% | Usually minimal | Broad support, good middle ground |
| INT8 | 25% | Small to moderate, calibration-dependent | Strong support on modern edge NPUs/DSPs |
| INT4 / lower | 12.5% or less | Can be significant without careful technique | Narrower hardware/tooling support |
FP16 is often close to a free lunch — a meaningful size and speed improvement with accuracy impact that's frequently negligible, because it retains floating-point semantics with just reduced range and precision. INT8 requires more care: converting to integer arithmetic means choosing how the model's floating-point value ranges map onto the available integer range (a process called calibration), and a poorly calibrated conversion can produce a real accuracy drop, particularly for models with layers sensitive to precision loss.
Post-training quantization vs. quantization-aware training
There are two general approaches, and which one is worth the effort depends on how much accuracy loss the use case can tolerate. Post-training quantization converts an already-trained FP32 model directly, using a representative dataset to calibrate the integer ranges — it's fast and requires no retraining, and for many models the accuracy loss is small enough to accept. Quantization-aware training instead simulates the effects of quantization during the training process itself, letting the model adapt its weights to be more robust to the precision reduction — it produces better accuracy at the lower precision, at the cost of needing to retrain (or fine-tune) the model, which requires the original training pipeline and data.
Where accuracy loss matters most
- Classification tasks with fine-grained distinctions between similar classes tend to be more sensitive to quantization noise than tasks with well-separated categories
- Models already operating near a decision boundary — small numerical shifts can flip borderline predictions
- Certain layer types (some normalization and attention mechanisms in particular) are more sensitive to reduced precision than standard convolutional or fully-connected layers, and sometimes benefit from being kept at higher precision even in an otherwise-quantized model (mixed-precision quantization)
Validate on the target hardware, not just in simulation
Quantization frameworks can simulate lower-precision behavior on a development machine, but the actual accuracy and performance on target hardware can differ due to how a specific chip's inference runtime implements quantized operations. A model that quantizes cleanly in simulation should still be validated end-to-end on the actual target device before considering the conversion final — this is where framework-specific quirks (TensorFlow Lite Micro, ONNX Runtime, or a vendor SDK) most often surface.
How we approach this
We treat the precision target as a decision made against the specific hardware and accuracy requirements of the product, validated on real target hardware rather than simulation alone. See our AI & Intelligence work for how this fits into deploying inference on constrained embedded hardware.