Faster Isn’t Free: The Real Trade-offs of Quantizing a Model from FP32 to INT8

I’ve shipped CPU-only inference, and I’ve also declined to ship a “faster” model when the trade-off did not hold up. Quantizing a model from FP32 to INT8 can help meet latency or memory goals. It can also change outputs in ways that matter to the people relying on them.

In my public demo, aceu-edge-infer, I use a small ONNX classifier converted to OpenVINO IR and maintain FP32 and INT8 variants. The service reports which precision it used. It returns four ACE-U dimensions—Authenticity, Credibility, Empathy, and Uniqueness—but these are illustrative outputs from a demo, not validated judgments of real people. The engineering questions are the useful part.

Why quantization can help—and why it changes behavior

Quantization represents selected model weights and activations using lower-precision values. With fewer bits to move and process, an inference path may use less memory and run faster on compatible hardware. The actual speedup depends on the model, operators, runtime, and target device; INT8 is not a universal fast switch.

Lower precision also introduces rounding. Small changes in early layers can flow through later layers, and a score close to a decision threshold may move across it. Some operators quantize cleanly; others may need different quantization settings, to remain in higher precision, or to run through a fallback path.

Look for where the errors gather

Aggregate accuracy can hide concentrated regressions. Compare FP32 and INT8 outputs by class and by meaningful input slice. Inspect examples near decision boundaries, rare patterns, and values near the extremes of the calibration distribution. A useful audit asks not only “How much did the average score move?” but also “Where did the model change its decision?”

Calibrate on data that resembles use

Post-training quantization uses calibration samples to estimate ranges for activations. If the samples do not resemble the inputs the service will see, those ranges can be a poor fit. Include representative cases and the edge cases you care about. When real data cannot be used, diverse synthetic examples can help exercise the pipeline, but they do not replace evaluation on an appropriate held-out set.

A minimal OpenVINO/NNCF sketch looks like this. calibration_samples must yield preprocessed examples in the model’s expected input format:

import nncf
import openvino as ov

core = ov.Core()
model = core.read_model("model_fp32.xml")
calibration_dataset = nncf.Dataset(calibration_samples)

model_int8 = nncf.quantize(model, calibration_dataset)
ov.save_model(model_int8, "model_int8.xml")

This produces a quantized model; it does not establish that the model is accurate enough for a particular use. Keep calibration data and evaluation data separate, and record the evaluation results alongside the model artifact.

Measure more than a mean

• Latency distribution: report p50, p95, and p99 for both precisions. A mean alone can hide a worse tail.
• Output deltas: compare score changes and decision changes across the evaluation set.
• Warm-up: exclude initialization and compilation work from steady-state measurements, but report cold-start behavior separately if it matters in production.
• Representative conditions: use the same device, request shape, batch size, concurrency, and preprocessing path.

The demo contains a small harness that runs synthetic requests against both variants. Its results are scaffolding until measured on an appropriate workload; synthetic measurements should never be presented as production performance.

Know when not to ship INT8

If the latency improvement is marginal while the output changes affect a high-importance class, the trade-off may be negative. For sensitive applications, stability can matter more than a modest speed gain. Compare other options too: batching, request scheduling, a different CPU pool, or optimizing a costly preprocessing step may achieve the goal with less model-level change.

Make precision visible and roll out in steps

Record the precision and execution device in service logs or response metadata so a change is diagnosable. Keep the public response contract minimal and avoid exposing sensitive implementation details.

Before a full rollout, run FP32 and INT8 in shadow for a representative sample and alert on meaningful divergence. If one slice regresses, route it back to FP32 or disable the INT8 path while you investigate. Maintaining both variants adds operational work, but gives the team a controlled recovery option.

Try it yourself

1. Build FP32 and INT8 variants and disclose the precision used for each request.
2. Measure warm and cold paths separately, then report latency percentiles and output-delta distributions.
3. Evaluate by class and edge-case slice, not only by aggregate accuracy.
4. Run a shadow comparison and define in advance what divergence will trigger a rollback.

Closing thought

Quantization is powerful, but faster is not free. Percentiles, output deltas, and slice-level evaluation tell you whether INT8 belongs in the production path—or whether FP32 is the more responsible choice.