Benchmarking Honestly: Building a Repeatable, Trustworthy Inference Benchmark

Benchmarks are easy to fool. Run a handful of requests on a quiet laptop, choose the number that looks good, and almost any model can appear fast. Production has warm-up, load, input variation, and hardware contention. A useful benchmark makes those conditions visible.

In aceu-edge-infer, I keep a small harness that compares FP32 and INT8 inference and records latency and output deltas. The sample inputs are synthetic, so its results are method scaffolding—not evidence of production performance. The measurement discipline is what I want the demo to teach.

Decide what question the benchmark answers

Are you measuring cold start, steady-state single-request latency, or throughput under concurrent work? Those are different measurements. State the workload, model artifact, runtime and device, input shape, batch size, thread/concurrency settings, and whether preprocessing and post-processing are included.

Warm up—and report cold start separately

The first call may include model compilation, memory allocation, cache fill, or runtime initialization. If the question is steady-state latency, run a planned warm-up and keep those samples out of the measured set. If cold start matters to users, report it as its own metric instead of mixing it into steady-state numbers.

Report percentiles as well as averages

Users feel slow requests, not just the mean. Report p50, p95, and p99 latency for each model variant, along with the sample count and run duration. Tail percentiles need enough samples to be meaningful; do not imply precision the benchmark cannot support.

Compare the same inputs and check outputs

Run the same representative inputs through FP32 and INT8. Compare both the score deltas and task-level decisions. For classification, inspect per-class changes and examples near decision boundaries; a low average absolute difference can still hide an important decision flip.

Here is a small summarizer for arrays collected by a benchmark harness. It assumes both latency arrays contain milliseconds and each output array corresponds to the same input order, with one row per request and one column per model output:

import numpy as np


def latency_summary(samples_ms):
    values = np.asarray(samples_ms, dtype=np.float64)
    return {
        "p50_ms": float(np.percentile(values, 50)),
        "p95_ms": float(np.percentile(values, 95)),
        "p99_ms": float(np.percentile(values, 99)),
    }


def parity_summary(outputs_fp32, outputs_int8):
    fp32 = np.asarray(outputs_fp32, dtype=np.float64)
    int8 = np.asarray(outputs_int8, dtype=np.float64)
    per_request_delta = np.mean(np.abs(fp32 - int8), axis=-1)
    return {
        "mean_absolute_delta": float(np.mean(per_request_delta)),
        "p95_absolute_delta": float(np.percentile(per_request_delta, 95)),
    }

These summaries do not replace task-specific evaluation. For example, also record whether top-class predictions or other decision thresholds changed.

Control and record variance

Use the same machine and runtime configuration for each comparison. Note background load, CPU power state, affinity or isolation settings if you use them, and any other workload sharing the device. Repeat runs and compare run-to-run variation. If the host cannot be isolated, state that plainly.

Use a representative validation set for accuracy and parity. Synthetic inputs are valuable for exercising a harness or pipeline, but they cannot establish how a model behaves on the real input distribution.

Keep calibration and benchmark series distinct

Calibration data influences the quantization parameters. Record its source and important characteristics, and keep the evaluation set separate. If calibration changes, treat the quantized model as a new artifact and rerun the full comparison rather than blending results from different model versions.

Make precision and execution path observable

If some requests fall back from INT8 to FP32, label them. Do not combine them into one “INT8” result. Emit structured output—JSON works well—with model hashes, precision, device, warm-up configuration, sample count, latency distribution, output deltas, and run metadata.

Trade-offs and failure modes

• Simple harness vs. realistic load: a single-request loop is easy to understand, but it does not characterize concurrency or resource contention.
• Synthetic inputs vs. representative data: synthetic inputs make a repeatable demo; production claims need appropriate evaluation inputs.
• Tail reporting vs. small samples: percentiles reveal latency spread, but only when there are enough measurements to support them.
• Mixed precision vs. clean comparison: unreported fallbacks make speed and parity comparisons untrustworthy.

Try it yourself

1. Write down whether you are testing cold start, latency, throughput, or a combination.
2. Warm up deliberately and record the sample count and test conditions.
3. Report FP32 and INT8 percentiles alongside output-delta and decision-change summaries.
4. Repeat on an idle host and under representative load; keep those results separate.

Closing thought

Benchmarking honestly is a habit. Warm up, report the distribution, compare outputs, and describe the machine and workload. Then “it’s faster” becomes a claim another engineer can reproduce and assess.