The Conversion Gap: From ONNX Export to an Optimized Runtime Format

Training and production are different environments. Exporting a model to ONNX is a useful boundary, but a production runtime still needs to understand the graph, shapes, operators, and precision choices. Conversion friction is easier to handle when it is planned for instead of discovered at release time.

In aceu-edge-infer, I convert a lightweight classifier from ONNX for use with OpenVINO. One custom activation exposed an operator gap and required a conservative CPU fallback. The demo’s ACE-U output is illustrative, not an evaluation of real people; the model-delivery practices here are the focus.

Check the graph before you convert

Inspect the ONNX graph, its opset, input names, and shapes. Look for custom operators and operations that may not be supported by the runtime version or target device. If the model has dynamic dimensions, decide which shapes production must support and test those explicitly.

OpenVINO can convert supported ONNX models to IR, and it can also read ONNX directly at runtime. Conversion is useful when you want a prepared artifact and predictable model-load behavior; first confirm the format is supported and that conversion solves a real deployment need.

Use the current conversion API

For a supported ONNX model, the OpenVINO Python API can convert and save an IR artifact:

import openvino as ov
from pathlib import Path

model = ov.convert_model("models/classifier.onnx")
output = Path("build/ir_fp32/classifier.xml")
output.parent.mkdir(parents=True, exist_ok=True)
ov.save_model(model, output)

The saved IR consists of an XML graph and a weights file. Alternatively, the ovc command-line tool can convert supported input models. Keep the exact command or script, OpenVINO version, input shapes, and conversion logs with the build so another engineer can reproduce it.

Read conversion errors as signals

• Unsupported or partially supported operators: check the runtime’s supported operations and determine whether a supported equivalent preserves behavior.
• Shape inference failures: inspect dynamic dimensions and provide explicit input shapes only when they match the model’s intended use.
• Precision-specific differences: validate the FP32 conversion first; quantization adds a separate source of numerical change.
• Custom layers: decide early whether to replace, isolate, or route around the operation.

Choose a fallback deliberately

If an operator cannot use the preferred execution path, there are several options:

• Replace it with a supported equivalent, then measure output parity and confirm the change fits the model’s accuracy budget.
• Keep a documented CPU implementation for the unsupported portion when the runtime and architecture support that boundary.
• Split a larger graph into a supported model and a small, explicit post-processing step in application code.

Do not label a fallback as “optimized” without measurement. Surface the selected runtime path in logs or response metadata, and make clear whether the service can continue safely when the preferred path is unavailable.

Validate outputs before optimizing precision

Create a golden evaluation set and compare the source ONNX model with the converted FP32 model first. Define acceptance criteria that fit the task—for example, maximum per-class delta or mean absolute error. If outputs differ, compare intermediate tensors where possible to locate the first divergence.

Only after conversion parity is understood should you evaluate INT8. This keeps operator-conversion differences separate from quantization differences and makes failures much easier to diagnose.

Make conversion part of the build

Script input shapes, conversion flags, substitutions, and output paths. Version IR artifacts with the model and code revision that produced them; record file hashes and conversion logs rather than overwriting a vague latest artifact. Run conversion in CI when the model, conversion script, or runtime version changes.

Trade-offs and failure modes

• Operator replacement vs. behavioral fidelity: an equivalent-looking function can change output; verify parity against the task’s acceptance criteria.
• CPU fallback vs. uniform execution: a fallback can preserve service behavior but adds a path to test and monitor.
• Fast manual fix vs. reproducibility: one-off flags may unblock a conversion, while a scripted fix makes future builds repeatable.

A model that converts and runs can still produce different outputs. A fallback that is not disclosed can make performance results misleading. A runtime update can change what converts successfully, so keep conversion inside the change-review process.

Try it yourself

1. Inventory operators, opset, shapes, and custom layers in your ONNX file.
2. Convert to IR with a scripted, versioned command and retain its logs.
3. Compare outputs on a golden set before you introduce quantization.
4. Document and test every fallback path, including how the service reports it.

Closing thought

The conversion gap is manageable when operator compatibility, model parity, and fallbacks are treated as product requirements. Script the path, version what it creates, and validate the outputs before optimizing further.