Device Settings Through Wrappers: ONNX Runtime and OpenVINO

Problem

Many teams use OpenVINO through another library, and ONNX Runtime's OpenVINO execution provider is one example. Each wrapper has its own configuration surface, and it is not identical to OpenVINO's own property names and behavior.

That difference causes a quiet class of problems. A setting that works when you call OpenVINO directly may be ignored or reshaped by a wrapper, and the default device may not be the one you expect.

The friction points are knowing the default device, knowing which OpenVINO version the wrapper bundles, and knowing which options your installed version really accepts.

Options and trade-offs

• Default device: the PyPI page for the OpenVINO execution provider package states that, by default, the Intel CPU is used for inference, and that you can change this to an integrated GPU, a discrete GPU, or an integrated NPU through the provider configuration's device type argument.
• Bundled runtime: the same page says the Linux wheels at that time came with pre-built libraries of OpenVINO 2025.4.1, so the OpenVINO inside the wrapper may differ from one you installed separately. Check the version your package carries.
• Provider options: ONNX Runtime GenAI's documentation for the OpenVINO provider shows device_type, cache_dir, and a load_config block with per-device properties, for example INFERENCE_PRECISION_HINT and PERFORMANCE_HINT under a CPU key. The page also advises choosing the device by goal: CPU for latency, GPU for throughput, and NPU for efficiency.
• Per-device properties: placing properties under the device they apply to, as in the load_config example, avoids applying a setting meant for one device to all of them.
• Cache directory: the same documentation shows cache_dir as a provider option to reduce model loading time on later runs. As with direct OpenVINO use, confirm with a restart test that it works.
• Direct OpenVINO use gives you every property OpenVINO exposes and clearer errors, at the cost of more integration code. If a wrapper does not expose something you need, consider using OpenVINO directly for that component.
• Recommendation: treat provider options as deployment configuration. Write them in one place, record the library versions beside them, and re-run a probe whenever you upgrade. Copy option names from the page for the version you installed, not from memory, and prefer explicit device names and per-device properties in production.
• Decision lens: convenience versus control. Example setup: a service that loads models through ONNX Runtime with one configuration file listing the device type, cache directory, and per-device properties, plus a note of the versions on which that file was tested.

Minimal working example

Prereqs

• Python 3 in a clean virtual environment with the onnxruntime-openvino package installed. Do not install it next to the plain onnxruntime package in the same environment.
• An ONNX model you already use.
• A writable cache folder.

provider_probe.py

import onnxruntime as ort

print("ONNX Runtime version:", ort.__version__)
print("Available providers:", ort.get_available_providers())

providers = [("OpenVINOExecutionProvider", {"device_type": "CPU", "cache_dir": "./ov_cache"})]
session = ort.InferenceSession("model.onnx", providers=providers)
print("Session providers:", session.get_providers())

If OpenVINOExecutionProvider is missing from the available providers, your installed package does not include it, and ONNX Runtime will fall back to another provider.

Adjust file names, model names, and paths to your own setup.

Test: Run provider_probe.py and check that OpenVINOExecutionProvider appears in the available providers and in the session's providers. Run it a second time to see whether load time drops with cache_dir. If the machine has a GPU or NPU, change device_type and record what changes.

Record: ONNX Runtime and provider package versions; the OpenVINO version bundled in the package; the providers available and the providers the session reports for each device type; load time with and without cache_dir, first versus second run; behavior when a requested device is absent; the configuration you chose for production.

Checklist

• Record every library version.
• Confirm the provider is actually available.
• Set the device type explicitly and place properties under the device they apply to.
• Test cache_dir with a restart.
• Keep the verified configuration in your repository and re-run the probe after every upgrade.
• Avoid: assuming the wrapper's OpenVINO version matches your standalone install.
• Avoid: copying option names from another version's documentation.

Where this doesn't apply

Applications that call OpenVINO directly, and systems using a different OpenVINO integration than ONNX Runtime.

Sources

• onnxruntime-openvino on PyPI — https://pypi.org/project/onnxruntime-openvino/ (accessed 2026-10-18).
• ONNX Runtime GenAI: OpenVINO execution provider — https://microsoft-onnxruntime-genai-88.mintlify.app/acceleration/openvino (accessed 2026-10-18).
• OpenVINO Model Caching Overview — https://docs.openvino.ai/nightly/openvino-workflow/running-inference/optimize-inference/optimizing-latency/model-caching-overview.html (accessed 2026-10-18).

---