AUTO vs Explicit Devices vs HETERO in OpenVINO

Problem

OpenVINO lets you name a device directly, ask for AUTO selection, or split a model across devices with HETERO. They look like three spellings of one decision, but they behave differently when a device is missing, when you restart, and when you cache compiled models.

The wrong choice rarely fails loudly. It shows up as a first request served by a different device than you expected, a cache that never hits, or a fallback that quietly runs on the CPU and misses a latency target.

The friction points are knowing what AUTO does at startup, knowing where work actually ran, and testing restarts instead of only first starts.

Options and trade-offs

• Explicit device: compile for CPU, GPU, or NPU by name. The behavior is easiest to reason about, and the compiled-model cache belongs to that concrete device. Routing and failover are your application's job.
• AUTO: the documentation says AUTO detects available devices and selects the best one automatically. By default it starts inference on the CPU unless there is a cached model for the best-suited device, because the CPU can start with no additional delay. While the CPU runs inference, AUTO keeps loading the model onto the better device and hands the work over when ready.
• AUTO startup behavior is configurable: the enable_startup_fallback property controls whether the CPU is used as the initial helper device (default true), and enable_runtime_fallback controls whether a failed inference request is retried on another device (default true). Excluding the CPU from the priority list disables the initial CPU acceleration.
• AUTO reports where work actually ran: the execution_devices property lists the devices currently executing inference. Logging it is the cheapest way to avoid surprises.
• A published OpenVINO blog describes AUTO trying the next best device if the model does not compile on the first choice, with the CPU as the final fallback because it supports all operations and precisions.
• HETERO splits one model across several devices. It is meant for cases where a model does not fit on one device or part of it needs different hardware. It adds configuration and testing work, so treat it as a special-purpose tool and test the exact model and device pair you intend to ship.
• Caching interaction: compiling explicitly for each device you use makes cache behavior easy to reason about. With AUTO, early requests may be served by the CPU while the better device loads, so first-request latency depends on whether a cached model exists for that better device.
• Recommendation: use an explicit device when you need predictable startup and cache behavior, and put routing and failover in your application where you can log and test them. Use AUTO when portability across machines matters more than deterministic timing, and always log execution_devices. Treat HETERO as an advanced option adopted only when a specific model needs it.
• Decision lens: predictability versus portability. Example setup: an application that runs on laptops with varied hardware uses AUTO and logs its execution device; a service on a fixed server fleet compiles explicitly for the device it has.

Minimal working example

Prereqs

• Python 3 with the openvino package installed.
• A machine with at least two device types if possible, such as CPU and an integrated GPU or NPU.
• A model you already run.

compare_devices.py

import sys, time
import openvino as ov

model_path = sys.argv[1]
core = ov.Core()
core.set_property({"CACHE_DIR": "./model_cache"})
print("Available devices:", core.available_devices)

for target in ["CPU", "AUTO"]:
    start = time.perf_counter()
    compiled = core.compile_model(model_path, target)
    elapsed = time.perf_counter() - start
    print(target, f"compile {elapsed:.3f}s", "executing on:", compiled.get_property("EXECUTION_DEVICES"))

Add "GPU" or "NPU" to the list if your machine has them.

Adjust file names, model names, and paths to your own setup.

Test: Run compare_devices.py twice in separate processes, once with an empty cache folder and once with a populated one. Note the device reported by EXECUTION_DEVICES right after compilation and again after a few inferences, because AUTO can switch devices. If you can do it safely, disable a device and rerun to see how each strategy reacts.

Record: devices present and OpenVINO version; device reported by AUTO right after compile and after warm-up; time to first inference for each strategy with an empty and a populated cache, as median and 95th percentile; behavior when the preferred device is unavailable; your decision and the reason.

Checklist

• Log the executing device in every run.
• Compile explicitly per device when cache behavior must be predictable.
• Review the startup and runtime fallback settings if you use AUTO.
• Test restarts, not only first starts, and pin the versions you tested.
• Document the failover behavior you observed.
• Avoid: assuming AUTO serves the first request on the best device.
• Avoid: adopting HETERO without testing the exact model.

Where this doesn't apply

Single-device deployments where the choice is trivial, systems whose hardware is fixed and known (explicit selection is simpler), and models too large for any single device, where HETERO may be unavoidable.

Sources

• Automatic Device Selection documentation — https://docs.openvino.ai/2025/openvino-workflow/running-inference/inference-devices-and-modes/auto-device-selection.html (accessed 2026-10-18).
• Automatic Device Selection and Configuration (OpenVINO blog) — https://blog.openvino.ai/blog-posts/automatic-device-selection-and-configuration (accessed 2026-10-18).
• OpenVINO Model Caching Overview — https://docs.openvino.ai/nightly/openvino-workflow/running-inference/optimize-inference/optimizing-latency/model-caching-overview.html (accessed 2026-10-18).

---