Compiled-Model Caching in OpenVINO

Problem

Every OpenVINO application reads a model and compiles it for a specific device when it starts. For a small model on CPU the cost is easy to ignore. For larger models, or for GPU and NPU targets, compilation can dominate the time before the first request completes. If a rollout restarts your service, the first user request can pay that price.

OpenVINO's model caching stores the compiled result so that a later start can import it instead of recompiling. You enable it by pointing OpenVINO at a cache folder. The catch is that not every device supports importing and exporting compiled models. OpenVINO's documentation says such devices work normally but do not get the compilation speed-up, so a cache you configured can quietly do nothing.

That quiet failure is the reason to measure rather than assume. The friction points are confirming device support, understanding how caching interacts with AUTO and the NPU, and measuring a restart in a way that is not fooled by process state.

Options and trade-offs

• Application-level caching: set the CACHE_DIR property on the OpenVINO core before compiling. The first compile has extra work because it also stores the result, so it can be slightly slower. Later compiles can import the stored model.
• Capability check: ask the device whether it supports import and export (the EXPORT_IMPORT capability). If it does not, caching will not shorten that device's compile step, whatever you configure.
• AUTO device selection: the AUTO documentation says AUTO starts inference on the CPU by default unless there is a cached model for the best-suited device, because the CPU can begin right away. While the CPU works, AUTO keeps loading the model onto the better device and switches over when ready. A warm cache can therefore change which device serves your first request.
• NPU: the NPU documentation and plugin README describe a driver-level compiled-model cache enabled by default in current drivers, and OpenVINO's own caching is enabled when you set cache_dir. Read how the two interact in the documentation for your driver and version.
• Model Server: OpenVINO Model Server provides a --cache_dir option and a documentation page on its model cache. The goal is the same as in application code: faster loading on later starts.
• Prefix caching is a different mechanism that works per request instead of per process. Model Server's performance tuning guide lists --enable_prefix_caching true as the default and describes it as reusing work for repeated prompt prefixes, which is common when chat clients resend conversation history.
• General rule for any compiled artifact: it is tied to the device, the software version, and the model and its input shapes. Do not share one cache folder across versions or machine types.
• Recommendation: turn caching on wherever the device supports it, keep the cache folder on storage that survives restarts, and treat the capability check as a gate. If a device does not report EXPORT_IMPORT, plan for the full compile time there. Measure restarts with fresh processes, because a second compile call inside one process does not tell you what a restart costs.
• Decision lens: restart speed against operational risk. A persistent cache is one more thing to manage, and a stale artifact left after an upgrade is a confusing failure. Put the OpenVINO version, model version, and device name in the folder path so you can delete a whole generation at once. Example setup: a container with a volume mounted at the cache path, a startup script that logs compile time, and a health check that confirms the folder contains files after the first run.

Minimal working example

Prereqs

• Python 3 with the openvino package installed (pip install openvino).
• An OpenVINO IR model (model.xml and model.bin) or an ONNX file you already use.
• A writable folder for the cache, on the same filesystem the production service will use.

capability_check.py

import openvino as ov
import openvino.properties.device as device

core = ov.Core()
for name in core.available_devices:
    supported = "EXPORT_IMPORT" in core.get_property(name, device.capabilities)
    print(name, "supports caching" if supported else "no import/export")

cold_vs_cached.py (run it in a new process each time)

import sys, time
import openvino as ov

model_path, device_name = sys.argv[1], sys.argv[2]
core = ov.Core()
core.set_property({"CACHE_DIR": "./model_cache"})

start = time.perf_counter()
compiled = core.compile_model(model_path, device_name)
print(f"compile_model on {device_name}: {time.perf_counter() - start:.3f}s")

Delete the model_cache folder to get a cold run. Run the script again without deleting it to get a cached run.

Adjust file names, model names, and paths to your own setup.

Test: Run capability_check.py and note which devices report support. Run cold_vs_cached.py ten times with the cache folder deleted before each run (cold), then ten times without deleting it (cached), each in a new process. If the cached median is clearly lower, caching is working on that device. If the two medians are close, the device probably does not support caching or the folder is not being used, which is also a useful answer.

Record: device, OpenVINO version, operating system, and model; capability check result; cold compile time as median and 95th percentile; cached compile time as median and 95th percentile; cache folder size; time to first token cold versus cached if you serve a language model; whether outputs match between the two.

Checklist

• Run the capability check for every device you plan to use.
• Set CACHE_DIR (or --cache_dir in Model Server) and confirm files appear after the first run.
• Put version, model, and device in the cache path and store it on persistent storage.
• Measure in separate processes and report percentiles.
• Compare outputs between a cold run and a cached run.
• Avoid: assuming caching works without checking the device's capabilities.
• Avoid: timing two compiles in one process and calling it a restart test.

Where this doesn't apply

Workloads where the model is compiled once for a long-running process and restarts are rare, cases where first-token latency is dominated by something other than compilation (network time or input parsing), and devices that do not support import and export.

Sources

• OpenVINO Model Caching Overview — https://docs.openvino.ai/nightly/openvino-workflow/running-inference/optimize-inference/optimizing-latency/model-caching-overview.html (accessed 2026-10-18).
• Automatic Device Selection documentation — https://docs.openvino.ai/2025/openvino-workflow/running-inference/inference-devices-and-modes/auto-device-selection.html (accessed 2026-10-18).
• Automatic Device Selection and Configuration (OpenVINO blog) — https://blog.openvino.ai/blog-posts/automatic-device-selection-and-configuration (accessed 2026-10-18).
• OpenVINO NPU device documentation — https://docs.openvino.ai/2025/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html (accessed 2026-10-18).
• Intel NPU plugin README — https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/intel_npu/README.md (accessed 2026-10-18).
• Model Server performance tuning guide — https://github.com/openvinotoolkit/model_server/blob/main/docs/performance_tuning.md (accessed 2026-10-18).
• OpenVINO Model Server parameters — https://docs.openvino.ai/2026/model-server/ovms_docs_parameters.html (accessed 2026-10-18).

---