Cache Correctness: Verifying Warm Starts

Problem

Caching exists to save time, so it is tempting to treat it as invisible. But a cached artifact is a compiled program saved to disk. Reusing it assumes that the conditions that produced it still hold: the same device, a compatible software version, and the same model and input shapes.

Two things make this risky in practice. OpenVINO's documentation says devices without import and export support run normally but do not get the speed-up, so a missing cache does not announce itself. And an artifact left over from an earlier version or a different model can be picked up by accident if folders are not separated.

The friction points are confirming that a cache was created, confirming that a warm start produces the same outputs as a cold start, and refusing to serve when it does not.

Options and trade-offs

• Check that something was cached: after the first run, the cache folder should contain new files. An empty folder after a successful compile suggests the device does not support caching or the folder is not writable.
• Compare outputs: run one fixed input through a freshly compiled model and through a model loaded from the cache, then compare the results with a tolerance you choose in advance. For deterministic models on the same device, exact equality is a reasonable expectation; choose a tolerance only if your pipeline needs one.
• Separate cache folders by software version, model version, device, and input shape. This costs nothing and removes a whole class of confusing failures.
• Fail closed: if the output comparison fails or loading from the cache raises an error, delete that cache entry, recompile, and log the event. Do not serve from a cache you could not verify.
• Keep the check cheap: a single fixed input at startup usually catches a gross failure, and it can run alongside your warm-up request.
• Be cautious with outside reports: issue trackers contain reports about specific edge cases. Treat them as leads for your own tests, and do not build policy around a problem you cannot reproduce on your version.
• Recommendation: add a verification step to startup. After the model loads, run one fixed input and compare it with a stored reference. If the comparison fails, discard the cache entry, recompile, and record that the warm start was rejected. Log whether each start was cold or cached, how long compilation took, and which device and versions were involved.
• Decision lens: speed against silent error risk. Example setup: a service checks one reference input after every restart, treats a mismatch as a failed start, and writes each version's cache to its own directory so an upgrade never reads the previous version's files.

Minimal working example

Prereqs

• Python 3 with openvino and numpy installed.
• A model with a floating-point input; adjust the dtype in the script if yours differs.
• A writable cache folder.

verify_cache.py (run with fresh first, then cached twice)

import sys
import numpy as np
import openvino as ov

model_path, mode = sys.argv[1], sys.argv[2]  # mode: fresh or cached
core = ov.Core()
if mode == "cached":
    core.set_property({"CACHE_DIR": "./model_cache"})
compiled = core.compile_model(model_path, "CPU")

rng = np.random.default_rng(0)
shape = [d.get_length() for d in compiled.input(0).partial_shape]
x = rng.random(shape, dtype=np.float32)  # adjust dtype to match your model
result = compiled(x)[compiled.output(0)]

if mode == "fresh":
    np.save("reference.npy", result)
    print("saved reference")
else:
    reference = np.load("reference.npy")
    print("max abs difference:", float(np.max(np.abs(reference - result))))

Adjust file names, model names, and paths to your own setup.

Test: Run verify_cache.py with fresh to store a reference, then with cached to populate the cache, then with cached again to compare an imported model against the reference. Next, change the model file (or its version folder) and confirm that your folder naming prevents the old cache from being used. Repeat after upgrading OpenVINO. A difference larger than the tolerance you chose is a failed start.

Record: device, operating system, OpenVINO version, and model; files present in the cache folder after the first run; maximum absolute output difference between cold and cached; the tolerance you chose and why; behavior after the model changes; behavior after an upgrade; time added by the startup check.

Checklist

• Confirm the cache folder is populated after the first run.
• Compare a fixed output between a cold run and a cached run, and record your tolerance.
• Separate cache folders by version, model, device, and shape.
• Fail closed and recompile on a mismatch.
• Log cold or cached on every start, and re-run the check after every upgrade.
• Avoid: assuming a cache exists because no error appeared.
• Avoid: reusing one cache folder across versions or models.

Where this doesn't apply

Services that never use compiled-model caching, and systems that already compare outputs against a reference on every start.

Sources

• OpenVINO Model Caching Overview — https://docs.openvino.ai/nightly/openvino-workflow/running-inference/optimize-inference/optimizing-latency/model-caching-overview.html (accessed 2026-10-18).
• OpenVINO Model Server parameters — https://docs.openvino.ai/2026/model-server/ovms_docs_parameters.html (accessed 2026-10-18).
• OpenVINO NPU device documentation — https://docs.openvino.ai/2025/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html (accessed 2026-10-18).

---