A Playbook for GenAI Latency on Intel Hardware

Problem

Teams adopting OpenVINO for generative workloads meet the same set of switches: compiled-model caching, prefix caching, device selection, KV-cache sizing, and device-specific properties. Each can help. Switched on in different combinations by different people, they produce behavior that nobody can explain.

A playbook turns those switches into decisions with owners and evidence. It starts from budgets, derives defaults from measurements, and records for each default what was measured and on which hardware.

The friction points are deciding the budgets, keeping defaults tied to evidence, and reviewing them when software or hardware changes.

Options and trade-offs

• Budgets: define time to first token at a stated percentile, throughput, and a memory limit for each service. Without a target, tuning has no end.
• Cold-start policy: use compiled-model caching where devices support it, and verify it with the capability check and the output comparison from earlier in the series.
• Device policy: choose explicit devices for predictable behavior, use AUTO where portability matters, and log the executing device. AUTO may serve early requests on the CPU unless a cached model exists for the better device.
• Memory policy: set the KV-cache size explicitly, keep prefix caching on for conversational traffic, limit prompt size and concurrency at the gateway, and define what happens to an oversize request.
• NPU gate: require a compile smoke test, a recorded driver version, a list of the device's supported properties, and a CPU fallback before enabling the NPU for users.
• Evidence rule: every default points to a measurement with the hardware and versions it came from. A default without evidence is a guess and should be labeled as one.
• Recommendation: write the playbook as configuration plus a short decision table in your repository, not as a document nobody reads. Each setting gets a reason, an owner, and a link to the measurement that justified it. Review it whenever OpenVINO or Model Server versions, models, hardware, or traffic change.
• Decision flow in four questions: Is cold start a risk? Enable caching and verify it. Are contexts long or repeated? Keep prefix caching on and set an explicit KV limit. Is an NPU planned? Pass the readiness checklist first. Do you need portability across machines? Use AUTO for routing and log where work ran.
• Decision lens: consistency and defensibility. Example setup: a platform team with one configuration repository, a benchmark job that reruns the key tests on each version upgrade, and a one-page decision flow that new services follow.

Minimal working example

Prereqs

• Results from the earlier articles for the services you run.
• A repository or folder where configuration and results files can live together.

defaults.yaml (outline of a policy file; fill every value from your own measurements)

service: chat-assistant
budget:
  ttft_p95_seconds: <your target>
  memory_limit_gb: <your limit>
caching:
  compiled_model_cache: true
  cache_path: /var/cache/ov/<ov-version>/<model-version>/<device>
  evidence: results/cold-vs-cached.md
device:
  strategy: explicit   # or auto
  log_execution_device: true
  evidence: results/device-strategy.md
kv_cache:
  cache_size_gb: <from observation>
  prefix_caching: true
  overflow_behavior: reject-with-message
  evidence: results/kv-cache.md

Every evidence entry should point to a results file you produced from the earlier articles.

Adjust file names, model names, and paths to your own setup.

Test: Pick one default in defaults.yaml and find the results file that justifies it. Upgrade OpenVINO or Model Server in a test environment and rerun the relevant measurement. If the numbers changed, update the default and its evidence. A default you cannot link to a measurement should be marked as a guess.

Record: the budget for each service; cold start before and after the caching policy; first-token percentile before and after the memory policy; the device strategy outcome on your hardware; the result of the rerun after the last upgrade; the evidence file for every default.

Checklist

• Define budgets for each service.
• Enable and verify compiled-model caching, with an output comparison for warm starts.
• Choose a device policy and log the executing device.
• Set an explicit KV-cache limit from observation.
• Require a readiness checklist for the NPU, attach evidence to every default, and review after upgrades.
• Avoid: copying another team's defaults without measuring.
• Avoid: tuning without budgets.

Where this doesn't apply

Small experiments on one machine where defaults are adequate, and teams that cannot run repeatable measurements.

Sources

• OpenVINO Model Caching Overview — https://docs.openvino.ai/nightly/openvino-workflow/running-inference/optimize-inference/optimizing-latency/model-caching-overview.html (accessed 2026-10-18).
• Automatic Device Selection documentation — https://docs.openvino.ai/2025/openvino-workflow/running-inference/inference-devices-and-modes/auto-device-selection.html (accessed 2026-10-18).
• Model Server performance tuning guide — https://github.com/openvinotoolkit/model_server/blob/main/docs/performance_tuning.md (accessed 2026-10-18).
• OpenVINO Model Server parameters — https://docs.openvino.ai/2026/model-server/ovms_docs_parameters.html (accessed 2026-10-18).
• OpenVINO NPU device documentation — https://docs.openvino.ai/2025/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html (accessed 2026-10-18).
• onnxruntime-openvino on PyPI — https://pypi.org/project/onnxruntime-openvino/ (accessed 2026-10-18).