KV Cache Governance: Bound LLM Memory in OpenVINO Model Server

Problem

A language-model server keeps attention keys and values for every token of every active sequence. That memory grows with context length and concurrency. Unlike model weights, it is not a fixed cost, so a deployment that looks fine in a short test can run short of memory under a different traffic mix.

OpenVINO Model Server lets you set the cache memory with --cache_size, and its documentation examples pass explicit values. Choosing that number deliberately is the heart of governing the cache.

The friction points are picking a limit, observing real usage against it, documenting the number and its basis, and deciding in advance what the service does when a request does not fit.

Options and trade-offs

• Explicit limit: pass --cache_size in gigabytes. The documentation's examples use values such as 20 on a large CPU system and 2 in a GPU example, which shows that the right value depends on the model, precision, and hardware.
• Prefix reuse: with prefix caching enabled, repeated prompt prefixes avoid recomputation. A larger cache can hold more reusable history, so cache size and the benefit of prefix caching are linked.
• Concurrency and context length are the two drivers of cache growth. Limit the maximum prompt size and the number of simultaneous requests at your gateway so the cache you chose is a realistic fit.
• Observation: use operating-system and container tools to watch the server process's memory over time. You do not need special server features to see a trend and compare it with the limit you set.
• Overflow policy: decide whether the service should reject oversize requests with a clear error, shorten context, or queue. Silent failure or a crash is the outcome to avoid.
• Model and precision changes: a different model or precision changes memory per token, so a limit chosen for one is not valid for another.
• Recommendation: start with a conservative value in the style of the documentation's examples, run a realistic mix of context lengths and concurrency, and watch the memory trend. Increase the cache until your largest realistic request fits with headroom, and stop there. Write the limit in the runbook with the model, precision, hardware, and the observed peak that justified it.
• Decision lens: predictable memory versus refusing some requests. Example setup: a single-host service with an explicit --cache_size, a gateway that rejects prompts over a set length with a clear message, and an alert when process memory reaches a chosen fraction of the host.

Minimal working example

Prereqs

• A Model Server container named ovms serving a language model, started with an explicit --cache_size.
• Docker and Python 3 with requests.

memory_watch.sh (sample the container's memory while a load test runs)

while true; do
  docker stats --no-stream --format "{{.Name}} {{.MemUsage}}" ovms
  sleep 5
done

load_profile.py

import requests

URL = "http://localhost:8000/v3/chat/completions"
MODEL = "OpenVINO/Qwen3-8B-int4-ov"  # use the model name you deployed

def send(words):
    prompt = "word " * words
    r = requests.post(URL, json={"model": MODEL, "messages": [{"role": "user", "content": prompt}], "max_tokens": 16})
    return r.status_code, r.elapsed.total_seconds()

for words in [200, 1000, 4000, 8000]:
    print(words, send(words))

Increase the prompt sizes until the server starts refusing or slowing down, and record what it returns.

Adjust file names, model names, and paths to your own setup.

Test: Start memory_watch.sh in one terminal and run load_profile.py in another with increasing prompt sizes. Note the memory trend, the largest prompt that completes, and exactly what the server returns for a prompt that is too large. Change --cache_size, restart, and repeat.

Record: model, precision, Model Server version, and host memory; the --cache_size values tested; the memory trend; the largest prompt that completed for each cache size; the server's response to an oversize prompt; latency as prompt size grows; your chosen limit and the evidence for it.

Checklist

• Set --cache_size explicitly.
• Observe memory over a realistic load and find the largest prompt that fits.
• Test the overflow path on purpose.
• Limit prompt size and concurrency at the gateway.
• Document the limit with hardware, model, and precision, and repeat after any change.
• Avoid: sizing from a single short test.
• Avoid: having no defined behavior for oversize requests.

Where this doesn't apply

Short prompts with low concurrency on a host with ample memory, and services behind a managed platform that sets cache limits for you.

Sources

• Agentic AI with OpenVINO Model Server (example commands) — https://docs.openvino.ai/2025/model-server/ovms_demos_continuous_batching_agent.html (accessed 2026-10-18).
• Model Server continuous batching demo — https://github.com/openvinotoolkit/model_server/blob/main/demos/continuous_batching/README.md (accessed 2026-10-18).
• Model Server performance tuning guide — https://github.com/openvinotoolkit/model_server/blob/main/docs/performance_tuning.md (accessed 2026-10-18).
• OpenVINO Model Server parameters — https://docs.openvino.ai/2026/model-server/ovms_docs_parameters.html (accessed 2026-10-18).

---