imshubh.

Product leader · IEEE Senior Member · CMU MSE board member

Connect↗
← Intel articles

Intel / LLM serving

Continuous Batching and KV Cache Sizing in OpenVINO Model Server

The Model Server options that control prompt reuse and KV-cache memory, example commands from its documentation, and a measurement plan for choosing values you can defend.

Shubham Anandani··4 min read

Problem

A generative model server keeps a key-value (KV) cache for every sequence it is working on. That cache grows with context length and with the number of sequences processed together. If both grow without limits, throughput looks good in a short test and latency becomes unpredictable under real traffic.

OpenVINO Model Server serves language models with continuous batching and paged attention. Its documentation shows the main task-specific options, including whether prompt tokens are cached across requests and how much memory the KV cache may use.

The friction points are choosing a cache size for your hardware, deciding whether prefix caching is worth it for your traffic, and measuring both honestly instead of trusting a short single-user test.

Options and trade-offs

  • --enable_prefix_caching: described in the documentation as caching prompt tokens (KV blocks) so repeated prefixes are not evaluated again. The performance tuning guide lists --enable_prefix_caching true as the default. It helps when clients resend the same conversation history and does little when every prompt is unique.
  • --cache_size: the memory for the KV cache, in gigabytes. The documentation's examples set it explicitly, for example 20 for an 8-billion-parameter 4-bit model on a 16-core CPU system and 2 for a GPU example. Treat those as illustrations of the option, not recommendations for your hardware.
  • --target_device: the continuous batching demo explains that, if you do not set it, the device is chosen from detected hardware with priority discrete GPU, then integrated GPU, then CPU. Set it explicitly when you need a particular device.
  • --task text_generation with --source_model and --model_repository_path is the documented way to start serving a language model from a model repository. The pull guide describes converting and quantizing a Hugging Face model, with options such as --weight-format int8.
  • Example from the documentation (CPU, Docker): docker run -d --name ovms --cpuset-cpus 0-15 --user $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models openvino/model_server:weekly --rest_port 8000 --model_repository_path /models --source_model OpenVINO/Qwen3-8B-int4-ov --enable_prefix_caching true --cache_size 20 --task text_generation.
  • Other batching limits exist in the server's task-specific options. List them with the help output for the version you run, and record any you change.
  • Recommendation: start from the documented command for your device, change one option at a time, and measure. Set --cache_size explicitly in production so memory use is a decision you made and wrote down. Keep prefix caching enabled for chat-style traffic and test it with a shared-prefix workload and a unique-prompt workload. Limit concurrency at the edge of your system as well.
  • Decision lens: throughput against tail latency and memory risk. Example setup: a single host serving a small model, with three configurations compared under the same request mix: a small cache, a larger cache, and a larger cache with a front-end concurrency limit.

Minimal working example

Prereqs

  • Docker, and a machine with enough memory for the model you choose. The documentation's example uses an 8-billion-parameter 4-bit model on a large CPU system; pick a smaller model if your hardware is smaller.
  • The Model Server image and a model from a repository, started with a command in the style shown above.
  • Python 3 with requests for the client.

client.py (request generator for the server's OpenAI-compatible endpoint)

import time, statistics, requests

URL = "http://localhost:8000/v3/chat/completions"
MODEL = "OpenVINO/Qwen3-8B-int4-ov"  # use the model name you deployed

def one_request(prompt):
    start = time.perf_counter()
    r = requests.post(URL, json={"model": MODEL, "messages": [{"role": "user", "content": prompt}], "max_tokens": 32, "stream": False})
    r.raise_for_status()
    return time.perf_counter() - start

shared_prefix = "Summarize the following notes. " * 200
latencies = [one_request(shared_prefix + f" Note {i}.") for i in range(30)]
print("median:", statistics.median(latencies))
print("p95:", sorted(latencies)[int(0.95 * len(latencies)) - 1])

This measures total request time with a small output, a rough proxy for time to first token. For a precise first-token time, use a streaming request and record when the first chunk arrives. Check the endpoint path and model name against the documentation for your version.

Adjust file names, model names, and paths to your own setup.

Test: Start the server with prefix caching on and an explicit --cache_size. Run client.py with a shared long prefix and again with unique prompts, and watch memory with docker stats. Then change one option, such as the cache size or prefix caching, and repeat. Compare the median and 95th percentile of request time between configurations.

Record: model, Model Server version, and host hardware; the full command line; the request mix (prompt lengths, shared-prefix share, concurrency); median and 95th percentile with prefix caching on and off; peak memory per configuration; your chosen --cache_size and the reason.

Checklist

  • Record the version and every option you set.
  • Set --cache_size explicitly.
  • Test shared-prefix and unique-prompt workloads separately.
  • Change one option per run and watch memory during the run.
  • Report percentiles, not averages, and write the chosen values and evidence in your runbook.
  • Avoid: trusting a short, single-user test.
  • Avoid: copying the documentation's example --cache_size to different hardware.

Where this doesn't apply

Single-user local tools where concurrency is one, workloads whose prompts never repeat a prefix (prefix caching adds little), and deployments where an upstream gateway already limits concurrency tightly.

Sources

---