Vision-Language Models: Memory and Latency Testing

Problem

A vision-language model turns images or video frames into tokens and processes them together with text. A handful of frames can contribute far more tokens than an ordinary chat message. That raises both the time to read the prompt and the memory held for it.

As a result, a service that handles video behaves differently from a text chat service even when the same server software runs both. A cache limit chosen from chat traffic may not hold, and neither may a latency target set for text.

The friction points are setting a frame budget, re-sizing memory with video in the mix, and not assuming that features which help text, such as prefix caching, help video.

Options and trade-offs

• Input types: check the documentation for your OpenVINO GenAI and Model Server versions for which image and video inputs the vision-language pipeline accepts and how frames are sampled. This area changes quickly, so use the documentation for the version you install.
• Frame budget: limit the number of frames and their resolution per request. Because each frame contributes tokens, this is usually the most direct control over both latency and memory.
• Cache limit: the Model Server --cache_size option sets KV-cache memory. A limit that works for text-only traffic should be re-examined with video requests in the mix.
• Prompt reuse: prefix caching reuses work for repeated prompt prefixes. Whether it helps when the changing part of a conversation is video is something to test directly and report either way.
• Device choice: use --target_device to select a device explicitly, and record the device and driver with every result.
• Recommendation: treat video as its own workload with its own limits. Choose a frame budget first, then size memory from measurements that include video requests. Measure the same text prompt with no image, one image, several images, and a short clip; the differences are the cost of each input type.
• Decision lens: input cost versus feature value. Example setup: an internal tool that accepts clips of a few seconds, with a fixed sampling rate, a fixed memory limit, and a clear error message for longer clips.

Minimal working example

Prereqs

• Model Server serving a vision-language model, set up as its documentation for your version describes.
• Docker and Python 3 with requests.
• A sample image and a short video clip.

modality_ladder.py

import time, requests

URL = "http://localhost:8000/v3/chat/completions"
MODEL = "your-vlm-model-name"  # the name you deployed

def ask(content):
    start = time.perf_counter()
    r = requests.post(URL, json={"model": MODEL, "messages": [{"role": "user", "content": content}], "max_tokens": 16})
    return r.status_code, round(time.perf_counter() - start, 3)

text = {"type": "text", "text": "Describe what you see."}
print("text only:", ask([text]))
# Add image and frame entries in the format your server version documents,
# then compare request time as you add more of them.

Follow your Model Server version's documentation for the exact request format for images and video frames.

Adjust file names, model names, and paths to your own setup.

Test: Send the same text prompt with no image, one image, several images, and a short clip, at least twenty times each. Watch the server process's memory. Then raise the number of frames until the server refuses or slows sharply, and note where that happens.

Record: model, precision, device, driver, and software versions; frame sampling and resolution; request time per input type as median and 95th percentile; peak memory per input type; the frame count at which the server refuses or slows; whether prefix caching helps with a stable text history and changing video; your chosen frame budget.

Checklist

• Use the documentation for your exact versions for input formats.
• Set a frame budget.
• Re-size memory using video requests.
• Measure each input type separately and record device and driver with every result.
• Test prefix reuse with changing video, and return a clear error for inputs over budget.
• Avoid: sizing memory from text-only traffic.
• Avoid: assuming prefix caching helps when the changing part is video.

Where this doesn't apply

Text-only services, models or versions without video support, and single-image tools where the cost difference is small.

Sources

• OpenVINO Model Server parameters — https://docs.openvino.ai/2026/model-server/ovms_docs_parameters.html (accessed 2026-10-18).
• Agentic AI with OpenVINO Model Server (example commands) — https://docs.openvino.ai/2025/model-server/ovms_demos_continuous_batching_agent.html (accessed 2026-10-18).
• Model Server continuous batching demo — https://github.com/openvinotoolkit/model_server/blob/main/demos/continuous_batching/README.md (accessed 2026-10-18).
• Model Server performance tuning guide — https://github.com/openvinotoolkit/model_server/blob/main/docs/performance_tuning.md (accessed 2026-10-18).

---