Packaging Inference for Production: A Checklist for Shipping an Optimized-Model Service

A model can be accurate and still be hard to operate if conversion is manual, artifacts are opaque, precision changes silently, or fallbacks have never been exercised. Production readiness is the process around the model as much as the model itself.

aceu-edge-infer is a modest inference demo, but I treat its packaging path as if another engineer will need to rebuild, inspect, and recover it. Its outputs remain illustrative, not evaluative of real people. This checklist is about making that kind of service repeatable.

Make model conversion reproducible

Keep conversion in a script or build step rather than a shell history. Record the source model hash, tool and runtime versions, input shapes, conversion options, and warnings. Use separate, reviewable configuration for FP32 and INT8 artifacts, and retain the conversion logs with the release build.

from pathlib import Path
import openvino as ov

source = Path("models/source.onnx")
output = Path("build/models/model_fp32.xml")
output.parent.mkdir(parents=True, exist_ok=True)

model = ov.convert_model(source)
ov.save_model(model, output)

This example shows the conversion step only. Production automation should also validate model inputs, run parity checks, record the tool version, and fail when expected artifacts are missing.

Version and identify the artifacts

Give model artifacts versions tied to the code and conversion configuration that produced them. Store IR XML and weights together, record hashes, and keep them immutable after release. If calibration or quantization changes, publish a new model version rather than overwriting a file called latest.

A release manifest can record the information operators need without embedding the model itself:

{
  "model_version": "<release-version>",
  "source_format": "ONNX",
  "runtime": "OpenVINO",
  "precision_artifacts": ["FP32", "INT8"],
  "device_policy": "CPU",
  "source_sha256": "<source-model-hash>",
  "conversion_config_sha256": "<config-hash>"
}

Replace the placeholders during the build. These are identifiers to calculate and record—not measured values to guess.

Disclose the execution path

Include the precision and device used in logs or response metadata. Where it makes sense for the product, provide a controlled option to route sensitive work through FP32. That small bit of transparency can make rollout analysis and incident response much faster. Keep the API contract stable and avoid exposing internal details that callers do not need.

Test fallbacks before a release

Simulate an unavailable optimized runtime and verify the intended behavior. A valid fallback should either return correct outputs on a supported slower path or fail clearly with a non-ready/unavailable state. It should not manufacture placeholder results or quietly claim the preferred precision.

If custom operators exist, test the fallback with representative inputs and confirm that the selected path is visible in logs. Document what operators or hardware are required for each execution mode.

Package the runtime deliberately

Use a multi-stage image where it keeps build tools out of the runtime. Run as a non-root user, keep the base image and dependencies maintained, copy only the application and model artifacts required at runtime, and provide a bounded health check.

# Illustrative Python service image; adapt paths and dependencies to your app.
FROM python:3.12-slim AS builder
WORKDIR /build
RUN python -m venv /venv
ENV PATH="/venv/bin:$PATH"
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

FROM python:3.12-slim AS runtime
RUN useradd --uid 10001 --create-home appuser
WORKDIR /app
COPY --from=builder /venv /venv
COPY --chown=10001:10001 app/ ./app/
COPY --chown=10001:10001 models/ ./models/
ENV PATH="/venv/bin:$PATH" \
    PYTHONDONTWRITEBYTECODE=1
USER 10001
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
  CMD python -c 'import urllib.request; urllib.request.urlopen("http://localhost:8000/health", timeout=2)' || exit 1
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Confirm that the chosen base image contains everything the service actually needs. The Docker health check should stay lightweight; configure a separate platform readiness check if model loading or a sample inference must gate incoming traffic.

Build, validate, and scan in CI

Use the same conversion scripts in local development and CI. Check that expected files exist, compare model outputs against a golden set, build the service image, and scan it before release. Make missing artifacts, failed parity checks, and scan failures visible release blockers. Keep exceptions owned and time-bounded.

Keep operational notes close to the code

Give the README short, findable sections for conversion, artifact versions, precision modes, fallback behavior, and hardware assumptions. Add an incident note describing what to inspect when latency or output parity changes, and point to the known-good artifact for rollback.

Trade-offs and failure modes

• Process rigor vs. prototype speed: scripts and manifests take effort once and save time on the next rebuild.
• More API metadata vs. smaller contract: publish only what helps callers; keep additional operational detail in logs.
• Strict readiness vs. faster first traffic: waiting for model load and checks reduces false-green deploys.

“Mystery IR” files, silent FP32 fallbacks, and changed base images without review all make a service harder to reproduce. Version the inputs and outputs, disclose the selected path, and exercise rollback before you need it.

Try it before your next release

1. Script FP32 and INT8 conversion and retain logs with immutable model artifacts.
2. Record source and configuration hashes in a release manifest.
3. Add actual precision/device metadata and a tested force-FP32 option if the product needs it.
4. Test missing-runtime and unsupported-operator behavior before launch.
5. Build, validate, and scan the same container image in CI that you intend to deploy.

Closing thought

Packaging inference for production is about repeatability, transparency, and safe recovery. When another engineer can recreate the model, identify the execution path, and trigger a tested fallback, the service is ready for more than a successful demo.