Serving ONNX models on AKS GPU without surprises

Problem

ONNX Runtime provides optimized inference across multiple accelerators. In production, AKS GPU node pools must be prepared (drivers/plugins), readiness and health checks must reflect model load state, and autoscaling must consider GPU allocation and throughput constraints.

Options and trade-offs

• NVIDIA Triton with ONNX backend: Multi-framework server with health/readiness endpoints; simplifies model repos and supports gRPC/HTTP. Operationally richer; larger image footprint; excellent for multi-model serving.
• ORT server containers: Leaner; you must select images/runtime versions and wire health probes yourself; good for single model or simple stacks.
• Node pool strategy: Dedicated GPU node pools with NVIDIA device plugin; enforce nvidia.com/gpu limits; separate system and user pools for stable scheduling.

Minimal working example

Prereqs

• AKS cluster; GPU node pool with NVIDIA device plugin; kubectl; Triton image; PVC for model repository.
• Model: ResNet-50 in ONNX.

Deployment (Triton + ONNX)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-server
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: triton-server
  template:
    metadata:
      labels:
        app: triton-server
    spec:
      containers:
      - name: triton
        image: nvcr.io/nvidia/tritonserver:24.08-py3
        args: ["tritonserver","--model-repository=/models/repository","--strict-model-config=false"]
        ports:
        - containerPort: 8000
          name: http
        - containerPort: 8001
          name: grpc
        readinessProbe:
          httpGet:
            path: /v2/health/ready
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
          failureThreshold: 6
        resources:
          limits:
            nvidia.com/gpu: 1
        volumeMounts:
        - name: model-vol
          mountPath: /models
      volumes:
      - name: model-vol
        persistentVolumeClaim:
          claimName: triton-model-repository-pvc

Test: kubectl get pods -n inference; wait for readiness; send inference to port 8000; confirm GPU assignment with kubectl describe. Version and runtime assumptions

• Triton 24.08; AKS GPU plugin installed; ONNX Runtime on server side via Triton; CUDA matching driver for your GPU.

Checklist

• Validate CUDA/cuDNN compatibility with the ORT/Triton image; mismatches silently fallback to CPU in some cases.
• Pin readinessProbe to /v2/health/ready; initialDelaySeconds accounts for model load.
• Use separate GPU node pools; confirm nvidia.com/gpu schedulable capacity.
• Cap replicas by GPU capacity; ensure PVC throughput supports model load.

Where this doesn't apply

CPU-only inference; use ORT CPU builds and different autoscaling signals.

Sources

• AI Inference on AKS with Triton + ONNX (GA blog) — https://blog.aks.azure.com/2026/04/07/ai-inference-on-aks-arc-part-4 (accessed 2026-10-13).
• ONNX Runtime install and CUDA requirements (GA) — https://onnxruntime.ai/docs/install/ (accessed 2026-10-13).
• Use GPUs on AKS Hybrid/Edge node pools (GA) — https://github.com/MicrosoftDocs/azure-stack-docs/blob/main/AKS-Arc/deploy-gpu-node-pool.md (accessed 2026-10-13).

---