API gateway enforcement for LLM APIs

Problem

When every team calls the model endpoint directly, governance is a spreadsheet: who has access, how much they use, and what they send. Putting a gateway in front of the model gives you one place to authenticate callers, enforce limits and log usage. Azure API Management (APIM) can authenticate to Azure OpenAI with its own managed identity, so no API keys are stored or shared, and it can validate caller tokens before anything reaches the model.

Options and trade-offs

• Managed identity to the backend: APIM holds the identity, callers hold nothing that works against the model directly. It is simple for service-to-service access, and every caller shares the same backend privileges.
• OAuth 2.0 validation at the gateway: callers present Entra tokens, and APIM checks audience and roles before forwarding. You get per-caller control and audit, with more setup.
• Per-caller limits: rate limits keyed on the caller's identity stop one team's runaway loop from using the shared quota.

My recommendation

Use both layers. Validate the caller's token at the gateway, then use the gateway's managed identity to call the backend. Add a rate limit keyed on the caller. Disable key-based access on the model resource once traffic flows through the gateway, so there is no way around it.

Minimal working example

Enable a managed identity on the APIM instance and grant it access to the model resource:

az role assignment create \
  --assignee <apim-principal-id> \
  --role "Cognitive Services OpenAI User" \
  --scope <azure-openai-resource-id>

Then add the inbound policy. It validates the caller, limits usage, and swaps the caller's token for the gateway's backend token:

<inbound>
  <base />

  <!-- 1. Validate the caller's Entra token -->
  <validate-jwt header-name="Authorization" failed-validation-httpcode="401"
                failed-validation-error-message="Unauthorized">
    <openid-config url="https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration" />
    <audiences>
      <audience><gateway-app-id></audience>
    </audiences>
    <required-claims>
      <claim name="roles" match="any">
        <value>LLM.Invoke</value>
      </claim>
    </required-claims>
  </validate-jwt>

  <!-- 2. Limit usage per caller -->
  <rate-limit-by-key calls="60" renewal-period="60"
      counter-key="@(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Subject)" />

  <!-- 3. Authenticate to the backend with the gateway's managed identity -->
  <authentication-managed-identity resource="https://cognitiveservices.azure.com"
      output-token-variable-name="managed-id-access-token" ignore-error="false" />
  <set-header name="Authorization" exists-action="override">
    <value>@("Bearer " + (string)context.Variables["managed-id-access-token"])</value>
  </set-header>
</inbound>

Test it: call the gateway with a valid token and confirm a normal response, call without a token and confirm a 401, and call in a loop to confirm the rate limit returns 429.

Checklist

• Use the model resource's custom subdomain endpoint, and confirm the role assignment scope.
• Validate the audience and a role claim at the gateway, not just that a token exists.
• Key the rate limit on the caller's identity, not on an IP address that may be shared.
• Disable local (key) authentication on the model resource after cutover.
• Log request metadata for audit, and keep prompt and completion content out of default logs.
• Check that your policy ordering does what you expect: validation first, then limits, then backend authentication.

Where this doesn't apply

Backends that do not support Entra authentication or gateway integration. There, the gateway still helps with limits and audit, but backend authentication needs another design.

Sources

• Authenticate and authorize to LLM APIs with Azure API Management — https://learn.microsoft.com/en-us/azure/api-management/api-management-authenticate-authorize-ai-apis
• authentication-managed-identity policy — https://learn.microsoft.com/en-us/azure/api-management/authentication-managed-identity-policy