Open source models Inference on Azure: Managed Foundry, Self-Hosted on AKS, or a Hybrid of Both

Problem Statement

One natural next question when reviewing open source models is not just "which model is best?" but "where should this model actually run?" Azure gives you several ways to serve the same model, and the differences are operational, not just about model quality:

• Fully managed — deploy a catalog model to Microsoft AI Foundry (Azure OpenAI) and get a callable endpoint in under a minute, with nothing to operate.
• Self-hosted on AKS — run an open-weight model on your own GPUs with vLLM, owning every layer from the GPU driver to the rollout strategy.
• Declarative on AKS — the same self-hosted outcome, but collapsed into a single KAITO (AKS AI Toolchain Operator) Workspace CRD.

From an application or landing-zone standpoint, this is a platform decision. Teams choose self-hosting for cost control, data sovereignty, use of their own GPUs, deeper customization, and co-location with existing AKS workloads. They choose managed for speed, elasticity, and zero operations. Most real deployments may end up wanting both; one being frontier reasoning on a Foundry managed endpoint, the other being bulk and sovereign traffic on their own cluster.

This post is grounded in an actual iterative comparison test I ran across all three, so the trade-offs are based on the test results. Beyond those three, it also surveys the full spectrum of GPU inference options on Azure, including Container Apps Serverless GPU and Foundry Managed Compute. That survey is an observation based on reviewing the available options as of September 2026, and the specifics may shift as these services evolve.

Solution

The key insight that makes the comparison fair is the /v1 boundary. Azure OpenAI exposes an OpenAI-compatible /openai/v1/ endpoint, which is the same API surface that a self-hosted vLLM server exposes at /v1. So the same Python program — with the plain openai SDK, no AzureOpenAI client, no api-version — talks to all three back ends. Switching between them is only a change of OPENAI_BASE_URL (plus the model name and auth).

# chat.py — identical for AKS-manual, AKS-KAITO, and Foundry
import os
from openai import OpenAI

base = os.environ["OPENAI_BASE_URL"]
key = os.environ.get("OPENAI_API_KEY", "")

if key in ("", "entra", "aad"):
    # Microsoft Entra ID token auth (used when key auth is disabled by policy)
    from azure.identity import DefaultAzureCredential, get_bearer_token_provider
    token = get_bearer_token_provider(DefaultAzureCredential(), "https://ai.azure.com/.default")
    client = OpenAI(base_url=base, api_key=token)
else:
    client = OpenAI()  # key auth: reads OPENAI_BASE_URL + OPENAI_API_KEY

resp = client.chat.completions.create(
    model=os.environ["OPENAI_MODEL"],
    messages=[{"role": "user", "content": "What is Kubernetes, in one sentence?"}],
    max_tokens=64, temperature=0,
)
print(f"[endpoint] {base}")
print(resp.choices[0].message.content)

To move the same program between tests you change three environment variables to point to point to right endpoint/URL:

The layered mental model: what "hosting a model" actually means

"Serving a model" is not one thing — it's a stack of layers, and each layer can independently live on AKS or stay managed.

The table reads bottom-up like a stack diagram — layer 1 (GPU compute) is the foundation at the bottom, and everything else is built on top. This post focuses on the lower four layers, so here's what each one is and what your options are:

The lower layers (1–4) are the GPU-and-serving plumbing. The one hard constraint: layer 2 for closed-weight models (like the GPT family) is managed-only — you cannot pull those weights onto your own GPUs. Open-weight models (Phi, Llama, DeepSeek, …) are yours to self-host.

The four layers, three ways

The test serves microsoft/Phi-3.5-mini-instruct — an open-weight model (~7.6 GB in fp16) that fits a single NVIDIA T4 and is also in the Foundry catalog, which makes the "same weights, different operator" point concrete. Here is how the same four layers play out across the three tests:

On Test A you author everything: the node pool, the NVIDIA device plugin, and a vLLM Deployment where there are multiple flags/args:

containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    args:
      - "--model=microsoft/Phi-3.5-mini-instruct"   # [Layer 2] artifact (pulled from HF)
      - "--served-model-name=phi-3.5-mini"          # [Layer 3] engine config
      - "--dtype=half"                               #   fp16: required on T4 (no bf16)
      - "--max-model-len=4096"                       #   caps KV-cache size per request
      - "--gpu-memory-utilization=0.9"
      - "--trust-remote-code"
    resources:
      limits: { nvidia.com/gpu: 1 }                  # [Layer 1] claim the whole GPU

On Test B, those same four layers collapse into a single CRD: KAITO does the node provisioning, device plugin, vLLM tuning, and Deployment/Service for you:

apiVersion: kaito.sh/v1beta1
kind: Workspace
metadata:
  name: workspace-phi
resource:
  instanceType: "Standard_NC4as_T4_v3"   # [Layer 1] KAITO auto-provisions this GPU node
  labelSelector:
    matchLabels: { apps: phi }
inference:
  preset:
    name: phi-3.5-mini-instruct          # [Layers 2–4] weights + tuned vLLM + Deployment/Service

On Test C, the whole stack is two az commands (or Foundry portal) — capacity + catalog, then a deployment. You get a callable endpoint.

az cognitiveservices account create -n $AOAI -g $RG -l $LOC \
  --kind AIServices --sku S0 --custom-domain $AOAI --yes

az cognitiveservices account deployment create -g $RG -n $AOAI \
  --deployment-name gpt-4.1-mini \
  --model-name gpt-4.1-mini --model-version 2025-04-14 --model-format OpenAI \
  --sku-name GlobalStandard --sku-capacity 10

What I measured — the scorecard

Running all three end-to-end makes the trade tangible. The headline numbers from the test:

The hybrid: the pattern most teams actually ship

Because all three speak the same OpenAI Chat Completions shape, you don't have to pick one. The pattern that shows up in real deployments is a simple router: open-weight, sovereign, or bulk traffic goes to the AKS endpoint; frontier reasoning goes to the Foundry endpoint — one application, one client class.

# hybrid_router.py — SAME OpenAI client class for both back ends; only base_url + model differ
aks = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

token = get_bearer_token_provider(DefaultAzureCredential(), "https://ai.azure.com/.default")
foundry = OpenAI(base_url=os.environ["AOAI_V1"], api_key=token)

def route(prompt: str, frontier: bool):
    """Bulk/sovereign work stays on AKS; hard reasoning goes to Foundry."""
    client, model = (foundry, "gpt-4.1-mini") if frontier else (aks, os.environ["AKS_MODEL"])
    r = client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": prompt}], max_tokens=128)
    return client.base_url, r.choices[0].message.content

The takeaway made concrete: both back ends use the exact same OpenAI client class — only the base_url (and model name) differs. The AI stack is commoditized at the /v1 boundary, so moving a workload between AKS and Foundry is an endpoint change, not a rewrite. What actually differs is who operates layers 1–4 — you (control + toil) or Microsoft (speed + less control) — and the hybrid puts each workload where that trade pays off.

Zooming out: the full spectrum of GPU inference on Azure

The three tests above made one point: switching between a frontier model served via Foundry and an OSS model self-hosted on AKS is the same code; only the endpoint changes. But those three are just three stops on a wider spectrum that Azure actually supports. Foundry documents two deployment options for its own catalog — Serverless API and Managed Compute (preview), and Azure Container Apps documents serverless GPU as a "middle layer between serverless APIs and managed compute." Adding the two AKS-based self-hosting options completes a five-stop line. The one axis that the docs make unambiguous is how much infrastructure you own:

Nothing (Serverless API) → a container (Container Apps) → a model deployment on dedicated GPUs (Managed Compute) → a cluster with an AI add-on (KAITO) → the whole GPU platform (AKS)

The three back ends I benchmarked map onto this spectrum as the two ends and the Kubernetes middle — Test C = Foundry Serverless API, Test B = KAITO + AKS, Test A = AKS + GPUs. The two I didn't benchmark — Container Apps Serverless GPU and Foundry Managed Compute — fill in the managed middle.

The same spectrum, compared across the aspects that actually drive the decision. Every cell below is grounded in the linked Azure docs — where a claim is a judgement call rather than a documented fact, I say so:

The one thing that is a clean, documented gradient is how much infrastructure you own: nothing (Serverless API) → a container (Container Apps) → a model deployment on dedicated GPUs (Managed Compute) → a cluster with an AI add-on (KAITO) → the whole GPU platform (AKS).

One nuance at the managed end of this spectrum: Fireworks on Microsoft Foundry is a first-party inference provider that surfaces the newest open-weight models (and the only Foundry path to custom weights) through the same Foundry endpoint, Entra ID identity, quota, and Azure billing, offered as serverless per-token, provisioned throughput, or bring-your-own-weights; the trade-off is that inference runs as a disclosed pass-through on Fireworks' own GPUs rather than inside your Azure tenant.

Where Foundry's decision tree stops — and where OSS picks up

Foundry's own deployment decision tree narrows to two options: Serverless API — the preferred path, covering Azure-sold and partner/community models — and Managed compute (preview) for open-source, partner, and custom weights. That's worth stating plainly: for an open-weight model, the only branch inside Foundry is managed compute. There is no "run it on your own cluster" leaf, because self-hosting is deliberately outside Foundry's scope.

The tree below keeps Foundry's two managed leaves and adds the three self-hosting branches, Container Apps Serverless GPU, KAITO + AKS, and AKS + GPUs, so the whole Azure decision is on one page.

Read left-to-right, the amount of infrastructure you own grows with every branch: Container Apps Serverless GPU lets you ship your own container with scale-to-zero, but only on T4/A100 and one GPU per replica; Foundry Managed Compute trades that container/runtime freedom for dedicated capacity and larger accelerators (A100/H100/MI300X) that Foundry sizes for you; KAITO + AKS gives you the cluster with an opinionated vLLM-based add-on; AKS + GPUs hands you every knob. As with anything, the more you can control and customize, the more you own it!

How to choose

Reach for self-hosted on AKS when at least one of these earns its keep:

• Cost control — steady, high-volume traffic on GPUs you already own can beat per-token pricing.
• Your own GPUs / data sovereignty — weights and prompts never leave your cluster.
• Deep customization — a specific GPU SKU, quantized or fine-tuned weights, a PVC weight cache.
• Co-location — the model sits next to the AKS apps, data, and network that call it. You are bypassing several proxies.

Reach for managed Foundry when speed and zero ops win: no cluster, no GPU quota, no drivers, an answer in under a minute, built-in elasticity (Global / regional / data-zone capacity), and consumption telemetry out of the box. And note that Foundry adds enterprise abstractions for free, such as Entra ID data-plane auth, rate-limit budgets, and request IDs, that you'd otherwise build yourself with a gateway (layer 7).

KAITO sits in the middle: you keep your cluster, your GPUs, your network and identity.

Conclusion

Foundry is a great place to run inference on Azure, so start here but there are other options when you need more control and customization. However, there is a trade-off, which is like using Azure Functions vs building a Functions-like service on a Kubernetes cluster. Azure supports a spectrum of options, and the right answer depends on which layers of the stack you want to own. Running layers 1–4 yourself on AKS buys deep control and customization at the cost of real operational effort. Foundry collapses those same layers into a pick-a-model-get-an-endpoint experience, trading knobs for speed. KAITO is the pragmatic middle. And because every option speaks the same OpenAI /v1 API, it's just a matter of routing between base URLs.

From a platform-foundation standpoint, the job isn't to standardize on one option; it's to offer all possible paths and let each workload land where the control-vs-convenience trade pays off.

References

Azure OpenAI in Microsoft AI Foundry
Deployment overview for Microsoft Foundry Models
Foundry Managed Compute (preview) — deploy open-source models
Bringing Open Models to Fireworks on Microsoft Foundry
Azure Container Apps — serverless GPU
KAITO — Kubernetes AI Toolchain Operator on AKS
vLLM — high-throughput LLM serving engine
Azure API Management AI Gateway capabilities