Platform Engineering - Part Two

Problem Statement

Imagine it's 2 AM and your on-call pager fires: "Production API latency exceeded SLA — customer-facing orders are failing." You're the SRE on duty. There are no AI agents, no MCP servers, no LLM-assisted tooling. It's you, a terminal, a bunch of dashboards, and a runbook that was last updated six months ago.

Here's what the next few hours actually look like:

The manual troubleshooting gauntlet

1. Triage the alert — Open PagerDuty or OpsGenie, read the alert, try to figure out which cluster, namespace, and service are affected. If the alert is vague ("high latency on ingress"), you're already guessing.

2. Authenticate and get context — Run az login, set the right subscription, pull kubeconfig for the correct cluster, switch kubectl context. If you manage multiple clusters across environments, this alone can eat 5–10 minutes.

3. Check pod and node health — kubectl get pods -A | grep -v Running, kubectl describe pod <name>, kubectl get events --sort-by=.lastTimestamp. Scroll through walls of YAML and event output looking for the signal in the noise.

4. Pull logs — kubectl logs <pod> --previous, maybe across multiple containers and replicas. If the pod already restarted, the logs may be gone. If you're lucky, they're in Log Analytics — so now you're writing KQL queries in the Azure portal, waiting for results, adjusting time ranges, and filtering namespaces.

5. Check metrics — Switch to Azure Monitor or Grafana. Look at CPU, memory, network, and disk for the node pool. Open a separate tab for Managed Prometheus and write PromQL queries. Compare current values to "normal" — which requires you to remember what normal looks like.

6. Correlate across layers — The pod is CrashLoopBackOff, but is it the app, the node, or something upstream? Check the node's VM status in the portal. Check NSG and UDR rules if there's a networking suspicion. Query Azure Resource Health. Open yet another tab for App Configuration or Key Vault to see if a secret expired.

7. Search for known issues — Open internal Confluence or wiki, search for similar symptoms. Check the AKS release notes for known regressions. Google the error message. Read three Stack Overflow threads that don't quite match.

8. Escalate and communicate — If you're stuck after 30–45 minutes, you page the next-tier engineer or the app team. Now you're writing a Slack/Teams summary of everything you've tried, copying log snippets, and hoping the other person can ramp up quickly.

9. Apply a fix and verify — Restart a deployment, scale a node pool, roll back a Helm release, or patch a config. Then wait. Re-run the same log and metric queries to confirm the issue is resolved. Update the incident channel.

10. Write the post-mortem — Document what happened, root cause, timeline, and action items. Manually reconstruct the sequence from browser history, terminal scrollback, and memory.

The real cost

Even well-run teams with solid automation (custom scripts, cron-based health checks, ChatOps bots) still face the same fundamental bottleneck: a human must manually connect the dots across logs, metrics, events, configurations, and institutional knowledge — under time pressure, often at odd hours, with incomplete information.

For the business, every minute counts. A prolonged outage doesn't just cost infrastructure dollars — it costs customer trust, SLA credits, and sometimes regulatory penalties. The gap between "alert fired" and "root cause identified" is where the damage accumulates.

In the rest of this post, we'll explore how an agentic approach — combining LLMs, MCP servers, and purpose-built tools like HolmesGPT, the Agentic CLI for AKS, and Azure SRE Agent — can compress that troubleshooting timeline from hours to minutes, reduce the knowledge barrier, and give SRE teams a co-pilot that reasons across your entire platform.

Solution

Now that we've felt the pain, let's look at the agentic tools and projects that can dramatically improve this workflow. This section introduces five complementary projects that enable AI-powered platform operations. Each tackles a different slice of the troubleshooting problem.

Related Projects

How They Relate

┌──────────────────────────────────────────────────────────────────────┐
│                      AI-Powered Platforms                            │
├──────────────────────────────────────────────────────────────────────┤
│  Azure MCP Server ► Azure Services (Storage, Cosmos, Monitor...)    │
│       │                                                              │
│       └──────────► Can be used alongside AKS MCP Server              │
│                                                                      │
│  AKS MCP Server ─► Kubernetes + Inspektor Gadget + Fleet             │
│       │                                                              │
│       └──────────► Consumed by HolmesGPT and Agentic CLI            │
│                                                                      │
│  HolmesGPT ──────► Agentic RCA framework (uses MCP servers)         │
│       │                                                              │
│       └──────────► Powers Agentic CLI for AKS                        │
│                                                                      │
│  Agentic CLI for AKS ────► HolmesGPT for Azure-native experience    │
│                                                                      │
│  Azure SRE Agent ─► Portal-based AI agent for Azure resources        │
│       │              (App Service, Container Apps, Cosmos DB, etc.)   │
│       └──────────► Incident management + automated remediation       │
└──────────────────────────────────────────────────────────────────────┘

💡 Quick Start: If you're new to these tools, start with Azure MCP Server for Azure resource management, then explore AKS MCP Server for Kubernetes operations. For troubleshooting, jump to Agentic CLI for AKS, HolmesGPT, or Azure SRE Agent.

Azure MCP Server

Azure MCP Server is Microsoft's official Model Context Protocol implementation that enables AI agents to interact with Azure services through natural language. It provides a standardized interface for AI tools to manage Azure resources, query data, and perform operations across the Azure ecosystem.

📚 Documentation: Azure MCP Server Overview

Supported Interaction Models:

Deployment options:

• Local (default) — Runs as a local process on the developer's workstation, authenticating via az login. Communication over stdio.

• Remote (preview) — Deployed as a container on Azure Container Apps with Managed Identity. Accessible over HTTP/SSE for use by Microsoft Foundry or custom remote agents.

• Tool categories: Storage (blob, container ops), Databases (Cosmos DB, App Configuration), Resource management (Resource Graph, subscriptions, resource groups), and Monitoring (metrics, logs).

AKS MCP Server

AKS MCP Server extends the Model Context Protocol specifically for Azure Kubernetes Service operations. It provides AI agents with deep Kubernetes integration including cluster management, workload operations, real-time observability via Inspektor Gadget, and multi-cluster fleet management.

📚 Resources: AKS MCP Server | GitHub

Deployment options:

• Local — Runs as a local binary using existing Azure CLI and kubeconfig credentials. Inherits user's Azure RBAC and Kubernetes RBAC.

• Remote (in-cluster) — Deployed via Helm chart for shared access and production diagnostics. Uses Workload Identity and ServiceAccount RBAC.

Tool categories: Cluster management (list, credentials, start/stop), Kubernetes operations (kubectl, resource CRUD, namespaces), Inspektor Gadget / eBPF (TCP trace, DNS trace, process monitor, file ops), Fleet management (multi-cluster view and operations), and Diagnostics (logs, events, describe).

HolmesGPT

HolmesGPT is an open-source agentic AI framework (CNCF Sandbox) that performs root cause analysis (RCA), executes diagnostic tools, and synthesizes insights using natural language prompts.
📚 Resources: HolmesGPT Website | GitHub

Core capabilities:

• Decides what data to fetch based on the issue
• Runs targeted queries against observability tools
• Iteratively refines its hypothesis using LLM reasoning
• Works with existing runbooks and MCP servers
• Runs locally or remotely (in-cluster)
• Read-only by design — safe for production, respects RBAC permissions

Key features:

Who benefits:

• Cluster Operator — Faster incident response, reduced MTTR
• Application Developer — Self-service troubleshooting without deep K8s expertise

Agentic CLI for AKS

The Agentic CLI for AKS (az aks agent) brings agentic AI capabilities directly into the Azure CLI, powered by the AKS Agent (based on HolmesGPT with AKS-specific enhancements). It enables natural language troubleshooting of AKS clusters.

Building Blocks

Source: AKS Blog

📚 Resources: Agentic CLI Overview

The AKS-MCP server acts as a universal, protocol-first bridge between AI agents and AKS. It combines:

Deployment modes:

Client Mode

Runs locally in a Docker container, inheriting the user's Azure and Kubernetes permissions.

Key Points:

🔐 Security Best Practices:

• Uses Azure CLI auth (inherits Azure identity and RBAC)
• Ensure proper RBAC permissions before use
• Use Microsoft Entra integration for authentication
• Audit command usage through Azure activity logs

Cluster Mode

Runs as a pod inside the AKS cluster with explicitly scoped Kubernetes RBAC permissions.

Key Points:

🌐 Networking Considerations:

• Pod connects directly to LLM provider endpoint (not through AKS-MCP)
• ServiceAccount required for in-cluster Kubernetes API access
• Workload Identity optional but necessary for Azure resource access (VM, VMSS, storage)
• Observability data sent to Azure Monitor (Managed Prometheus, Container Insights)
• Optional: Network policies to restrict pod communication

Azure SRE Agent

Azure SRE Agent (preview) is a fully managed, AI-powered service in the Azure portal that continuously monitors your Azure resources, automatically investigates incidents, and can take remediation actions with your approval.

📚 Resources: Azure SRE Agent – Create and use | App Service tutorial | Container Apps tutorial

What it does:

• Receives alerts from Azure Monitor, PagerDuty, or ServiceNow
• Automatically collects and analyzes logs, health probes, metrics, and telemetry
• Determines if an alert is a false positive, summarizes findings, and identifies root cause
• Suggests remediation steps — or, with your explicit approval, executes them (e.g., rolling back a slot swap, scaling a resource)
• Learns from past sessions via a built-in memory system (session insights, investigation quality scoring)

Key differentiators from the other tools:

Azure SRE Agent is best suited for teams that want a portal-first, always-on monitoring experience across multiple Azure services — while HolmesGPT and the Agentic CLI shine for deep, developer-driven Kubernetes troubleshooting with full control over the agent runtime and LLM provider.

When to Use What?

Integration Patterns

References