Skip to content

scenario: AKS pods can't pull from ACR — missing private DNS zone link (enterprise Azure) #58

Description

@AhmadHammad21

Context

Real-world enterprise Azure case described by a practicing SRE: AKS cluster
can't pull images from ACR (Azure Container Registry), but no clear error
surfaces
— pulls just hang and time out. The triage path goes:

  1. Check IAM / permissions (cluster identity has AcrPull role) — fine.
  2. Check pod / pull-secret config — fine.
  3. Deploy a diagnostic pod, kubectl exec into it, try to resolve the ACR
    hostname from inside the cluster — nxdomain / no answer.
  4. Realize the cluster runs on a private VNet, ACR is in a private endpoint
    with a private DNS zone, and the VNet isn't linked to the zone.
    The pod has no DNS route to the ACR's private IP.

Root cause: missing private-DNS-zone-to-VNet link (or a misconfigured
vnet link / coreDNS override). Symptom: pods can't resolve the ACR's
private FQDN; only visible by nslookup from inside a pod.

Why this is a good eval scenario

  • Multi-layer diagnosis (RBAC → pod config → DNS → networking).
  • Forces the agent to use kubectl exec to run a diagnostic pod — the same
    pattern human SREs use.
  • Tests cross-resource reasoning (the failing resource is the pod; the
    broken resource is the DNS zone link).
  • Realistic: this exact incident shows up regularly in enterprise Azure
    deployments and is hard to diagnose without the right mental model.

Designed at `demos/eval/scenarios/_planned/011_aks_dns_resolution.md` so
the framework knows about it but the runner skips it until the setup
script exists.

TODO

Setup script `demos/azure/aks_dns_resolution.py` needs to:

  1. Create resource group + VNet with subnet for AKS.
  2. Create an AKS cluster with kubenet/CNI and private API server.
  3. Create an ACR with private endpoint connected to the VNet.
  4. Deliberately skip the `az network private-dns link vnet create`
    that would link `privatelink.azurecr.io` to the cluster's VNet.
  5. Deploy a small workload (`kubectl apply`) that references an image
    in the ACR. Pod stays in `ImagePullBackOff`.

Teardown: delete the resource group (async, releases all child resources).

Cost / time: AKS provisioning takes 10-15 min. Standard SKU ~$0.10/hr.

Acceptance

  • `uv run python demos/eval/run.py --scenario 011 --setup-profile ` passes.
  • Move `011_aks_dns_resolution` out of `_planned/` so the runner picks it up.
  • Add the scenario to the README's headline numbers when the full sweep is re-run.

Filed from chat 2026-05-31 while expanding the eval suite.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions