[BUG] Azure Linux 3 Node Image containerd config uses deprecated v1 API path causing ACR authentication failures

[msftsiwei]

Describe the bug

Azure Linux 3 node pools (Node Image 202607.20.0) fail to pull private images from Azure Container Registry (ACR) with 401 Unauthorized errors.

Root cause: /etc/containerd/config.toml on Azure Linux 3 nodes uses the deprecated containerd v1 API path (plugins.cri.v1.images) instead of the correct containerd v2 path (plugins.grpc.v1.cri). This causes containerd v2.2.4 to ignore the registry configuration, preventing kubelet credential provider from being invoked for ACR authentication.

To Reproduce

Steps to reproduce the behavior:

1. Create an Azure Linux 3 node pool:

az aks nodepool add \
  --resource-group <resource-group> \
  --cluster-name <cluster-name> \
  --name testpool \
  --node-count 1 \
  --node-vm-size Standard_D4s_v3 \
  --os-sku AzureLinux \
  --kubernetes-version 1.33.0

2. Deploy a pod with a private ACR image:

kubectl run test-acr --image=<your-acr>.azurecr.io/your-private-image:latest \
  --overrides='{"spec":{"nodeSelector":{"agentpool":"testpool"}}}'

3. Check pod status and events:

kubectl get pod test-acr
kubectl describe pod test-acr | grep -A5 "Events:"

4. Observe error: Failed to pull image: rpc error: code = Unknown desc = failed to pull and unpack image: failed to resolve reference: pulling from host <acr>.azurecr.io failed with status code [manifests latest]: 401 Unauthorized

Expected behavior

Pod should pull private ACR images successfully. The kubelet credential provider (acr-credential-provider) should automatically handle ACR authentication without requiring imagePullSecrets.

This works correctly on:

• Ubuntu node pools (same AKS version)
• Older Azure Linux node images (e.g., 202607.02.0)

Screenshots

N/A - This is a configuration issue visible in logs and config files.

Environment (please complete the following information):

• CLI Version: 2.68.0
• Kubernetes version: 1.33.0, 1.33.1 (affects both)
• CLI Extension version: aks-preview 10.0.0b8
• Node OS: Azure Linux 3.0 (CBL-Mariner)
• Node Image Version: 202607.20.0
• containerd Version: 2.2.4
• Affected SKUs: All VM sizes with Azure Linux 3 (tested on Standard_NC48ads_A100_v4, Standard_D4s_v3)

Additional context

Root Cause: Deprecated containerd v1 API Path

The node image uses an incorrect containerd configuration path. On affected Azure Linux 3 nodes (/etc/containerd/config.toml):

Current (broken):

version = 2
[plugins."io.containerd.cri.v1.images"]          ← v1 API (deprecated)
  [plugins."io.containerd.cri.v1.images".registry]
    config_path = "/etc/containerd/certs.d"

Expected (working on older images):

version = 2
[plugins."io.containerd.grpc.v1.cri"]            ← v2 API (correct)
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"

containerd v2.x cannot read the deprecated plugins.cri.v1.* path, causing it to ignore registry configuration and skip the kubelet credential provider.

Cross-Environment Testing

Tested on two clusters to isolate the issue:

Environment	Node Image	containerd	Config Path	ACR Pull
Cluster A	202607.02.0	2.2.4	plugins.grpc.v1.cri ✅	✅ Success
Cluster B	202607.20.0	2.2.4	plugins.cri.v1.images ❌	❌ 401 Error

Both clusters have identical:

• containerd version (2.2.4)
• ACR credential provider binary (/opt/azure/containers/bin/acr-credential-provider)
• kubelet credential config (/var/lib/kubelet/credential-provider-config.yaml)

Only difference: containerd config API path

Verification on Affected Node

# Check node image version
cat /etc/os-release
# NAME="Microsoft Azure Linux"
# VERSION="3.0.20240807"

# Check containerd config
cat /etc/containerd/config.toml | grep -A3 "plugins.*images"
# [plugins."io.containerd.cri.v1.images"]   ← Wrong: v1 API

# Verify containerd version
containerd --version
# containerd github.com/containerd/containerd/v2 v2.2.4

Workarounds

Temporary fix (lost on node reboot):

# On affected node (via kubectl debug)
cp /etc/containerd/config.toml /etc/containerd/config.toml.bak
sed -i 's/io\.containerd\.cri\.v1\.images/io.containerd.grpc.v1.cri/g' /etc/containerd/config.toml
systemctl restart containerd

Production workaround (use imagePullSecrets):

kubectl create secret docker-registry acr-secret \
  --docker-server=<acr>.azurecr.io \
  --docker-username=<sp-id> \
  --docker-password=<sp-password>

Requested Fix

Update the containerd config template in Azure Linux 3 node image build pipeline to use containerd v2-compatible API path:

-[plugins."io.containerd.cri.v1.images"]
+[plugins."io.containerd.grpc.v1.cri"]

This aligns with:

• containerd v2 migration guide
• Existing working configuration in Node Image 202607.02.0
• Kubernetes best practices for containerd v2

Impact

• Affected: All Azure Linux 3 node pools (Node Image 202607.20.0+)
• Scope: All private ACR image pulls fail without imagePullSecrets
• Urgency: Medium (workaround available, but breaks default ACR integration)

[microsoft-github-policy-service]

Sylvain Boily (@djsly) would you be able to assist?