Skip to content

feat: install Kubernetes to Flatcar from sysexts on MAR #7680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
awesomenix merged 12 commits into from
Feb 18, 2026
Merged

feat: install Kubernetes to Flatcar from sysexts on MAR #7680

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
6b1328c
refactor: simplify configureKubeletAndKubectl logic
chewi Aug 28, 2025
e0efe31
refactor: simplify credential provider installation logic
chewi Sep 23, 2025
2f77b43
style: rename installKubeletKubectlPkgFromPMC to *FromPkg
chewi Aug 28, 2025
e77f241
rename: installCredentialProviderFromPMC to *FromPkg
chewi Sep 23, 2025
3c50a0a
feat: install kubelet and kubectl to Flatcar from sysexts on MAR
chewi Aug 28, 2025
16a6654
feat: install credential provider to Flatcar from sysext on MAR
chewi Sep 23, 2025
bfc61f8
test: don't mix distro scripts as they clash and use BeforeEach properly
chewi Jan 21, 2026
39f421a
test: add shellspec tests for new Flatcar and sysext code
chewi Sep 1, 2025
d6d5408
fix: adjust Flatcar aks-sysext to not use /etc or /opt/bin
chewi Jan 16, 2026
e8a40e4
fix: move apt_get_download to Ubuntu-specific script file
chewi Jan 19, 2026
37a5e3a
fix: only provision Ubuntu with /opt/azure/manifest.json
chewi Jan 19, 2026
5ed3e71
chore: update test data
chewi Sep 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/README-RENOVATE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,8 @@ where
- `${version}` will be resolved at runtime with the `latestVersion` and `previousLatestVersion` defined above.
- `${CPU_ARCH}` will be resolved at runtime depending on the CPU architecture of the Node (VM) under provisioning.

systemd system extensions (sysexts) are also hosted as MAR OCI artifacts, but they use a slightly different `extractVersion` rule and `downloadURL`. The distribution (e.g. `azlinux3`) is included in the version to allow different distributions within groups of artifacts. `${SYSTEMD_ARCH}` rather than `${CPU_ARCH}` is used in the URL, as systemd has different architecture names in some cases.

## `REVISION` in Dalec built container images
Dalec-built container images use static tags in the form `vMAJOR.MINOR.PATCH-REVISION` (see the Dalec FAQ https://github.com/Azure/dalec-build-defs/blob/main/faq.md#how-do-floating-vs-static-tags-work for details). For clarity and deterministic caching we represent these container images in Agent Baker's `components.json` using the exact static tag `vMAJOR.MINOR.PATCH-REVISION`.

Expand Down
13 changes: 12 additions & 1 deletion .github/renovate.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,8 @@
},
{
"matchPackageNames": [
"oss/v2/**"
"oss/v2/**",
"!oss/v2/kubernetes/*-sysext"
Comment thread
djsly marked this conversation as resolved.
],
Comment thread
chewi marked this conversation as resolved.
"versioning": "regex:^v(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)-(?<prerelease>\\d+)$",
"ignoreUnstable": false
Expand Down Expand Up @@ -479,6 +480,16 @@
],
"extractVersion": "^(?P<version>.*?)-[^-]*-[^-]*$"
},
{
"matchDatasources": [
"docker"
],
"matchPackageNames": [

@djsly djsly Jan 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

who would be the owner of such packages ? I'm not sure if they would have automatic assigned in the PR

@cameronmeissner cameronmeissner Jan 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably Chewi + Jeremi as primary owners, and maybe us as secondary?

@chewi chewi Jan 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that we need to be owners. These sysexts are published in tandem with their .deb and .rpm counterparts from the very same Dalec YAML definition. Adding sysext support is usually just a case of adding one line.

"oss/v2/kubernetes/*-sysext"
],
"matchCurrentVersion": "/-azlinux3$/",
Comment thread
djsly marked this conversation as resolved.
"extractVersion": "^(?P<version>.+-azlinux3)-"
},
{
"matchPackageNames": [
"aks/aks-gpu-cuda"
Expand Down
145 changes: 117 additions & 28 deletions parts/common/components.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1470,6 +1470,53 @@
}
]
}
},
"flatcar": {
"current": {
"versionsV2": [
{
"k8sVersion": "1.28",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.28.101-4-azlinux3"
Comment thread
chewi marked this conversation as resolved.
},
{
"k8sVersion": "1.29",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.29.100-2-azlinux3"
},
{
"k8sVersion": "1.30",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.30.100-2-azlinux3"
},
{
"k8sVersion": "1.31",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.31.14-1-azlinux3"
},
{
"k8sVersion": "1.32",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.32.11-1-azlinux3"
},
{
"k8sVersion": "1.33",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.33.7-1-azlinux3"
},
{
"k8sVersion": "1.34",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.34.3-1-azlinux3"
},
{
"k8sVersion": "1.35",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubelet-sysext",
"latestVersion": "v1.35.0-1-azlinux3"
}
],
"downloadURL": "mcr.microsoft.com/oss/v2/kubernetes/kubelet-sysext:${version}-${SYSTEMD_ARCH}"

@djsly djsly Jan 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we check for azlinux3 for the versions when we overwrite the systemd arch when we donwload ?

@chewi chewi Jan 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I don't understand the question.

}
}
}
},
Expand Down Expand Up @@ -1536,6 +1583,53 @@
}
]
}
},
"flatcar": {
"current": {
"versionsV2": [
{
"k8sVersion": "1.28",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.28.101-6-azlinux3"
},
{
"k8sVersion": "1.29",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.29.100-3-azlinux3"
},
{
"k8sVersion": "1.30",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.30.100-3-azlinux3"
},
{
"k8sVersion": "1.31",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.31.14-1-azlinux3"
},
{
"k8sVersion": "1.32",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.32.11-1-azlinux3"
},
{
"k8sVersion": "1.33",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.33.7-1-azlinux3"
},
{
"k8sVersion": "1.34",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.34.3-1-azlinux3"
},
{
"k8sVersion": "1.35",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/kubectl-sysext",
"latestVersion": "v1.35.0-1-azlinux3"
}
],
"downloadURL": "mcr.microsoft.com/oss/v2/kubernetes/kubectl-sysext:${version}-${SYSTEMD_ARCH}"
}
}
}
},
Expand Down Expand Up @@ -1646,34 +1740,7 @@
},
"flatcar": {
"current": {
"versionsV2": [
{
"k8sVersion": "1.30",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.30.15"
},
{
"k8sVersion": "1.31",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.31.12"
},
{
"k8sVersion": "1.32",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.32.11"
},
{
"k8sVersion": "1.33",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.33.6"
},
{
"k8sVersion": "1.34",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/binaries/kubernetes/azure-acr-credential-provider",
"latestVersion": "v1.34.3"
}
],
"downloadURL": "mcr.microsoft.com/oss/binaries/kubernetes/azure-acr-credential-provider:${version}-linux-${CPU_ARCH}"
"versionsV2": []
}
}
}
Expand Down Expand Up @@ -1783,6 +1850,28 @@
}
]
}
},
"flatcar": {
"current": {
"versionsV2": [
{
"k8sVersion": "1.32",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/azure-acr-credential-provider-sysext",
"latestVersion": "v1.32.11-1-azlinux3"
},
{
"k8sVersion": "1.33",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/azure-acr-credential-provider-sysext",
"latestVersion": "v1.33.6-1-azlinux3"
},
{
"k8sVersion": "1.34",
"renovateTag": "OCI_registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/azure-acr-credential-provider-sysext",
"latestVersion": "v1.34.3-1-azlinux3"
}
],
"downloadURL": "mcr.microsoft.com/oss/v2/kubernetes/azure-acr-credential-provider-sysext:${version}-${SYSTEMD_ARCH}"
}
}
}
},
Expand Down
4 changes: 2 additions & 2 deletions parts/linux/cloud-init/artifacts/azlosguard/cse_install_osguard.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ stub() {
echo "${FUNCNAME[1]} stub"
}

installKubeletKubectlPkgFromPMC() {
installKubeletKubectlFromPkg() {
local desiredVersion="${1}"
installRPMPackageFromFile "kubelet" $desiredVersion || exit $ERR_KUBELET_INSTALL_FAIL
installRPMPackageFromFile "kubectl" $desiredVersion || exit $ERR_KUBECTL_INSTALL_FAIL
Comment thread
chewi marked this conversation as resolved.
Expand Down Expand Up @@ -67,7 +67,7 @@ downloadPkgFromVersion() {
echo "Succeeded to download ${packageName} version ${packageVersion}"
}

installCredentialProviderFromPMC() {
installCredentialProviderFromPkg() {
k8sVersion="${1:-}"
os=${AZURELINUX_OS_NAME}
if [ -z "$OS_VERSION" ]; then
Expand Down
61 changes: 27 additions & 34 deletions parts/linux/cloud-init/artifacts/cse_config.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,27 +539,22 @@ EOF
}

configureKubeletAndKubectl() {
# Install kubelet and kubectl binaries from URL for Custom Kube binary and Private Kube binary
if [ -n "${CUSTOM_KUBE_BINARY_DOWNLOAD_URL}" ] || [ -n "${PRIVATE_KUBE_BINARY_DOWNLOAD_URL}" ]; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromURL" installKubeletKubectlFromURL
# only install kube pkgs from pmc if k8s version >= 1.34.0 or skip_bypass_k8s_version_check is true
elif [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != "true" ] && ! semverCompare ${KUBERNETES_VERSION:-"0.0.0"} "1.34.0"; then
# Install kubelet and kubectl binaries from URL:
# 1. For Custom Kube binary or Private Kube binary.
# 2. If k8s version < 1.34.0, skip_bypass_k8s_version_check != true, and not Flatcar (which falls back to URL later).
# 3. For Azure Linux v2 due to lack of PMC packages (if not network isolated).
if [ -n "${CUSTOM_KUBE_BINARY_DOWNLOAD_URL}" ] || [ -n "${PRIVATE_KUBE_BINARY_DOWNLOAD_URL}" ] ||
{ ! isFlatcar && [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != true ] && ! semverCompare "${KUBERNETES_VERSION:-0.0.0}" 1.34.0; } ||
Comment thread
djsly marked this conversation as resolved.
{ isMarinerOrAzureLinux && [ "${OS_VERSION}" = 2.0 ] && [ -z "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ]; }
then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromURL" installKubeletKubectlFromURL
elif [ -n "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ]; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromBootstrapProfileRegistry" "installKubeletKubectlFromBootstrapProfileRegistry ${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER} ${KUBERNETES_VERSION}"
elif [ "$(type -t installKubeletKubectlFromPkg)" = function ]; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromPkg" "installKubeletKubectlFromPkg ${KUBERNETES_VERSION}"
else
if [ -n "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ] ; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromBootstrapProfileRegistry" "installKubeletKubectlFromBootstrapProfileRegistry ${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER} ${KUBERNETES_VERSION}"
elif isMarinerOrAzureLinux "$OS"; then
if [ "$OS_VERSION" = "2.0" ]; then
# we do not publish packages to PMC for azurelinux V2
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromURL" installKubeletKubectlFromURL
else
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlPkgFromPMC" "installKubeletKubectlPkgFromPMC ${KUBERNETES_VERSION}"
fi
elif [ "${OS}" = "${UBUNTU_OS_NAME}" ]; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlPkgFromPMC" "installKubeletKubectlPkgFromPMC ${KUBERNETES_VERSION}"
elif [ "${OS}" = "${FLATCAR_OS_NAME}" ]; then
logs_to_events "AKS.CSE.configureKubeletAndKubectl.installKubeletKubectlFromURL" installKubeletKubectlFromURL
fi
echo "installKubeletKubectlFromPkg is not defined for this OS"
exit $ERR_K8S_INSTALL_ERR
fi
}

Expand Down Expand Up @@ -754,23 +749,21 @@ EOF
if [[ $KUBELET_FLAGS == *"image-credential-provider-config"* && $KUBELET_FLAGS == *"image-credential-provider-bin-dir"* ]]; then
echo "Configure credential provider for both image-credential-provider-config and image-credential-provider-bin-dir flags are specified in KUBELET_FLAGS"
logs_to_events "AKS.CSE.ensureKubelet.configCredentialProvider" configCredentialProvider
if { [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != "true" ] && ! semverCompare ${KUBERNETES_VERSION:-"0.0.0"} "1.34.0"; }; then
# Install credential provider from URL:
# 1. If k8s version < 1.34.0, skip_bypass_k8s_version_check != true, and not Flatcar (which falls back to URL later).
# 2. For Azure Linux v2 due to lack of PMC packages (if not network isolated).
if { ! isFlatcar && [ "${SHOULD_ENFORCE_KUBE_PMC_INSTALL}" != true ] && ! semverCompare "${KUBERNETES_VERSION:-0.0.0}" 1.34.0; } ||
{ isMarinerOrAzureLinux && [ "${OS_VERSION}" = 2.0 ] && [ -z "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ]; }
then
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromUrl" installCredentialProviderFromUrl
elif [ -n "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ]; then
# For network isolated clusters, try distro packages first and fallback to binary installation
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromBootstrapProfileRegistry" installCredentialProviderPackageFromBootstrapProfileRegistry ${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER} ${KUBERNETES_VERSION}
elif [ "$(type -t installCredentialProviderFromPkg)" = function ]; then
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromPkg" "installCredentialProviderFromPkg ${KUBERNETES_VERSION}"
else
if [ -n "${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER}" ] ; then
# For network isolated clusters, try distro packages first and fallback to binary installation
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromBootstrapProfileRegistry" installCredentialProviderPackageFromBootstrapProfileRegistry ${BOOTSTRAP_PROFILE_CONTAINER_REGISTRY_SERVER} ${KUBERNETES_VERSION}
elif isMarinerOrAzureLinux "$OS"; then
if [ "$OS_VERSION" = "2.0" ]; then # PMC package installation not supported for AzureLinux V2, only V3
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromUrl" installCredentialProviderFromUrl
else
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromPMC" "installCredentialProviderFromPMC ${KUBERNETES_VERSION}"
fi
elif isFlatcar "$OS"; then # Flatcar cannot use PMC. It will use sysext soon.
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromUrl" installCredentialProviderFromUrl
else
logs_to_events "AKS.CSE.ensureKubelet.installCredentialProviderFromPMC" "installCredentialProviderFromPMC ${KUBERNETES_VERSION}"
fi
echo "installCredentialProviderFromPkg is not defined for this OS"
exit $ERR_CREDENTIAL_PROVIDER_DOWNLOAD_TIMEOUT
Comment thread
chewi marked this conversation as resolved.
fi
fi

Expand Down
35 changes: 9 additions & 26 deletions parts/linux/cloud-init/artifacts/cse_helpers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ ERR_NVIDIA_DCGM_EXPORTER_FAIL=229 # Error starting or enabling NVIDIA DCGM Expor
ERR_LOOKUP_ENABLE_MANAGED_GPU_EXPERIENCE_TAG=230 # Error checking nodepool tags for whether we need to enable managed GPU experience

ERR_PULL_POD_INFRA_CONTAINER_IMAGE=225 # Error pulling pause image
ERR_ORAS_PULL_SYSEXT_FAIL=231 # Error pulling systemd system extension artifact via oras from registry

# ----------------------- AKS Node Controller----------------------------------
ERR_AKS_NODE_CONTROLLER_ERROR=240 # Generic error in AKS Node Controller
Expand Down Expand Up @@ -565,32 +566,6 @@ semverCompare() {
return 1
}



apt_get_download() {
retries=$1; wait_sleep=$2; shift && shift;
local ret=0
pushd $APT_CACHE_DIR || return 1
for i in $(seq 1 "$retries"); do
dpkg --configure -a --force-confdef
wait_for_apt_locks

# Pull the first quoted URL from --print-uris
url="$(apt-get --print-uris -o Dpkg::Options::=--force-confold download -y -- "$@" \
| awk -F"'" 'NR==1 && $2 {print $2}')"
if [ -n "$url" ]; then
# This avoids issues with the naming in the package. `apt-get download`
# encodes the package names with special characters and does not decode
# them when saving to disk, but `curl -J` handles the names correctly.
if curl -fLJO -- "$url"; then ret=0; break; fi
fi

if [ "$i" -eq "$retries" ]; then ret=1; else sleep "$wait_sleep"; fi
done
popd || return 1
return "$ret"
}

getCPUArch() {
arch=$(uname -m)
# shellcheck disable=SC3010
Expand All @@ -601,6 +576,14 @@ getCPUArch() {
fi
}

getSystemdArch() {
local seArch=$(getCPUArch)
case ${seArch} in
amd64) echo x86-64 ;;
*) echo "${seArch}" ;;
esac
}

isARM64() {
if [ "$(getCPUArch)" = "arm64" ]; then
echo 1
Expand Down
Loading
Loading