diff --git a/docs/design/storage-backed-bootstrap.md b/docs/design/storage-backed-bootstrap.md new file mode 100644 index 00000000..8d3150cb --- /dev/null +++ b/docs/design/storage-backed-bootstrap.md @@ -0,0 +1,688 @@ +# Generated bootstrap script + +## Status + +Draft design for a minimal, cluster-specific AKS Flex Node bootstrap script. + +## Context + +A reusable host does not need Azure CLI or a configuration-generation tool. The +control-plane-side publisher already has the cluster bootstrap data and can +render it into a shell script. The caller only needs to download that script, +select runtime Azure authentication when necessary, and run it as root. + +The script downloads `aks-flex-node`, writes the final config, runs preflight, +and starts the node. Configuration manipulation uses the host's existing `jq` +binary rather than adding a bootstrap subcommand to `aks-flex-node`. + +## Goals + +- Keep first-boot orchestration in one portable shell script. +- Embed the cluster/pool-specific base config in the distributed script. +- Support CLI and environment overrides with deterministic precedence. +- Select managed identity or service-principal runtime authentication. +- Download an agent archive from an exact URL or a versioned default URL. +- Generate a protected config and run preflight before start. +- Work from cloud-init, image provisioning, or an interactive root shell. +- Leave the existing agent commands and `scripts/install.sh` unchanged. + +## Non-goals + +- Discovering cluster metadata from the host. +- Creating bootstrap tokens or cluster RBAC. +- Authenticating to the URL used to download the script itself. +- Installing host packages, Azure CLI, curl, tar, or jq. +- Providing a new Go bootstrap command. +- Providing an agent artifact signing or rollback system. + +## Host requirements + +The publisher and operator must call out these prerequisites: + +- Bash +- curl +- tar +- jq +- standard Linux core utilities +- `sha256sum` when an artifact checksum is supplied +- the packages required by normal `aks-flex-node preflight`, such as + `systemd-container`, `nftables`, and `util-linux` + +The script verifies its direct command dependencies and fails with an actionable +message. It never installs missing software. + +## Publishing model + +`scripts/bootstrap.sh` contains exactly one config marker: + +```text +__AKS_FLEX_NODE_BASE_CONFIG_JSON__ +``` + +The publisher replaces that marker with a valid cluster/pool-specific partial +config and stores the generated script at a caller-accessible URL. The +replacement must preserve the shell heredoc delimiters. + +A simple publisher operation is conceptually: + +```python +script = open("scripts/bootstrap.sh").read() +config = open("start-config.json").read().rstrip() +assert script.count("__AKS_FLEX_NODE_BASE_CONFIG_JSON__") == 1 +open("generated-bootstrap.sh", "w").write( + script.replace("__AKS_FLEX_NODE_BASE_CONFIG_JSON__", config) +) +``` + +The generated script contains a bootstrap token and CA material. It is a secret +for the lifetime of those credentials. Its download URL and backing object must +therefore use appropriate confidentiality, expiry, and authorization controls. +The script must not be committed after generation. + +`AKS_FLEX_NODE_BASE_CONFIG_FILE` exists as a development/test escape hatch. The +normal distributed flow uses the embedded config. + +## Expected first-boot flow + +The flow has four actors with deliberately separate responsibilities: + +- The **script publisher** obtains cluster bootstrap data and renders the + cluster-specific script. +- The **image owner** prepares a reusable host image with operating-system + prerequisites, but no cluster credentials or node identity. +- The **provisioner** creates the machine, configures network and Azure identity, + and gives the machine a way to download the generated script. +- The **first-boot environment** downloads and invokes the script once, then + records success outside the script. + +Keeping these responsibilities separate prevents the generic VHD from becoming +cluster-specific and makes it clear that `--auth` configures the installed +agent, not the initial script download. + +### 1. Prepare the reusable image + +Before the image is captured, install the documented host dependencies: + +- Bash, curl, tar, jq, and standard core utilities; +- systemd and nspawn prerequisites; +- nftables and other packages required by agent preflight; +- optional organization-specific CA certificates and proxy configuration. + +Do not place any of the following in the reusable image: + +- the generated bootstrap script; +- a bootstrap token or SP credential; +- `/etc/aks-flex-node/config.json` from another node; +- `/var/lib/aks-flex-node` state; +- a first-boot completion marker from image construction. + +The image can include a wrapper or systemd oneshot unit that performs the +caller-owned script download, but it should obtain all cluster-specific inputs +from provisioning data at instance creation time. + +### 2. Publish a bounded-lifetime script + +For each pool or node enrollment operation, the publisher: + +1. Either embeds the target cluster resource ID, pool name, and ARM endpoint, + or arranges for the provisioner to pass them through dedicated runtime flags. +2. Either embeds fresh `listBootstrapData`, or enables the runtime fetch so the + host obtains a fresh token and pool settings immediately before bootstrap. +3. Leaves host-derived fields such as node name and node IP unset. +4. Replaces the single marker in `scripts/bootstrap.sh` when embedding policy or + bootstrap data. A runtime-fetch flow can leave the marker unpopulated; the + script starts from `{}` when `--fetch-bootstrap-data` is enabled. +5. Selects an agent version or exact archive URL and records the archive + SHA-256. +6. Uploads the generated script to a protected location. +7. Returns the script URL plus non-embedded invocation metadata to the + provisioner. + +A script with an embedded bootstrap token should not outlive that token. A +script using `--fetch-bootstrap-data` can remain reusable longer, but its +runtime identity must be authorized for the AKS `listBootstrapData` action and +any signed artifact URLs still bound its useful lifetime. A node-level script +should be preferred when enrollment is tied to a specific machine resource. + +The publisher must never log the generated script body. Diagnostics can report +the cluster, pool, script object path, token expiry, and artifact digest without +reporting the token or config. + +### 3. Provision identity, network, and download access + +Before first boot runs, the provisioner prepares the host's external +prerequisites: + +- network reachability to the AKS API server, artifact URL, and required image + registries; +- a unique host name suitable for a Kubernetes Node name; +- the selected managed identity or service principal with the permissions + required by the agent's runtime machine-client mode; +- authorization to download the generated script when its URL is private; +- a readable agent artifact URL, such as a short-lived read-only SAS URL. + +For MSI runtime auth, grant the identity the required AKS role before invoking +bootstrap and allow time for role-assignment propagation. For a user-assigned +identity, provide the same client ID both to the runtime config override and to +any caller-owned script-download logic that uses that identity. + +Script-download authorization and agent runtime authorization are independent. +For example, cloud-init can use a user-assigned identity to download a private +script, while the generated script receives a signed agent archive URL and +writes that identity into `azure.managedIdentity` for ARM Machine operations. + +### 4. Download the generated script + +The caller downloads the generated script before invoking it. The bootstrap +script does not authenticate its own download because it is not running yet. +Depending on the environment, the caller can use a signed URL, cloud-init's +provisioning channel, IMDS plus an OAuth request, or an existing configuration +management system. + +For a URL that curl can already read: + +```console +install -d -m 0700 /run/aks-flex-node-bootstrap +curl -fsSLo /run/aks-flex-node-bootstrap/bootstrap.sh "$BOOTSTRAP_SCRIPT_URL" +chmod 0700 /run/aks-flex-node-bootstrap/bootstrap.sh +``` + +When the script is stored in a private Blob container, caller-owned code should +obtain a Storage token and download it without printing the token or URL. The +caller should validate the HTTP result before execution. If the publisher also +returns a script digest or signature, verify it at this boundary. + +Do not use `curl ... | bash` for a generated credential-bearing script. Saving +it first allows permissions, download success, and optional integrity metadata +to be checked before root execution. + +### 5. Invoke once with runtime overrides + +Run the script as root after all roles and network paths are ready. Environment +overrides are convenient for cloud-init and orchestration systems: + +```console +sudo env \ + AKS_FLEX_NODE_AUTH=msi \ + AKS_FLEX_NODE_AGENT_URL="$AGENT_URL" \ + AKS_FLEX_NODE_AGENT_SHA256="$AGENT_SHA256" \ + AKS_FLEX_NODE_CONFIG_OVERRIDES='{"node":{"labels":{"bootstrap":"first-boot"}}}' \ + bash /run/aks-flex-node-bootstrap/bootstrap.sh +``` + +A user-assigned identity can be selected explicitly: + +```console +sudo bash /run/aks-flex-node-bootstrap/bootstrap.sh \ + --auth msi \ + --msi-client-id "$MANAGED_IDENTITY_CLIENT_ID" \ + --agent-url "$AGENT_URL" \ + --agent-sha256 "$AGENT_SHA256" +``` + +For SP auth, deliver the credential as a root-only file and pass its path. Do +not place the secret in cloud-init command arguments, a systemd `ExecStart`, or +a generic JSON override. + +The caller does not need to set node IP. The script leaves it absent unless the +base config or a generic override intentionally supplies it. + +During invocation, the script: + +1. validates prerequisites and the embedded JSON; +2. optionally acquires an ARM token and merges fresh `listBootstrapData`; +3. applies generic environment and CLI overrides; +4. resolves node name and runtime auth; +5. downloads, verifies, and atomically installs the agent; +6. atomically writes the mode `0600` config; +7. runs non-mutating preflight; +8. invokes the existing start lifecycle only after preflight succeeds. + +### 6. Record completion and remove the downloaded script + +A successful invocation ends only after `aks-flex-node start` installs and +starts the long-running service. The first-boot wrapper should then: + +1. remove the downloaded generated script; +2. preserve or arrange recreation of any configured SP `clientSecretFile` for + future agent service restarts; +3. remove only credentials that are not referenced by the durable config; +4. create a root-owned completion marker outside the reusable image; +5. stop retrying the full bootstrap workflow. + +For example: + +```bash +if bash /run/aks-flex-node-bootstrap/bootstrap.sh; then + rm -f /run/aks-flex-node-bootstrap/bootstrap.sh + install -d -m 0755 /var/lib/aks-flex-node + touch /var/lib/aks-flex-node/first-boot-complete +fi +``` + +The completion marker is caller-owned because the caller decides whether and +how to retry failed provisioning. A systemd oneshot wrapper can use +`ConditionPathExists=!/var/lib/aks-flex-node/first-boot-complete` to prevent a +successful node from being bootstrapped again after reboot. + +### 7. Verify convergence + +The provisioning system should not treat script exit alone as complete cluster +convergence. Verify at least: + +- `/etc/aks-flex-node/config.json` is `0600 root:root`; +- `aks-flex-node-agent.service` is active; +- the ARM Machine exists and has reached `Succeeded`; +- the Kubernetes Node has the expected host-derived name and is Ready; +- the Node's InternalIP matches the expected host address even when node IP was + omitted from config; +- the Node joined the intended network site and received a pod CIDR; +- a test workload can start on the node when the environment requires an + end-to-end networking check. + +If the cluster does not run the daemon CSR controller, certificate approval is +an explicit environment prerequisite. A pending daemon CSR can otherwise cause +service retries even after kubelet registration succeeds. + +### Failure and retry expectations + +A nonzero script exit means the first-boot wrapper must not write its completion +marker. Preserve root-only logs and config for diagnosis. The safe retry point +depends on where failure occurred: + +- Download, checksum, JSON rendering, and preflight failures occur before + `start` and can be retried after correcting the input. +- A failure during `start` may leave partial host or ARM Machine state; use the + existing agent diagnostics and lifecycle guidance before retrying. +- After a successful start, do not rerun the full script. The + existing-deployment preflight is expected to reject a second first-boot + attempt. +- If the embedded bootstrap token expires before retry, publish a new generated + script rather than editing the old one on the host. + +## Suggested storage layout + +A private container per cluster keeps generated scripts and artifacts organized +and permits cluster-level data-plane role scoping: + +```text +/ + pools//bootstrap.sh + pools//nodes//bootstrap.sh + agents//linux/amd64/aks-flex-node.tar.gz + agents//linux/arm64/aks-flex-node.tar.gz +``` + +A pool-level script is appropriate when the bootstrap payload can be reused by +multiple nodes for its bounded lifetime. A node-level script is appropriate +when the payload is issued for one machine. + +The script's agent URL must already be readable by curl. It may be public, +network-restricted but anonymously readable from the host, a `file://` URL, or +an HTTPS URL with an embedded SAS. Runtime MSI/SP selection changes the rendered +agent config; it does not add OAuth to curl downloads. + +## Base config contract + +The embedded JSON follows the existing agent config shape and supplies values +that the host cannot derive: + +- subscription and tenant; +- target cluster resource ID and location; +- target FlexNodes pool; +- bootstrap token; +- API server FQDN and CA; +- Kubernetes/component versions; +- DNS/CNI settings; +- machine-client policy and cluster-issued node defaults. + +The base config should omit `agent.nodeName`. The script defaults it to the +lowercase host name. It should also omit `node.kubelet.nodeIP`; the node runtime +can discover the host's primary address. + +Example shape, with credentials redacted: + +```json +{ + "azure": { + "subscriptionId": "", + "tenantId": "", + "targetAgentPoolName": "aksflexnodes", + "bootstrapToken": { + "token": "" + }, + "arc": { + "enabled": false + }, + "targetCluster": { + "resourceId": "/subscriptions/.../managedClusters/", + "location": "eastus" + } + }, + "agent": { + "machineClient": { + "mode": "arm" + }, + "requireMachineRegistration": true + }, + "components": { + "kubernetes": "1.35.6" + }, + "node": { + "kubelet": { + "clusterFQDN": ":443", + "caCertData": "" + } + } +} +``` + +## Input precedence + +Runtime values use this precedence: + +```text +CLI flags > environment variables > embedded base config/script defaults +``` + +The script supports: + +```text +AKS_FLEX_NODE_AUTH +AKS_FLEX_NODE_MSI_CLIENT_ID +AKS_FLEX_NODE_SP_TENANT_ID +AKS_FLEX_NODE_SP_CLIENT_ID +AKS_FLEX_NODE_SP_CLIENT_SECRET +AKS_FLEX_NODE_SP_CLIENT_SECRET_FILE +AKS_FLEX_NODE_AGENT_URL +AKS_FLEX_NODE_AGENT_VERSION +AKS_FLEX_NODE_AGENT_SHA256 +AKS_FLEX_NODE_FETCH_BOOTSTRAP_DATA +AKS_FLEX_NODE_BOOTSTRAP_DATA_API_VERSION +AKS_FLEX_NODE_AUTHORITY_HOST +AKS_FLEX_NODE_CLUSTER_RESOURCE_ID +AKS_FLEX_NODE_AGENT_POOL_NAME +AKS_FLEX_NODE_RESOURCE_MANAGER_ENDPOINT +AKS_FLEX_NODE_BOOTSTRAP_OCI_IMAGE +AKS_FLEX_NODE_BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE +AKS_FLEX_NODE_CONFIG_OVERRIDES +AKS_FLEX_NODE_INSTALL_DIR +AKS_FLEX_NODE_CONFIG_PATH +``` + +The equivalent non-secret values have CLI flags. A service-principal client +secret has no CLI value because command arguments are process-visible. Use a +protected secret file or, when unavoidable, the dedicated environment variable. +The secret file takes precedence over the direct secret environment value. + +When using sudo, the caller must explicitly preserve the needed variables: + +```console +sudo --preserve-env=AKS_FLEX_NODE_AUTH,AKS_FLEX_NODE_AGENT_URL \ + bash bootstrap.sh +``` + +## Config generation + +The script processes JSON in this order: + +1. Write the embedded base config into a mode `0700` temporary workspace. +2. Validate that it is a JSON object. +3. Apply dedicated cluster resource ID, pool name, and ARM endpoint overrides so + they are available to the bootstrap-data request. +4. When enabled, acquire an ARM token with MSI or SP, call + `listBootstrapData`, and deep-merge the response. +5. Deep-merge `AKS_FLEX_NODE_CONFIG_OVERRIDES`, when present. +6. Deep-merge each CLI `--config-overrides` object in invocation order. +7. Reapply dedicated cluster/pool/endpoint overrides so they remain + authoritative. +8. Apply dedicated rootfs and offline-artifact source overrides. +9. Set `agent.nodeName` from the lowercase host name only when absent. +10. Apply the dedicated auth selection. +11. Validate the final JSON with jq. +12. Keep the rendered result in the protected workspace while the agent archive + is downloaded and installed. +13. Atomically install the config at `/etc/aks-flex-node/config.json` with mode + `0600`. +14. Clear bootstrap environment variables, including signed artifact URLs and + any direct SP secret, before launching the agent commands. + +The ARM token, request body, authorization header, and bootstrap-data response +are stored only in mode `0600` files inside the temporary workspace. The script +never prints the response or token and does not require Azure CLI. Resource +Manager and Microsoft Entra authority endpoints are required to use HTTPS. + +When an offline artifact source is present, its manifest remains authoritative +for containerd, runc, and CNI versions. Explicit values for those fields are +removed after the RP and caller merges so stale pool metadata cannot conflict +with the selected bundle. Kubernetes version remains explicit because it +selects the versioned archive. + +Dedicated source and auth selections run after generic merges so callers can +replace publisher/RP artifact URLs without constructing JSON and cannot +accidentally leave multiple incompatible Azure runtime authentication methods +configured. Generic override arguments must not contain secrets because they are +visible in the process list. + +The source overrides map directly to: + +```text +bootstrap.ociImage +bootstrap.offlineArtifacts.source +``` + +They accept HTTPS, local, or OCI source forms supported by the agent. The +offline source preserves `{{ .KubernetesVersion }}` for goal-state rendering. + +### Runtime bootstrap-data refresh + +`--fetch-bootstrap-data` calls the pool action with cluster and pool coordinates +provided by dedicated flags/environment variables or the embedded config: + +```text +POST /agentPools//listBootstrapData + ?api-version=2026-05-02-preview +``` + +For MSI, the script requests an ARM token from IMDS and honors the optional +user-assigned client ID. For SP, it reads the protected secret file, requests a +client-credentials token from Microsoft Entra, and uses the configured ARM +endpoint as the token scope. When `--auth` is omitted, the fetch mode is inferred +from `azure.managedIdentity` or `azure.servicePrincipal` in the base config. + +The response replaces stale cluster-issued values such as the bootstrap token, +API endpoint/CA, component version, DNS, and CNI settings while preserving +publisher-owned settings absent from the response. Caller overrides and the +final auth selection are then applied normally. + +With `--cluster-resource-id`, `--agent-pool-name`, explicit auth inputs, and +artifact source overrides, a reusable script can embed only `{}`. This separates +the generic first-boot executable from cluster coordinates and credentials while +retaining the generated-script option for environments that prefer it. + +### Managed identity + +```console +AKS_FLEX_NODE_AUTH=msi \ +AKS_FLEX_NODE_AGENT_URL="$AGENT_URL" \ +sudo --preserve-env=AKS_FLEX_NODE_AUTH,AKS_FLEX_NODE_AGENT_URL \ + bash bootstrap.sh +``` + +For a user-assigned identity: + +```console +bash bootstrap.sh \ + --auth msi \ + --msi-client-id "$MANAGED_IDENTITY_CLIENT_ID" \ + --agent-url "$AGENT_URL" +``` + +The renderer removes `azure.servicePrincipal`, disables Arc authentication, and +writes either an empty `azure.managedIdentity` object or one with `clientId`. +Bootstrap-token authentication can remain alongside MSI because it serves the +Kubernetes bootstrap path. + +### Service principal + +```console +bash bootstrap.sh \ + --auth service-principal \ + --sp-client-id "$CLIENT_ID" \ + --sp-client-secret-file /run/credentials/aks-flex-node-sp \ + --agent-url "$AGENT_URL" +``` + +The secret file must be non-empty, regular, not a symlink, and inaccessible by +group/other users. The tenant defaults to `azure.tenantId` in the base config +and can be overridden separately. The renderer writes the path to +`azure.servicePrincipal.clientSecretFile` rather than embedding the secret, +removes managed identity, and disables Arc authentication. The credential file +must remain available for config loads on preflight, start, and future service +restarts, or be recreated by the credential manager before service startup. + +## Agent download and installation + +The caller supplies either: + +- `--agent-url` / `AKS_FLEX_NODE_AGENT_URL`; or +- `--agent-version` / `AKS_FLEX_NODE_AGENT_VERSION`. + +A version resolves to the normal GitHub release archive: + +```text +https://github.com/Azure/AKSFlexNode/releases/download//aks-flex-node-linux-.tar.gz +``` + +An exact URL may use these placeholders: + +- `{{OS}}` +- `{{ARCH}}` +- `{{VERSION}}` +- `{{ARCHIVE_NAME}}` + +The script: + +1. Detects AMD64 or ARM64. +2. Downloads the tar.gz with curl retries without printing the URL. +3. Optionally validates the archive SHA-256. +4. Rejects absolute and parent-traversal tar paths. +5. Extracts `aks-flex-node-linux-` or `aks-flex-node`. +6. Atomically replaces `/usr/local/bin/aks-flex-node` with mode `0755`. + +The checksum covers the downloaded archive. Supplying a digest is strongly +recommended, especially for signed URLs or mirrors. + +## Execution flow + +After installing the binary and config, the script executes: + +```console +aks-flex-node preflight --config /etc/aks-flex-node/config.json --output text +aks-flex-node start --config /etc/aks-flex-node/config.json +``` + +Preflight failure stops the script before start. The existing binary owns host +mutation, ARM Machine registration, nspawn provisioning, and systemd service +installation. + +```mermaid +sequenceDiagram + participant Caller + participant Script as Generated bootstrap.sh + participant Artifact as Agent artifact URL + participant Agent as aks-flex-node + participant AKS + + Caller->>Script: download and run with flags/environment + Script->>Script: base config + overrides + host name + auth + Script->>Artifact: curl tar.gz + Artifact-->>Script: agent archive + Script->>Script: checksum, extract, atomic install + Script->>Script: atomic config install mode 0600 + Script->>Agent: preflight + Agent-->>Script: checks pass + Script->>Agent: start + Agent->>AKS: ensure Machine and register Node +``` + +## Cloud-init + +A stock-image test or image-build pipeline can install prerequisites and invoke +the downloaded generated script: + +```yaml +#cloud-config +packages: + - curl + - jq + - nftables + - systemd-container + - tar + - util-linux +runcmd: + - - env + - AKS_FLEX_NODE_AUTH=msi + - AKS_FLEX_NODE_AGENT_URL=https://example/aks-flex-node-linux-amd64.tar.gz + - AKS_FLEX_NODE_CONFIG_OVERRIDES={"node":{"labels":{"bootstrap":"cloud-init"}}} + - bash + - /var/lib/aks-flex-node/bootstrap.sh +``` + +The mechanism that downloads the generated script is caller-owned. For a +private Blob, cloud-init can use a pre-authorized managed identity; for a signed +URL, the custom-data security implications of embedding that URL must be +considered. + +## Security properties + +- The generated script and final config are credential-bearing secrets. +- Temporary files use a mode `0700` directory and process umask `077`. +- Final config is root-owned mode `0600`. +- Service-principal secrets are not accepted as CLI arguments. +- Secret files are rejected when accessible by group/other users. +- Bootstrap environment values and signed agent URLs are cleared before + preflight and start child processes are launched. +- jq performs JSON encoding rather than shell string interpolation. +- Download URLs are not logged by the script. +- Archive traversal paths are rejected. +- Binary and config installation use same-filesystem atomic replacement. +- An optional SHA-256 detects artifact corruption or substitution. +- Preflight must pass before start. + +An embedded SAS is still visible to privileged process inspection while curl is +running. Prefer short expiry and read-only scope. Protect cloud-init custom data +when it contains signed artifact URLs. + +## Recovery and idempotency + +This is a first-boot script. Download and config rendering are safe to repeat +before host mutation. Once `start` has completed, rerunning the whole script may +fail the existing-deployment preflight. A caller should record first-boot +completion and avoid automatic reruns after success. + +If agent download or checksum validation fails, the installed binary is not +replaced. If preflight fails, the rendered config remains available for root-only +diagnostics. The script does not keep an additional rollback binary. + +## Validation + +The final generated-script flow was validated on a fresh Ubuntu 24.04 VM using +cloud-init: + +- cloud-init installed curl, tar, jq, and host prerequisites; +- a user-assigned identity downloaded the private generated script; +- runtime settings were supplied through environment overrides; +- the script downloaded a signed agent archive and verified SHA-256; +- the embedded config omitted node name and node IP; +- the generated node name matched the host name; +- the node runtime discovered the expected private IP; +- config was root-owned mode `0600`; +- all preflight checks passed; +- ARM Machine provisioning reached `Succeeded`; +- the Kubernetes node became Ready in the expected Unbounded site; +- the configured jq label appeared on the Node. + +The lab lacked the daemon CSR controller, so the daemon CSR required manual +approval. That prerequisite is independent of script download, config rendering, +and agent installation. diff --git a/docs/usages/operator-first-boot.md b/docs/usages/operator-first-boot.md new file mode 100644 index 00000000..f4b4bdee --- /dev/null +++ b/docs/usages/operator-first-boot.md @@ -0,0 +1,694 @@ +# Operator Guide: Bootstrap an AKS Flex Node + +This guide describes the current end-to-end operator flow for creating an AKS +cluster with no CNI, installing Unbounded networking, creating a FlexNodes pool, +and joining a prepared Linux host with [`scripts/bootstrap.sh`](../../scripts/bootstrap.sh). + +The bootstrap script is downloaded and run interactively on the host. This guide +does not use cloud-init. + +For the architecture and security rationale, see +[Generated Bootstrap Script Design](../design/storage-backed-bootstrap.md). + +## Flow + +1. Create an AKS cluster with `networkPlugin=none`. +2. Install the Unbounded operator and initialize the cluster and Flex sites. +3. Install the temporary AKS Flex daemon MachineOperation RBAC. +4. Create a FlexNodes agent pool. +5. Prepare a host, download `bootstrap.sh`, and run it with SP or MSI authentication. +6. Approve the daemon CSR when no AKS Flex CSR controller is deployed. +7. Verify the ARM Machine, Kubernetes Node, networking, and agent service. + +## Prerequisites + +The operator workstation needs: + +- Azure CLI authenticated to the target subscription; +- the subscription-level `Microsoft.ContainerService/PutMachinePreview` feature + registered, followed by Microsoft.ContainerService provider re-registration; +- `kubectl`; +- permission to create AKS and networking resources and to grant the selected + pre-provisioned identity access to the AKS cluster; +- access to the AKS admin kubeconfig; +- the `kubectl-unbounded` release matching the Unbounded artifacts. + +The target host needs: + +- Ubuntu 24.04; +- at least 4 vCPU for the validated example; +- a root filesystem with at least 8 GiB free under `/var/lib`; +- Bash, curl, tar, jq, nftables, systemd-container, and util-linux; +- network access to the AKS API server, GitHub agent release, Azure Front Door + artifact mirror, and required container registries; +- a unique lowercase hostname suitable for a Kubernetes Node name. + +This guide uses: + +```bash +export SUBSCRIPTION_ID="" +export RESOURCE_GROUP="" +export AKS_NAME="" +export AKS_LOCATION="" +export AKS_SUBNET_ID="" + +export AKS_VERSION="1.36.2" +export FLEX_VERSION="${FLEX_VERSION:-$AKS_VERSION}" +export FLEX_POOL_NAME="aksflexnodes" + +export SERVICE_CIDR="10.94.0.0/16" +export DNS_SERVICE_IP="10.94.0.10" +export CLUSTER_NODE_CIDR="10.91.0.0/16" +export CLUSTER_POD_CIDR="10.93.0.0/16" +export FLEX_NODE_CIDR="10.92.0.0/16" +export FLEX_POD_CIDR="10.95.0.0/16" + +export UNBOUNDED_VERSION="v0.2.0" +export AKS_FLEX_NODE_VERSION="v0.1.5" +export CENTRAL_ARTIFACTS_ENDPOINT="https://unbounded-azure-mirror-ejd3aeefdrhncchk.b01.azurefd.net" + +# bootstrap.sh downloads the agent, rootfs, and Kubernetes bootstrap bundle +# from this central artifact endpoint. +export AKS_FLEX_NODE_AGENT_URL="${CENTRAL_ARTIFACTS_ENDPOINT}/releases/aks-flex-node/${AKS_FLEX_NODE_VERSION}/{{ARCHIVE_NAME}}" +# For an NVIDIA GPU host, use rootfs-agent-ubuntu2404-nvidia-v20260619.oci.tar.gz. +export BOOTSTRAP_OCI_IMAGE="${CENTRAL_ARTIFACTS_ENDPOINT}/releases/${UNBOUNDED_VERSION}/rootfs/rootfs-agent-ubuntu2404-v20260619.oci.tar.gz" +export BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE="${CENTRAL_ARTIFACTS_ENDPOINT}/releases/${UNBOUNDED_VERSION}/bootstrap-artifacts/bootstrap-artifacts-k8s-{{ .KubernetesVersion }}.tar.gz" +``` + +`FLEX_VERSION` defaults to the AKS control-plane version so the new FlexNodes +pool is aligned with the cluster. Override it only when intentionally using an +AKS-supported version skew. Host bootstrap does not need a separate version +flag; `listBootstrapData` returns the pool's accepted full patch version and the +artifact template resolves from that value. + +The cluster and Flex host networks must have private L3 connectivity. Use one +routed VNet, VNet peering, VPN, ExpressRoute, or an equivalent network design. +The CIDRs above must not overlap. + +## 1. Create a no-CNI AKS cluster + +Select the subscription: + +```bash +az account set --subscription "$SUBSCRIPTION_ID" +``` + +Register the preview feature that permits AKS Flex Node to create or update ARM +Machine resources. Registration is subscription-scoped and only needs to be +completed once: + +```bash +az feature register \ + --namespace Microsoft.ContainerService \ + --name PutMachinePreview +``` + +Wait until registration reports `Registered`: + +```bash +az feature show \ + --namespace Microsoft.ContainerService \ + --name PutMachinePreview \ + --query properties.state \ + --output tsv +``` + +After the state becomes `Registered`, re-register the resource provider so the +feature takes effect: + +```bash +az provider register \ + --namespace Microsoft.ContainerService \ + --wait +``` + +Do not continue to host bootstrap while the feature is still `Registering`. +Without `PutMachinePreview`, the agent cannot complete its ARM Machine +create/update step even when its managed identity or service principal has the +correct AKS role assignment. + +Create or select the resource group and networking before this step. The AKS +subnet ID must refer to the subnet where the managed system pool will run. + +Create AKS with no CNI plugin: + +```bash +az aks create \ + --resource-group "$RESOURCE_GROUP" \ + --name "$AKS_NAME" \ + --location "$AKS_LOCATION" \ + --kubernetes-version "$AKS_VERSION" \ + --nodepool-name nodepool1 \ + --node-count 1 \ + --node-vm-size Standard_D4s_v5 \ + --network-plugin none \ + --vnet-subnet-id "$AKS_SUBNET_ID" \ + --service-cidr "$SERVICE_CIDR" \ + --dns-service-ip "$DNS_SERVICE_IP" \ + --enable-managed-identity \ + --ssh-key-value "$HOME/.ssh/id_rsa.pub" +``` + +Load the admin kubeconfig: + +```bash +az aks get-credentials \ + --resource-group "$RESOURCE_GROUP" \ + --name "$AKS_NAME" \ + --admin \ + --overwrite-existing +``` + +The system Node can initially be `NotReady` because no component has installed a +CNI configuration yet. Unbounded handles that in the next step. + +Verify the cluster version and no-CNI setting: + +```bash +az aks show \ + --resource-group "$RESOURCE_GROUP" \ + --name "$AKS_NAME" \ + --query '{state:provisioningState,version:kubernetesVersion,networkPlugin:networkProfile.networkPlugin}' \ + --output yaml +``` + +## 2. Install Unbounded and initialize the sites + +Download the `kubectl-unbounded` binary appropriate for the workstation: + +```bash +case "$(uname -m)" in + x86_64) UNBOUNDED_ARCH=amd64 ;; + aarch64|arm64) UNBOUNDED_ARCH=arm64 ;; + *) echo "unsupported workstation architecture" >&2; exit 1 ;; +esac + +curl -fsSLo /tmp/kubectl-unbounded.tar.gz \ + "https://github.com/Azure/unbounded/releases/download/${UNBOUNDED_VERSION}/kubectl-unbounded-linux-${UNBOUNDED_ARCH}.tar.gz" + +tar -xzf /tmp/kubectl-unbounded.tar.gz -C /tmp +sudo install -m 0755 /tmp/kubectl-unbounded /usr/local/bin/kubectl-unbounded +kubectl unbounded version +``` + +Initialize the cluster and Flex sites: + +```bash +kubectl unbounded site init \ + --name flex-site \ + --cluster-node-cidr "$CLUSTER_NODE_CIDR" \ + --cluster-pod-cidr "$CLUSTER_POD_CIDR" \ + --node-cidr "$FLEX_NODE_CIDR" \ + --pod-cidr "$FLEX_POD_CIDR" \ + --install-timeout 10m +``` + +This installs the Unbounded operator and enables the networking and Machina +components. It creates: + +- the `cluster` Site for the managed AKS Nodes; +- the `flex-site` Site for external Flex Nodes; +- Unbounded networking CRDs and controllers; +- Machina Machine and MachineOperation CRDs; +- the Unbounded node DaemonSet. + +Create peering between the two sites. The validated VNet-peered environment used +node mesh mode: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: net.unbounded-cloud.io/v1alpha1 +kind: SitePeering +metadata: + name: cluster-flex-private-l3 +spec: + sites: + - cluster + - flex-site + meshNodes: true + tunnelProtocol: Auto +EOF +``` + +Wait for the CNI to converge: + +```bash +kubectl -n unbounded-system rollout status \ + deployment/unbounded-net-controller --timeout=5m + +kubectl -n unbounded-system rollout status \ + daemonset/unbounded-net-node --timeout=5m + +kubectl get nodes -L net.unbounded-cloud.io/site -o wide +kubectl get sites,sitepeerings -o wide +``` + +The managed system Node should become `Ready` and carry the `cluster` site +label. + +## 3. Install temporary AKS Flex daemon RBAC + +> [!IMPORTANT] +> **Temporary preview requirement:** When the Machina MachineOperation CRD is +> installed, AKS Flex Node discovers it and enables its MachineOperation +> reconciler. A future AKS RP release will install and manage the required +> ClusterRole and ClusterRoleBinding automatically as part of FlexNodes pool +> setup. Until that release is deployed in the target region, operators must +> apply the temporary RBAC below manually. + +The Flex daemon certificate belongs to: + +```text +aks-flex-node-daemons +``` + +Install the current integration RBAC: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: aks-flex-node-daemon + labels: + kubernetes.azure.com/managedby: aks +rules: +- apiGroups: + - unbounded-cloud.io + resources: + - machineoperations + - machines + verbs: + - get + - list + - watch +- apiGroups: + - unbounded-cloud.io + resources: + - machineoperations/status + verbs: + - update +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: aks-flex-node-daemon + labels: + kubernetes.azure.com/managedby: aks +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: aks-flex-node-daemon +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: Group + name: aks-flex-node-daemons +EOF +``` + +Without this binding, the host agent can authenticate but exits after its +controller-runtime cache fails to synchronize: + +```text +failed to wait for aks-flex-node-daemon caches to sync +kind source: *v1alpha3.MachineOperation +``` + +This manual RBAC step is a temporary preview requirement. Remove it from the +operator workflow after the AKS RP release that installs the Flex daemon RBAC +automatically is deployed in the target region. + +## 4. Create the FlexNodes pool + +> **TODO:** Azure CLI will support creating FlexNodes pools in a future release. +> Replace this preview `az rest` flow with the supported `az aks nodepool` +> command when that capability becomes available. + +Use the preview agent-pool ARM API. A FlexNodes pool supports only a minimal set +of properties. + +Create the request body: + +```bash +cat > /tmp/flex-pool.json <@" +sudo -i +``` + +Set the operator-provided values: + +```bash +export AKS_RESOURCE_ID="" +export FLEX_POOL_NAME="aksflexnodes" +export AKS_FLEX_NODE_VERSION="v0.1.5" +export UNBOUNDED_VERSION="v0.2.0" +export CENTRAL_ARTIFACTS_ENDPOINT="https://unbounded-azure-mirror-ejd3aeefdrhncchk.b01.azurefd.net" + +# bootstrap.sh downloads the agent, rootfs, and Kubernetes bootstrap bundle +# from this central artifact endpoint. +export AKS_FLEX_NODE_AGENT_URL="${CENTRAL_ARTIFACTS_ENDPOINT}/releases/aks-flex-node/${AKS_FLEX_NODE_VERSION}/{{ARCHIVE_NAME}}" +# For an NVIDIA GPU host, use rootfs-agent-ubuntu2404-nvidia-v20260619.oci.tar.gz. +export BOOTSTRAP_OCI_IMAGE="${CENTRAL_ARTIFACTS_ENDPOINT}/releases/${UNBOUNDED_VERSION}/rootfs/rootfs-agent-ubuntu2404-v20260619.oci.tar.gz" +export BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE="${CENTRAL_ARTIFACTS_ENDPOINT}/releases/${UNBOUNDED_VERSION}/bootstrap-artifacts/bootstrap-artifacts-k8s-{{ .KubernetesVersion }}.tar.gz" + +export FLEX_SP_TENANT_ID="" +export FLEX_SP_CLIENT_ID="" +export FLEX_SP_CLIENT_SECRET_FILE="/etc/aks-flex-node/credentials/sp-client-secret" +``` + +The shell expands `${AKS_FLEX_NODE_VERSION}`, and the bootstrap script expands +`{{ARCHIVE_NAME}}` for the host architecture. On an AMD64 host this resolves to: + +```text +https://unbounded-azure-mirror-ejd3aeefdrhncchk.b01.azurefd.net/releases/aks-flex-node/v0.1.5/aks-flex-node-linux-amd64.tar.gz +``` + +These URLs download the agent, rootfs, and bootstrap bundle from the central +artifact endpoint. Change `CENTRAL_ARTIFACTS_ENDPOINT` to use another mirror. + +AKS RP bootstrap data does not currently include the mirrored rootfs and +offline-artifact locations, so the command supplies them through dedicated CLI +overrides. + +Download the raw script instead of piping it directly to Bash: + +```bash +install -d -m 0700 /run/aks-flex-node-bootstrap + +curl -fsSLo /run/aks-flex-node-bootstrap/bootstrap.sh \ + https://raw.githubusercontent.com/Azure/AKSFlexNode/refs/heads/main/scripts/bootstrap.sh + +chmod 0700 /run/aks-flex-node-bootstrap/bootstrap.sh +bash -n /run/aks-flex-node-bootstrap/bootstrap.sh +``` + +The raw repository script contains an unpopulated embedded-config marker. When +`--fetch-bootstrap-data`, `--cluster-resource-id`, and `--agent-pool-name` are +provided together, the script automatically starts from an empty config and +obtains fresh cluster-issued join settings from AKS RP. No base config file is +required. + +Install the service-principal credential from the operator's protected secret +delivery path. Keep this file available after bootstrap because the running +agent uses it for ARM Machine reconciliation: + +```bash +install -d -o root -g root -m 0700 /etc/aks-flex-node/credentials +install -o root -g root -m 0600 \ + "" \ + "$FLEX_SP_CLIENT_SECRET_FILE" +``` + +Run bootstrap with the service principal: + +```bash +bash /run/aks-flex-node-bootstrap/bootstrap.sh \ + --auth service-principal \ + --sp-tenant-id "$FLEX_SP_TENANT_ID" \ + --sp-client-id "$FLEX_SP_CLIENT_ID" \ + --sp-client-secret-file "$FLEX_SP_CLIENT_SECRET_FILE" \ + --fetch-bootstrap-data \ + --cluster-resource-id "$AKS_RESOURCE_ID" \ + --agent-pool-name "$FLEX_POOL_NAME" \ + --agent-url "$AKS_FLEX_NODE_AGENT_URL" \ + --bootstrap-oci-image "$BOOTSTRAP_OCI_IMAGE" \ + --bootstrap-offline-artifacts-source "$BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE" +``` + +For an Azure VM with a system-assigned managed identity, use the same command +but replace the service-principal flags with: + +```bash + --auth msi \ +``` + +For a user-assigned managed identity, use: + +```bash + --auth msi \ + --msi-client-id "" \ +``` + +The selected managed identity must be assigned to the VM and have the same AKS +Contributor role at the cluster scope. + +The script performs these operations: + +1. Loads the empty JSON base. +2. Applies the cluster and pool overrides. +3. Uses the selected service principal or managed identity to request an ARM + token. +4. Calls `listBootstrapData` for a fresh bootstrap token, API endpoint, CA, and + component version. +5. Applies rootfs, offline artifact, and runtime config overrides. +6. Downloads and verifies the AKS Flex Node agent archive. +7. Writes `/etc/aks-flex-node/config.json` as `0600 root:root`. +8. Runs non-mutating preflight. +9. Registers the ARM Machine and starts the nspawn worker. +10. Installs and starts `aks-flex-node-agent.service`. + +On success, remove the transient files and record completion: + +```bash +rm -f /run/aks-flex-node-bootstrap/bootstrap.sh + +install -d -m 0755 /var/lib/aks-flex-node +install -m 0600 /dev/null /var/lib/aks-flex-node/first-boot-complete +``` + +## 7. Approve the daemon CSR when required + +> [!IMPORTANT] +> **TODO:** A future AKS RP release will automate the Flex daemon CSR approval +> flow. Until that release is deployed in the target region, operators must +> inspect and approve the daemon CSR manually when the AKS Flex CSR approver is +> not present. + +The kubelet node-client CSR is normally approved by the cluster bootstrap RBAC. +The long-running Flex daemon CSR requires the AKS Flex CSR approver. + +If that controller is not deployed, find the pending CSR and inspect it before +manual approval: + +```bash +kubectl get csr + +kubectl get csr -o jsonpath='{.spec.request}' \ + | base64 -d \ + | openssl req -noout -subject +``` + +For the target Node, the daemon CSR should contain: + +```text +CN = system:node: +O = system:nodes +O = aks-flex-node-daemons +``` + +Approve only the verified daemon CSR: + +```bash +kubectl certificate approve +``` + +Manual approval is a lab fallback. Production environments should deploy the +AKS Flex CSR approver. + +## 8. Verify the joined node + +Check the Node and network site: + +```bash +kubectl get nodes -L net.unbounded-cloud.io/site -o wide +kubectl get sites,sitepeerings -o wide +``` + +Check the ARM Machine: + +```bash +MACHINE_NAME="" + +az rest \ + --method get \ + --uri "https://management.azure.com${AKS_RESOURCE_ID}/agentPools/${FLEX_POOL_NAME}/machines/${MACHINE_NAME}?api-version=2025-10-02-preview" \ + --query '{name:name,state:properties.provisioningState,kubernetes:properties.kubernetes}' \ + --output yaml +``` + +On the host: + +```bash +stat -c '%a %U:%G %n' /etc/aks-flex-node/config.json +systemctl is-active aks-flex-node-agent +systemctl is-enabled aks-flex-node-agent +machinectl list +systemctl -M kube1 is-active kubelet containerd +``` + +Expected config permissions: + +```text +600 root:root /etc/aks-flex-node/config.json +``` + +Expected bootstrap log entries: + +```text +bootstrap: fetching fresh bootstrap data from AKS RP +bootstrap: downloading AKS Flex Node agent (URL redacted) +bootstrap: rendered config at /etc/aks-flex-node/config.json +bootstrap: running preflight +bootstrap: starting AKS Flex Node +``` + +Expected ARM Machine registration log: + +```text +level=INFO msg=started task=ensure-machine +level=INFO msg="creating or updating AKS machine" machine= pool=aksflexnodes +level=INFO msg=completed task=ensure-machine status=ok +``` + +Run a smoke workload pinned to the Flex Node to verify pod networking. + +## Troubleshooting + +### System Node remains NotReady + +Ensure the Unbounded operator, controller, and node DaemonSet are running and +that the managed AKS Node has the `cluster` site label: + +```bash +kubectl -n unbounded-system get pods -o wide +kubectl get nodes -L net.unbounded-cloud.io/site +``` + +### Preflight reports insufficient disk space + +Resize the host OS disk. The agent requires at least 8 GiB free under `/var/lib`. +Use 32 GiB or more for the validated examples. + +### ARM Machine registration fails + +Confirm: + +- the selected managed identity is assigned to the host, or the service + principal credential file is present and mode `0600`; +- AKS Contributor is scoped to the target cluster for that identity; +- role assignment propagation has completed; +- `--cluster-resource-id` and `--agent-pool-name` are correct. + +### MachineOperation cache synchronization times out + +Confirm the temporary ClusterRole and ClusterRoleBinding from step 3 exist. A +future AKS RP release will install these resources automatically. + +### Node joins but the host agent restarts + +Inspect pending CSRs. The kubelet can become Ready while the separate daemon CSR +still requires approval. + +```bash +journalctl -u aks-flex-node-agent --no-pager -n 200 +kubectl get csr +``` + +### Bootstrap token expired + +Generate fresh data by rerunning bootstrap with `--fetch-bootstrap-data`. Do not +reuse a generated script or config after its bootstrap token has expired. diff --git a/scripts/bootstrap.sh b/scripts/bootstrap.sh new file mode 100755 index 00000000..1d25eda3 --- /dev/null +++ b/scripts/bootstrap.sh @@ -0,0 +1,673 @@ +#!/bin/bash +# AKS Flex Node first-boot script. +# +# Requirements: bash, curl, tar, and jq. sha256sum is required only when an +# artifact checksum is configured. This script does not install dependencies. +# +# A publisher can replace the marker in write_embedded_base_config with a +# cluster/pool-specific partial config. When --fetch-bootstrap-data is used with +# explicit cluster and pool coordinates, the raw script starts from {} instead. +# Usage guide: docs/usages/operator-first-boot.md + +set -euo pipefail +umask 077 + +readonly DEFAULT_REPOSITORY="Azure/AKSFlexNode" +readonly DEFAULT_INSTALL_DIR="/usr/local/bin" +readonly DEFAULT_CONFIG_PATH="/etc/aks-flex-node/config.json" +readonly DEFAULT_BOOTSTRAP_DATA_API_VERSION="2026-05-02-preview" +readonly DEFAULT_AUTHORITY_HOST="https://login.microsoftonline.com" +readonly DEFAULT_IMDS_ENDPOINT="http://169.254.169.254/metadata/identity/oauth2/token" + +# Publisher-managed cluster/pool config. Keep this block near the top so the +# script generation path only needs to replace the single marker below. +write_embedded_base_config() { + cat <<'AKS_FLEX_NODE_EMBEDDED_CONFIG' +__AKS_FLEX_NODE_BASE_CONFIG_JSON__ +AKS_FLEX_NODE_EMBEDDED_CONFIG +} + +AUTH_MODE="${AKS_FLEX_NODE_AUTH:-}" +MSI_CLIENT_ID="${AKS_FLEX_NODE_MSI_CLIENT_ID:-}" +SP_TENANT_ID="${AKS_FLEX_NODE_SP_TENANT_ID:-}" +SP_CLIENT_ID="${AKS_FLEX_NODE_SP_CLIENT_ID:-}" +SP_CLIENT_SECRET="${AKS_FLEX_NODE_SP_CLIENT_SECRET:-}" +SP_CLIENT_SECRET_FILE="${AKS_FLEX_NODE_SP_CLIENT_SECRET_FILE:-}" +AGENT_URL="${AKS_FLEX_NODE_AGENT_URL:-}" +AGENT_VERSION="${AKS_FLEX_NODE_AGENT_VERSION:-}" +AGENT_SHA256="${AKS_FLEX_NODE_AGENT_SHA256:-}" +INSTALL_DIR="${AKS_FLEX_NODE_INSTALL_DIR:-$DEFAULT_INSTALL_DIR}" +CONFIG_PATH="${AKS_FLEX_NODE_CONFIG_PATH:-$DEFAULT_CONFIG_PATH}" +ENV_CONFIG_OVERRIDES="${AKS_FLEX_NODE_CONFIG_OVERRIDES:-}" +BOOTSTRAP_OCI_IMAGE="${AKS_FLEX_NODE_BOOTSTRAP_OCI_IMAGE:-}" +BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE="${AKS_FLEX_NODE_BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE:-}" +CLUSTER_RESOURCE_ID="${AKS_FLEX_NODE_CLUSTER_RESOURCE_ID:-}" +AGENT_POOL_NAME="${AKS_FLEX_NODE_AGENT_POOL_NAME:-}" +RESOURCE_MANAGER_ENDPOINT="${AKS_FLEX_NODE_RESOURCE_MANAGER_ENDPOINT:-}" +BASE_CONFIG_FILE="${AKS_FLEX_NODE_BASE_CONFIG_FILE:-}" +FETCH_BOOTSTRAP_DATA="${AKS_FLEX_NODE_FETCH_BOOTSTRAP_DATA:-false}" +BOOTSTRAP_DATA_API_VERSION="${AKS_FLEX_NODE_BOOTSTRAP_DATA_API_VERSION:-$DEFAULT_BOOTSTRAP_DATA_API_VERSION}" +AUTHORITY_HOST="${AKS_FLEX_NODE_AUTHORITY_HOST:-$DEFAULT_AUTHORITY_HOST}" +IMDS_ENDPOINT="${AKS_FLEX_NODE_IMDS_ENDPOINT:-$DEFAULT_IMDS_ENDPOINT}" +ALLOW_INSECURE_TEST_ENDPOINTS="${AKS_FLEX_NODE_ALLOW_INSECURE_TEST_ENDPOINTS:-false}" +CONFIG_OVERRIDES=() + +# Keep sensitive caller values in shell memory instead of exporting them to +# every jq, curl, tar, and agent child process. +unset \ + AKS_FLEX_NODE_SP_CLIENT_SECRET \ + AKS_FLEX_NODE_AGENT_URL \ + AKS_FLEX_NODE_BOOTSTRAP_OCI_IMAGE \ + AKS_FLEX_NODE_BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE \ + AKS_FLEX_NODE_CONFIG_OVERRIDES || true +TEMP_DIR="" + +log() { + printf 'bootstrap: %s\n' "$*" >&2 +} + +fatal() { + printf 'bootstrap: error: %s\n' "$*" >&2 + exit 1 +} + +usage() { + cat <<'EOF' +Usage: bootstrap.sh [options] + +Render the embedded AKS Flex Node config, install the agent, run preflight, and +start the node. CLI flags override environment variables, which override the +embedded base config and script defaults. + +Options: + --auth MODE msi or service-principal + --msi-client-id ID Optional user-assigned managed identity client ID + --sp-tenant-id ID Optional SP tenant; defaults to azure.tenantId + --sp-client-id ID Service-principal client ID + --sp-client-secret-file PATH Protected file containing the SP credential + --agent-url URL Exact agent tar.gz URL; may contain a SAS + --agent-version VERSION Version used with the default GitHub release URL + --agent-sha256 SHA256 Expected SHA-256 of the downloaded tar.gz + --fetch-bootstrap-data Fetch and merge fresh AKS RP bootstrap data + --bootstrap-data-api-version V API version for listBootstrapData + --cluster-resource-id ID Target AKS managed-cluster resource ID + --agent-pool-name NAME Target FlexNodes agent pool name + --resource-manager-endpoint URL + ARM endpoint; defaults to public Azure + --bootstrap-oci-image SOURCE Override bootstrap.ociImage + --bootstrap-offline-artifacts-source SOURCE + Override bootstrap.offlineArtifacts.source + --config-overrides JSON JSON object deep-merged into the base config; + repeatable and not suitable for secrets + --install-dir PATH Binary destination directory + --config-path PATH Rendered config destination + -h, --help Show this help + +Environment overrides: + AKS_FLEX_NODE_AUTH + AKS_FLEX_NODE_MSI_CLIENT_ID + AKS_FLEX_NODE_SP_TENANT_ID + AKS_FLEX_NODE_SP_CLIENT_ID + AKS_FLEX_NODE_SP_CLIENT_SECRET + AKS_FLEX_NODE_SP_CLIENT_SECRET_FILE + AKS_FLEX_NODE_AGENT_URL + AKS_FLEX_NODE_AGENT_VERSION + AKS_FLEX_NODE_AGENT_SHA256 + AKS_FLEX_NODE_FETCH_BOOTSTRAP_DATA + AKS_FLEX_NODE_BOOTSTRAP_DATA_API_VERSION + AKS_FLEX_NODE_AUTHORITY_HOST + AKS_FLEX_NODE_CLUSTER_RESOURCE_ID + AKS_FLEX_NODE_AGENT_POOL_NAME + AKS_FLEX_NODE_RESOURCE_MANAGER_ENDPOINT + AKS_FLEX_NODE_BOOTSTRAP_OCI_IMAGE + AKS_FLEX_NODE_BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE + AKS_FLEX_NODE_CONFIG_OVERRIDES + AKS_FLEX_NODE_INSTALL_DIR + AKS_FLEX_NODE_CONFIG_PATH + +The client-secret file takes precedence over AKS_FLEX_NODE_SP_CLIENT_SECRET. +When invoking through sudo, explicitly preserve the required environment +variables. The script requires preinstalled bash, curl, tar, and jq. +EOF +} + +require_value() { + local option="$1" + local value="${2:-}" + [[ -n "$value" ]] || fatal "$option requires a value" +} + +parse_args() { + while (($# > 0)); do + case "$1" in + --auth|--msi-client-id|--sp-tenant-id|--sp-client-id|--sp-client-secret-file|--agent-url|--agent-version|--agent-sha256|--bootstrap-data-api-version|--cluster-resource-id|--agent-pool-name|--resource-manager-endpoint|--bootstrap-oci-image|--bootstrap-offline-artifacts-source|--config-overrides|--install-dir|--config-path) + require_value "$1" "${2:-}" + case "$1" in + --auth) AUTH_MODE="$2" ;; + --msi-client-id) MSI_CLIENT_ID="$2" ;; + --sp-tenant-id) SP_TENANT_ID="$2" ;; + --sp-client-id) SP_CLIENT_ID="$2" ;; + --sp-client-secret-file) SP_CLIENT_SECRET_FILE="$2" ;; + --agent-url) AGENT_URL="$2" ;; + --agent-version) AGENT_VERSION="$2" ;; + --agent-sha256) AGENT_SHA256="$2" ;; + --bootstrap-data-api-version) BOOTSTRAP_DATA_API_VERSION="$2" ;; + --cluster-resource-id) CLUSTER_RESOURCE_ID="$2" ;; + --agent-pool-name) AGENT_POOL_NAME="$2" ;; + --resource-manager-endpoint) RESOURCE_MANAGER_ENDPOINT="$2" ;; + --bootstrap-oci-image) BOOTSTRAP_OCI_IMAGE="$2" ;; + --bootstrap-offline-artifacts-source) BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE="$2" ;; + --config-overrides) CONFIG_OVERRIDES+=("$2") ;; + --install-dir) INSTALL_DIR="$2" ;; + --config-path) CONFIG_PATH="$2" ;; + esac + shift 2 + ;; + --fetch-bootstrap-data) + FETCH_BOOTSTRAP_DATA=true + shift + ;; + -h|--help) + usage + exit 0 + ;; + --) + shift + break + ;; + *) fatal "unknown option: $1" ;; + esac + done + (($# == 0)) || fatal "unexpected positional arguments: $*" +} + +cleanup() { + if [[ -n "$TEMP_DIR" && -d "$TEMP_DIR" ]]; then + rm -rf "$TEMP_DIR" + fi +} + +require_command() { + command -v "$1" >/dev/null 2>&1 || fatal "required command not found: $1" +} + +check_prerequisites() { + local command + for command in curl tar jq mktemp install find; do + require_command "$command" + done + if [[ -n "$AGENT_SHA256" ]]; then + require_command sha256sum + fi +} + +write_base_config() { + local output="$1" + if [[ -n "$BASE_CONFIG_FILE" ]]; then + [[ -f "$BASE_CONFIG_FILE" ]] || fatal "base config file not found: $BASE_CONFIG_FILE" + install -m 0600 "$BASE_CONFIG_FILE" "$output" + return + fi + write_embedded_base_config > "$output" + local embedded + embedded=$(<"$output") + if [[ "$embedded" == __AKS_FLEX_NODE_*__ ]]; then + if is_true "$FETCH_BOOTSTRAP_DATA"; then + log "embedded base config is not populated; starting from an empty config" + printf '{}\n' > "$output" + else + fatal "embedded base config is not populated; use a generated script, a base config file, or --fetch-bootstrap-data" + fi + fi +} + +merge_config_override() { + local current="$1" + local raw="$2" + local label="$3" + local override="$TEMP_DIR/config-override.json" + local merged="$TEMP_DIR/config-merged.json" + + printf '%s' "$raw" > "$override" + jq -e 'type == "object"' "$override" >/dev/null || fatal "$label must be a JSON object" + jq -s '.[0] * .[1]' "$current" "$override" > "$merged" + mv -f "$merged" "$current" +} + +check_secret_file_permissions() { + local path="$1" + local permissions + [[ ! -L "$path" ]] || fatal "service-principal client-secret file must not be a symlink" + [[ -f "$path" ]] || fatal "service-principal client-secret file must be a regular file" + permissions=$(stat -c '%a' "$path") + if (((8#$permissions & 077) != 0)); then + fatal "service-principal client-secret file must not be accessible by group or other users" + fi + [[ -s "$path" ]] || fatal "service-principal client-secret file is empty" +} + +is_true() { + case "${1,,}" in + true|1|yes|y) return 0 ;; + *) return 1 ;; + esac +} + +resolve_fetch_auth_mode() { + local current="$1" + local mode="${AUTH_MODE,,}" + if [[ -z "$mode" ]]; then + if jq -e '.azure.managedIdentity != null' "$current" >/dev/null; then + mode=msi + elif jq -e '.azure.servicePrincipal != null' "$current" >/dev/null; then + mode=service-principal + else + fatal "fetching bootstrap data requires MSI or service-principal authentication" + fi + fi + case "$mode" in + msi|managed-identity) printf 'msi' ;; + sp|service-principal) printf 'service-principal' ;; + *) fatal "fetching bootstrap data does not support auth mode: $mode" ;; + esac +} + +require_https_endpoint() { + local endpoint="$1" + local label="$2" + case "$endpoint" in + https://*) return 0 ;; + http://127.0.0.1:*|http://localhost:*) + is_true "$ALLOW_INSECURE_TEST_ENDPOINTS" && return 0 + ;; + esac + fatal "$label must use HTTPS" +} + +resolve_sp_secret_file() { + local current="$1" + local secret_file="$SP_CLIENT_SECRET_FILE" + if [[ -n "$secret_file" ]]; then + check_secret_file_permissions "$secret_file" + elif [[ -n "$SP_CLIENT_SECRET" ]]; then + secret_file="$TEMP_DIR/fetch-sp-client-secret" + printf '%s' "$SP_CLIENT_SECRET" > "$secret_file" + chmod 0600 "$secret_file" + else + secret_file=$(jq -r '.azure.servicePrincipal.clientSecretFile // ""' "$current") + if [[ -n "$secret_file" ]]; then + check_secret_file_permissions "$secret_file" + else + secret_file="$TEMP_DIR/fetch-sp-client-secret" + jq -er '.azure.servicePrincipal.clientSecret' "$current" > "$secret_file" || \ + fatal "service-principal bootstrap-data fetch requires a client secret or clientSecretFile" + chmod 0600 "$secret_file" + fi + fi + printf '%s' "$secret_file" +} + +acquire_arm_token() { + local current="$1" + local arm_endpoint="$2" + local output="$3" + local mode token_response + mode=$(resolve_fetch_auth_mode "$current") + token_response="$TEMP_DIR/arm-token-response.json" + + case "$mode" in + msi) + local client_id="$MSI_CLIENT_ID" + if [[ -z "$client_id" ]]; then + client_id=$(jq -r '.azure.managedIdentity.clientId // ""' "$current") + fi + local curl_args=( + -fsS --get + --connect-timeout 10 --max-time 60 + --retry 3 --retry-delay 2 --retry-all-errors + -H 'Metadata: true' + --data-urlencode 'api-version=2018-02-01' + --data-urlencode "resource=${arm_endpoint%/}/" + -o "$token_response" + ) + if [[ -n "$client_id" ]]; then + curl_args+=(--data-urlencode "client_id=$client_id") + fi + curl "${curl_args[@]}" "$IMDS_ENDPOINT" || fatal "failed to acquire managed identity ARM token" + ;; + service-principal) + local tenant_id="$SP_TENANT_ID" + local client_id="$SP_CLIENT_ID" + local secret_file request_body token_url + [[ -n "$tenant_id" ]] || tenant_id=$(jq -er '.azure.servicePrincipal.tenantId // .azure.tenantId' "$current") + [[ -n "$client_id" ]] || client_id=$(jq -er '.azure.servicePrincipal.clientId' "$current") + secret_file=$(resolve_sp_secret_file "$current") + request_body="$TEMP_DIR/sp-token-request" + jq -nr --arg clientID "$client_id" --arg scope "${arm_endpoint%/}/.default" --rawfile clientSecret "$secret_file" ' + ($clientSecret | rtrimstr("\n") | rtrimstr("\r")) as $secret | + "client_id=\($clientID | @uri)&client_secret=\($secret | @uri)&scope=\($scope | @uri)&grant_type=client_credentials" + ' > "$request_body" + chmod 0600 "$request_body" + require_https_endpoint "$AUTHORITY_HOST" "service-principal authority host" + token_url="${AUTHORITY_HOST%/}/${tenant_id}/oauth2/v2.0/token" + curl -fsS --connect-timeout 10 --max-time 60 \ + --retry 3 --retry-delay 2 --retry-all-errors \ + -H 'Content-Type: application/x-www-form-urlencoded' \ + --data-binary "@$request_body" -o "$token_response" "$token_url" || \ + fatal "failed to acquire service-principal ARM token" + ;; + esac + + jq -erj '.access_token' "$token_response" > "$output" || fatal "ARM token response did not contain an access token" + chmod 0600 "$output" +} + +apply_target_overrides() { + local current="$1" + local rendered="$TEMP_DIR/config-target.json" + if [[ -z "$CLUSTER_RESOURCE_ID" && -z "$AGENT_POOL_NAME" && -z "$RESOURCE_MANAGER_ENDPOINT" ]]; then + return 0 + fi + jq --arg resourceID "$CLUSTER_RESOURCE_ID" --arg poolName "$AGENT_POOL_NAME" --arg armEndpoint "$RESOURCE_MANAGER_ENDPOINT" ' + .azure = (.azure // {}) | + if $resourceID != "" then + .azure.targetCluster = ((.azure.targetCluster // {}) * {"resourceId": $resourceID}) + else . end | + if $poolName != "" then .azure.targetAgentPoolName = $poolName else . end | + if $armEndpoint != "" then .azure.resourceManagerEndpoint = $armEndpoint else . end + ' "$current" > "$rendered" + mv -f "$rendered" "$current" +} + +fetch_latest_bootstrap_data() { + local current="$1" + local arm_endpoint resource_id pool_name token_file header_file response merged url + arm_endpoint=$(jq -r '.azure.resourceManagerEndpoint // "https://management.azure.com"' "$current") + resource_id=$(jq -er '.azure.targetCluster.resourceId' "$current") + pool_name=$(jq -er '.azure.targetAgentPoolName' "$current") + require_https_endpoint "$arm_endpoint" "resource manager endpoint" + shopt -s nocasematch + if [[ ! "$resource_id" =~ ^/subscriptions/[^/?#[:space:]]+/resourceGroups/[^/?#[:space:]]+/providers/Microsoft\.ContainerService/managedClusters/[^/?#[:space:]]+$ ]]; then + shopt -u nocasematch + fatal "embedded config contains an invalid target cluster resource ID" + fi + shopt -u nocasematch + [[ "$pool_name" =~ ^[A-Za-z0-9-]+$ ]] || fatal "embedded config contains an invalid target agent pool name" + [[ "$BOOTSTRAP_DATA_API_VERSION" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}(-preview)?$ ]] || \ + fatal "bootstrap-data API version has an invalid format" + + token_file="$TEMP_DIR/arm-access-token" + header_file="$TEMP_DIR/arm-headers" + response="$TEMP_DIR/bootstrap-data.json" + merged="$TEMP_DIR/config-with-bootstrap-data.json" + acquire_arm_token "$current" "$arm_endpoint" "$token_file" + { + printf 'Authorization: Bearer ' + cat "$token_file" + printf '\nContent-Type: application/json\n' + } > "$header_file" + chmod 0600 "$header_file" + + url="${arm_endpoint%/}${resource_id}/agentPools/${pool_name}/listBootstrapData?api-version=${BOOTSTRAP_DATA_API_VERSION}" + log "fetching fresh bootstrap data from AKS RP" + curl -fsS --connect-timeout 10 --max-time 120 \ + -X POST -H "@$header_file" --data '' -o "$response" "$url" || \ + fatal "failed to fetch AKS bootstrap data" + chmod 0600 "$response" + jq -e 'type == "object" and (.azure.bootstrapToken.token | type == "string" and length > 0)' "$response" >/dev/null || \ + fatal "AKS bootstrap-data response did not contain a bootstrap token" + jq -s '.[0] * .[1]' "$current" "$response" > "$merged" + mv -f "$merged" "$current" +} + +apply_bootstrap_source_overrides() { + local current="$1" + local rendered="$TEMP_DIR/config-bootstrap-sources.json" + if [[ -z "$BOOTSTRAP_OCI_IMAGE" && -z "$BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE" ]]; then + return 0 + fi + jq --arg ociImage "$BOOTSTRAP_OCI_IMAGE" --arg offlineSource "$BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE" ' + .bootstrap = (.bootstrap // {}) | + if $ociImage != "" then .bootstrap.ociImage = $ociImage else . end | + if $offlineSource != "" then + .bootstrap.offlineArtifacts = ((.bootstrap.offlineArtifacts // {}) * {"source": $offlineSource}) + else . end + ' "$current" > "$rendered" + mv -f "$rendered" "$current" +} + +normalize_offline_artifact_versions() { + local current="$1" + local rendered="$TEMP_DIR/config-offline-versions.json" + jq ' + if (.bootstrap.offlineArtifacts.source // "") != "" then + .components = (.components // {}) | + .networking = (.networking // {}) | + del(.components.containerd) | + del(.components.runc) | + del(.networking.cniVersion) + else . end + ' "$current" > "$rendered" + mv -f "$rendered" "$current" +} + +apply_auth_override() { + local current="$1" + local rendered="$TEMP_DIR/config-auth.json" + local mode="${AUTH_MODE,,}" + + [[ -n "$mode" ]] || return 0 + case "$mode" in + msi|managed-identity) + jq --arg clientID "$MSI_CLIENT_ID" ' + .azure = (.azure // {}) | + del(.azure.servicePrincipal) | + .azure.arc = ((.azure.arc // {}) * {"enabled": false}) | + .azure.managedIdentity = if $clientID == "" then {} else {"clientId": $clientID} end + ' "$current" > "$rendered" + ;; + sp|service-principal) + [[ -n "$SP_CLIENT_ID" ]] || fatal "service-principal auth requires --sp-client-id or AKS_FLEX_NODE_SP_CLIENT_ID" + if [[ -n "$SP_CLIENT_SECRET_FILE" ]]; then + check_secret_file_permissions "$SP_CLIENT_SECRET_FILE" + jq --arg clientID "$SP_CLIENT_ID" --arg tenantID "$SP_TENANT_ID" --arg clientSecretFile "$SP_CLIENT_SECRET_FILE" ' + .azure = (.azure // {}) | + del(.azure.managedIdentity) | + .azure.arc = ((.azure.arc // {}) * {"enabled": false}) | + .azure.servicePrincipal = { + "tenantId": (if $tenantID == "" then .azure.tenantId else $tenantID end), + "clientId": $clientID, + "clientSecretFile": $clientSecretFile + } + ' "$current" > "$rendered" + elif [[ -n "$SP_CLIENT_SECRET" ]]; then + local secret_file="$TEMP_DIR/sp-client-secret" + printf '%s' "$SP_CLIENT_SECRET" > "$secret_file" + chmod 0600 "$secret_file" + jq --arg clientID "$SP_CLIENT_ID" --arg tenantID "$SP_TENANT_ID" --rawfile clientSecret "$secret_file" ' + ($clientSecret | rtrimstr("\n") | rtrimstr("\r")) as $secret | + .azure = (.azure // {}) | + del(.azure.managedIdentity) | + .azure.arc = ((.azure.arc // {}) * {"enabled": false}) | + .azure.servicePrincipal = { + "tenantId": (if $tenantID == "" then .azure.tenantId else $tenantID end), + "clientId": $clientID, + "clientSecret": $secret + } + ' "$current" > "$rendered" + else + fatal "service-principal auth requires a protected secret file or AKS_FLEX_NODE_SP_CLIENT_SECRET" + fi + ;; + *) fatal "unsupported auth mode: $AUTH_MODE (expected msi or service-principal)" ;; + esac + mv -f "$rendered" "$current" +} + +render_config() { + local output="$1" + local current="$TEMP_DIR/config-current.json" + local rendered="$TEMP_DIR/config-rendered.json" + local node_name + + write_base_config "$current" + jq -e 'type == "object"' "$current" >/dev/null || fatal "embedded base config must be a JSON object" + apply_target_overrides "$current" + + if is_true "$FETCH_BOOTSTRAP_DATA"; then + fetch_latest_bootstrap_data "$current" + fi + + if [[ -n "$ENV_CONFIG_OVERRIDES" ]]; then + merge_config_override "$current" "$ENV_CONFIG_OVERRIDES" "AKS_FLEX_NODE_CONFIG_OVERRIDES" + fi + local override + for override in "${CONFIG_OVERRIDES[@]}"; do + merge_config_override "$current" "$override" "--config-overrides" + done + apply_target_overrides "$current" + apply_bootstrap_source_overrides "$current" + normalize_offline_artifact_versions "$current" + + node_name=$(hostname | tr '[:upper:]' '[:lower:]') + jq --arg nodeName "$node_name" ' + .agent = (.agent // {}) | + if (.agent.nodeName // "") == "" then .agent.nodeName = $nodeName else . end + ' "$current" > "$rendered" + mv -f "$rendered" "$current" + + apply_auth_override "$current" + jq -e . "$current" > "$output" + chmod 0600 "$output" +} + +detect_architecture() { + case "$(uname -m)" in + x86_64) printf 'amd64' ;; + aarch64) printf 'arm64' ;; + *) fatal "unsupported architecture: $(uname -m)" ;; + esac +} + +resolve_agent_url() { + local arch="$1" + local archive="aks-flex-node-linux-${arch}.tar.gz" + local url="$AGENT_URL" + if [[ -z "$url" ]]; then + [[ -n "$AGENT_VERSION" ]] || fatal "set --agent-url/AKS_FLEX_NODE_AGENT_URL or --agent-version/AKS_FLEX_NODE_AGENT_VERSION" + url="https://github.com/${DEFAULT_REPOSITORY}/releases/download/${AGENT_VERSION}/${archive}" + fi + url="${url//\{\{OS\}\}/linux}" + url="${url//\{\{ARCH\}\}/$arch}" + url="${url//\{\{VERSION\}\}/$AGENT_VERSION}" + url="${url//\{\{ARCHIVE_NAME\}\}/$archive}" + printf '%s' "$url" +} + +validate_archive_paths() { + local archive="$1" + local listing="$TEMP_DIR/archive-listing" + tar -tzf "$archive" > "$listing" || fatal "agent download is not a readable tar.gz archive" + local entry + while IFS= read -r entry; do + [[ "$entry" != /* && "$entry" != ".." && "$entry" != ../* && "$entry" != */../* ]] || fatal "agent archive contains an unsafe path" + done < "$listing" +} + +download_and_install_agent() { + local arch="$1" + local url archive extract_dir expected candidate staged + url=$(resolve_agent_url "$arch") + archive="$TEMP_DIR/agent.tar.gz" + extract_dir="$TEMP_DIR/agent" + expected="aks-flex-node-linux-${arch}" + + log "downloading AKS Flex Node agent (URL redacted)" + curl -fsSL --retry 3 --retry-delay 2 -o "$archive" "$url" || fatal "failed to download AKS Flex Node agent" + + if [[ -n "$AGENT_SHA256" ]]; then + local actual + actual=$(sha256sum "$archive" | awk '{print $1}') + [[ "${actual,,}" == "${AGENT_SHA256,,}" ]] || fatal "agent archive SHA-256 mismatch" + fi + + validate_archive_paths "$archive" + mkdir -p "$extract_dir" + tar -xzf "$archive" -C "$extract_dir" + candidate=$(find "$extract_dir" -type f \( -name "$expected" -o -name aks-flex-node \) -print -quit) + [[ -n "$candidate" ]] || fatal "agent binary not found in archive" + + install -d -o root -g root -m 0755 "$INSTALL_DIR" + staged=$(mktemp "$INSTALL_DIR/.aks-flex-node.XXXXXX") + install -o root -g root -m 0755 "$candidate" "$staged" + mv -f "$staged" "$INSTALL_DIR/aks-flex-node" + log "installed agent at $INSTALL_DIR/aks-flex-node" +} + +clear_bootstrap_environment() { + SP_CLIENT_SECRET="" + AGENT_URL="" + ENV_CONFIG_OVERRIDES="" + unset \ + AKS_FLEX_NODE_AUTH \ + AKS_FLEX_NODE_MSI_CLIENT_ID \ + AKS_FLEX_NODE_SP_TENANT_ID \ + AKS_FLEX_NODE_SP_CLIENT_ID \ + AKS_FLEX_NODE_SP_CLIENT_SECRET \ + AKS_FLEX_NODE_SP_CLIENT_SECRET_FILE \ + AKS_FLEX_NODE_AGENT_URL \ + AKS_FLEX_NODE_AGENT_VERSION \ + AKS_FLEX_NODE_AGENT_SHA256 \ + AKS_FLEX_NODE_FETCH_BOOTSTRAP_DATA \ + AKS_FLEX_NODE_BOOTSTRAP_DATA_API_VERSION \ + AKS_FLEX_NODE_AUTHORITY_HOST \ + AKS_FLEX_NODE_IMDS_ENDPOINT \ + AKS_FLEX_NODE_ALLOW_INSECURE_TEST_ENDPOINTS \ + AKS_FLEX_NODE_CLUSTER_RESOURCE_ID \ + AKS_FLEX_NODE_AGENT_POOL_NAME \ + AKS_FLEX_NODE_RESOURCE_MANAGER_ENDPOINT \ + AKS_FLEX_NODE_BOOTSTRAP_OCI_IMAGE \ + AKS_FLEX_NODE_BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE \ + AKS_FLEX_NODE_CONFIG_OVERRIDES \ + AKS_FLEX_NODE_INSTALL_DIR \ + AKS_FLEX_NODE_CONFIG_PATH || true +} + +install_config() { + local rendered="$1" + local config_dir staged + config_dir=$(dirname "$CONFIG_PATH") + install -d -o root -g root -m 0755 "$config_dir" + staged=$(mktemp "$config_dir/.config.json.XXXXXX") + install -o root -g root -m 0600 "$rendered" "$staged" + mv -f "$staged" "$CONFIG_PATH" + log "rendered config at $CONFIG_PATH" +} + +main() { + parse_args "$@" + [[ $EUID -eq 0 ]] || fatal "run this script as root" + check_prerequisites + + TEMP_DIR=$(mktemp -d) + chmod 0700 "$TEMP_DIR" + trap cleanup EXIT + + local arch rendered_config + arch=$(detect_architecture) + rendered_config="$TEMP_DIR/config.json" + render_config "$rendered_config" + download_and_install_agent "$arch" + install_config "$rendered_config" + clear_bootstrap_environment + + log "running preflight" + "$INSTALL_DIR/aks-flex-node" preflight --config "$CONFIG_PATH" --output text + log "starting AKS Flex Node" + umask 022 + "$INSTALL_DIR/aks-flex-node" start --config "$CONFIG_PATH" +} + +if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then + main "$@" +fi diff --git a/scripts/bootstrap_test.sh b/scripts/bootstrap_test.sh new file mode 100755 index 00000000..d1af7445 --- /dev/null +++ b/scripts/bootstrap_test.sh @@ -0,0 +1,350 @@ +#!/bin/bash + +set -euo pipefail + +if [[ $EUID -ne 0 ]]; then + exec sudo -E bash "$0" "$@" +fi + +REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +SCRIPT="$REPO_ROOT/scripts/bootstrap.sh" +WORK_DIR=$(mktemp -d) +SERVER_PID="" +cleanup() { + if [[ -n "$SERVER_PID" ]]; then + kill "$SERVER_PID" 2>/dev/null || true + fi + rm -rf "$WORK_DIR" +} +trap cleanup EXIT + +fail() { + printf 'bootstrap_test: %s\n' "$*" >&2 + exit 1 +} + +command -v jq >/dev/null || fail "jq is required" +bash -n "$SCRIPT" + +case "$(uname -m)" in + x86_64) ARCH=amd64 ;; + aarch64) ARCH=arm64 ;; + *) fail "unsupported test architecture" ;; +esac + +make_agent_archive() { + local dir="$1" + mkdir -p "$dir" + cat > "$dir/aks-flex-node-linux-$ARCH" <<'AGENT' +#!/bin/bash +for variable in AKS_FLEX_NODE_AGENT_URL AKS_FLEX_NODE_SP_CLIENT_SECRET AKS_FLEX_NODE_CONFIG_OVERRIDES AKS_FLEX_NODE_FETCH_BOOTSTRAP_DATA AKS_FLEX_NODE_AUTHORITY_HOST AKS_FLEX_NODE_IMDS_ENDPOINT AKS_FLEX_NODE_ALLOW_INSECURE_TEST_ENDPOINTS AKS_FLEX_NODE_CLUSTER_RESOURCE_ID AKS_FLEX_NODE_AGENT_POOL_NAME AKS_FLEX_NODE_RESOURCE_MANAGER_ENDPOINT AKS_FLEX_NODE_BOOTSTRAP_OCI_IMAGE AKS_FLEX_NODE_BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE; do + [[ -z "${!variable+x}" ]] || exit 23 +done +printf '%s\n' "$*" >> "${BOOTSTRAP_TEST_CALLS:?}" +AGENT + chmod 0755 "$dir/aks-flex-node-linux-$ARCH" + tar -C "$dir" -czf "$dir/agent.tar.gz" "aks-flex-node-linux-$ARCH" +} + +make_agent_archive "$WORK_DIR/agent" +AGENT_URL="file://$WORK_DIR/agent/agent.tar.gz" +AGENT_SHA256=$(sha256sum "$WORK_DIR/agent/agent.tar.gz" | awk '{print $1}') + +cat > "$WORK_DIR/base.json" <<'JSON' +{ + "azure": { + "tenantId": "base-tenant", + "servicePrincipal": { + "tenantId": "old", + "clientId": "old", + "clientSecret": "old" + }, + "arc": {"enabled": true} + }, + "agent": {}, + "node": {"labels": {"base": "true"}} +} +JSON +chmod 0600 "$WORK_DIR/base.json" + +BOOTSTRAP_TEST_CALLS="$WORK_DIR/msi-calls" \ +AKS_FLEX_NODE_BASE_CONFIG_FILE="$WORK_DIR/base.json" \ +AKS_FLEX_NODE_AUTH=service-principal \ +AKS_FLEX_NODE_SP_CLIENT_ID=environment-client \ +AKS_FLEX_NODE_SP_CLIENT_SECRET=environment-secret \ +AKS_FLEX_NODE_AGENT_URL="$AGENT_URL" \ +AKS_FLEX_NODE_AGENT_SHA256="$AGENT_SHA256" \ +AKS_FLEX_NODE_BOOTSTRAP_OCI_IMAGE='https://environment.example/rootfs.tar.gz' \ +AKS_FLEX_NODE_BOOTSTRAP_OFFLINE_ARTIFACTS_SOURCE='https://environment.example/bootstrap-k8s-{{ .KubernetesVersion }}.tar.gz' \ +AKS_FLEX_NODE_CONFIG_OVERRIDES='{"node":{"labels":{"environment":"true"}}}' \ + bash "$SCRIPT" \ + --auth msi \ + --msi-client-id cli-msi \ + --bootstrap-oci-image 'https://cli.example/rootfs.tar.gz' \ + --config-overrides '{"node":{"labels":{"cli":"true"}},"bootstrap":{"offlineArtifacts":{"source":"https://generic-cli.example/ignored.tar.gz"}}}' \ + --install-dir "$WORK_DIR/msi-bin" \ + --config-path "$WORK_DIR/msi-etc/config.json" >/dev/null + +jq -e ' + .azure.managedIdentity.clientId == "cli-msi" and + (.azure | has("servicePrincipal") | not) and + .azure.arc.enabled == false and + .bootstrap.ociImage == "https://cli.example/rootfs.tar.gz" and + .bootstrap.offlineArtifacts.source == "https://environment.example/bootstrap-k8s-{{ .KubernetesVersion }}.tar.gz" and + .node.labels == {"base":"true", "environment":"true", "cli":"true"} and + (.agent.nodeName | length > 0) +' "$WORK_DIR/msi-etc/config.json" >/dev/null +[[ $(stat -c '%a' "$WORK_DIR/msi-etc/config.json") == 600 ]] || fail "MSI config mode is not 0600" +grep -Fx "preflight --config $WORK_DIR/msi-etc/config.json --output text" "$WORK_DIR/msi-calls" >/dev/null +grep -Fx "start --config $WORK_DIR/msi-etc/config.json" "$WORK_DIR/msi-calls" >/dev/null + +printf 's"e\\cret\n' > "$WORK_DIR/client-secret" +chmod 0600 "$WORK_DIR/client-secret" +BOOTSTRAP_TEST_CALLS="$WORK_DIR/sp-calls" \ +AKS_FLEX_NODE_BASE_CONFIG_FILE="$WORK_DIR/base.json" \ + bash "$SCRIPT" \ + --auth service-principal \ + --sp-client-id cli-client \ + --sp-client-secret-file "$WORK_DIR/client-secret" \ + --agent-url "$AGENT_URL" \ + --agent-sha256 "$AGENT_SHA256" \ + --install-dir "$WORK_DIR/sp-bin" \ + --config-path "$WORK_DIR/sp-etc/config.json" >/dev/null + +jq -e --arg secretFile "$WORK_DIR/client-secret" ' + .azure.servicePrincipal == { + "tenantId":"base-tenant", + "clientId":"cli-client", + "clientSecretFile":$secretFile + } and + (.azure | has("managedIdentity") | not) and + .azure.arc.enabled == false +' "$WORK_DIR/sp-etc/config.json" >/dev/null + +BOOTSTRAP_TEST_CALLS="$WORK_DIR/sp-inline-calls" \ +AKS_FLEX_NODE_BASE_CONFIG_FILE="$WORK_DIR/base.json" \ +AKS_FLEX_NODE_SP_CLIENT_SECRET='inline-secret' \ +AKS_FLEX_NODE_AGENT_URL="$AGENT_URL" \ + bash "$SCRIPT" \ + --auth service-principal \ + --sp-client-id inline-client \ + --install-dir "$WORK_DIR/sp-inline-bin" \ + --config-path "$WORK_DIR/sp-inline-etc/config.json" >/dev/null +jq -e ' + .azure.servicePrincipal.clientId == "inline-client" and + .azure.servicePrincipal.clientSecret == "inline-secret" and + (.azure.servicePrincipal | has("clientSecretFile") | not) +' "$WORK_DIR/sp-inline-etc/config.json" >/dev/null + +ln -s "$WORK_DIR/client-secret" "$WORK_DIR/client-secret-link" +if BOOTSTRAP_TEST_CALLS="$WORK_DIR/sp-link-calls" \ + AKS_FLEX_NODE_BASE_CONFIG_FILE="$WORK_DIR/base.json" \ + AKS_FLEX_NODE_AGENT_URL="$AGENT_URL" \ + bash "$SCRIPT" --auth service-principal \ + --sp-client-id link-client \ + --sp-client-secret-file "$WORK_DIR/client-secret-link" \ + --install-dir "$WORK_DIR/sp-link-bin" \ + --config-path "$WORK_DIR/sp-link-etc/config.json" \ + >"$WORK_DIR/sp-link.log" 2>&1; then + fail "symlink client-secret file was accepted" +fi +grep -q 'client-secret file must not be a symlink' "$WORK_DIR/sp-link.log" || \ + fail "symlink client-secret rejection was not reported" + +command -v python3 >/dev/null || fail "python3 is required by the bootstrap-data test" +cat > "$WORK_DIR/bootstrap-data-server.py" <<'PY' +import json +import sys +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +class Handler(BaseHTTPRequestHandler): + def log_message(self, *_): + pass + + def write_json(self, value): + data = json.dumps(value).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + def do_GET(self): + if self.path.startswith("/metadata/identity/oauth2/token?"): + self.write_json({"access_token": "test-arm-token", "expires_in": "3600"}) + else: + self.send_error(404) + + def do_POST(self): + if self.path == "/base-tenant/oauth2/v2.0/token": + length = int(self.headers.get("Content-Length", "0")) + body = self.rfile.read(length).decode() + if "client_id=base-sp-client" not in body or "client_secret=base-sp-secret" not in body: + self.send_error(401) + return + self.write_json({"access_token": "test-arm-token", "expires_in": 3600}) + return + expected = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/cluster/agentPools/aksflexnodes/listBootstrapData?api-version=2026-05-02-preview" + if self.path != expected: + self.send_error(404) + return + if self.headers.get("Authorization") != "Bearer test-arm-token": + self.send_error(401) + return + self.write_json({ + "azure": {"bootstrapToken": {"token": "fresh1.0123456789abcdef"}}, + "components": {"kubernetes": "1.35.6"}, + "networking": {"dnsServiceIP": "10.0.0.10", "cniVersion": "stale-cni"}, + }) + +server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) +with open(sys.argv[1], "w") as handle: + handle.write(str(server.server_address[1])) +server.serve_forever() +PY +python3 "$WORK_DIR/bootstrap-data-server.py" "$WORK_DIR/bootstrap-data-port" & +SERVER_PID=$! +for _ in $(seq 1 50); do + [[ -s "$WORK_DIR/bootstrap-data-port" ]] && break + sleep 0.1 +done +[[ -s "$WORK_DIR/bootstrap-data-port" ]] || fail "bootstrap-data test server did not start" +port=$(<"$WORK_DIR/bootstrap-data-port") +cat > "$WORK_DIR/fetch-base.json" </dev/null + +jq -e --arg armEndpoint "http://127.0.0.1:${port}" ' + .azure.targetCluster.resourceId == "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/cluster" and + .azure.targetAgentPoolName == "aksflexnodes" and + .azure.resourceManagerEndpoint == $armEndpoint and + .azure.bootstrapToken.token == "fresh1.0123456789abcdef" and + .azure.managedIdentity == {} and + .components.kubernetes == "1.35.6" and + .networking.dnsServiceIP == "10.0.0.10" and + (.networking | has("cniVersion") | not) and + (.components | has("containerd") | not) and + (.components | has("runc") | not) and + .node.labels.fresh == "true" +' "$WORK_DIR/fetch-etc/config.json" >/dev/null + +# The raw repository script can start from an implicit empty object when fresh +# bootstrap data and the target cluster/pool are supplied. +BOOTSTRAP_TEST_CALLS="$WORK_DIR/fetch-empty-base-calls" \ +AKS_FLEX_NODE_IMDS_ENDPOINT="http://127.0.0.1:${port}/metadata/identity/oauth2/token" \ +AKS_FLEX_NODE_ALLOW_INSECURE_TEST_ENDPOINTS=true \ +AKS_FLEX_NODE_AGENT_URL="$AGENT_URL" \ + bash "$SCRIPT" \ + --fetch-bootstrap-data \ + --auth msi \ + --cluster-resource-id '/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/cluster' \ + --agent-pool-name aksflexnodes \ + --resource-manager-endpoint "http://127.0.0.1:${port}" \ + --install-dir "$WORK_DIR/fetch-empty-base-bin" \ + --config-path "$WORK_DIR/fetch-empty-base-etc/config.json" >/dev/null + +jq -e --arg armEndpoint "http://127.0.0.1:${port}" ' + .azure.targetCluster.resourceId == "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/cluster" and + .azure.targetAgentPoolName == "aksflexnodes" and + .azure.resourceManagerEndpoint == $armEndpoint and + .azure.bootstrapToken.token == "fresh1.0123456789abcdef" and + .azure.managedIdentity == {} and + .components.kubernetes == "1.35.6" and + (.agent.nodeName | length > 0) +' "$WORK_DIR/fetch-empty-base-etc/config.json" >/dev/null + +if BOOTSTRAP_TEST_CALLS="$WORK_DIR/no-base-calls" \ + AKS_FLEX_NODE_AGENT_URL="$AGENT_URL" \ + bash "$SCRIPT" --auth msi \ + --install-dir "$WORK_DIR/no-base-bin" \ + --config-path "$WORK_DIR/no-base-etc/config.json" \ + >"$WORK_DIR/no-base.log" 2>&1; then + fail "unpopulated embedded config was accepted without bootstrap-data fetch" +fi +grep -q 'embedded base config is not populated' "$WORK_DIR/no-base.log" || \ + fail "missing base config rejection was not reported" + +if BOOTSTRAP_TEST_CALLS="$WORK_DIR/insecure-calls" \ + AKS_FLEX_NODE_BASE_CONFIG_FILE="$WORK_DIR/fetch-base.json" \ + AKS_FLEX_NODE_IMDS_ENDPOINT="http://127.0.0.1:${port}/metadata/identity/oauth2/token" \ + AKS_FLEX_NODE_AGENT_URL="$AGENT_URL" \ + bash "$SCRIPT" --fetch-bootstrap-data --auth msi \ + --cluster-resource-id '/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/cluster' \ + --agent-pool-name aksflexnodes \ + --resource-manager-endpoint "http://127.0.0.1:${port}" \ + --install-dir "$WORK_DIR/insecure-bin" \ + --config-path "$WORK_DIR/insecure-etc/config.json" \ + >"$WORK_DIR/insecure.log" 2>&1; then + fail "HTTP resource manager endpoint was accepted" +fi +grep -q 'resource manager endpoint must use HTTPS' "$WORK_DIR/insecure.log" || \ + fail "HTTP endpoint rejection was not reported" + +printf '%s' 'base-sp-secret' > "$WORK_DIR/fetch-sp-secret" +chmod 0600 "$WORK_DIR/fetch-sp-secret" +cat > "$WORK_DIR/fetch-sp-base.json" </dev/null + +jq -e --arg secretFile "$WORK_DIR/fetch-sp-secret" ' + .azure.bootstrapToken.token == "fresh1.0123456789abcdef" and + .azure.servicePrincipal == { + "tenantId":"base-tenant", + "clientId":"base-sp-client", + "clientSecretFile":$secretFile + } and + .components.kubernetes == "1.35.6" +' "$WORK_DIR/fetch-sp-etc/config.json" >/dev/null + +echo "bootstrap script tests passed"