From cc3683ae13c4676d25c51fd3f9b027ab2976e881 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Sun, 7 Jun 2026 17:49:13 +0200 Subject: [PATCH 1/5] docs: correct overlay selection description in CI pipeline section The build-images workflow selects exactly one overlay directory (tenant-specific if it exists, otherwise shared) rather than applying both in layers. Update the README to match the actual either/or behavior. Co-Authored-By: Claude Sonnet 4.6 --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index de28b57..3014c25 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ The `build-images` workflow does the following on relevant pushes: 2. Iterates over `tenants/*/*.yaml`. 3. Reads `source_image` and optional `rootfs_size_mb` from each tenant file. 4. Builds `--rootfs.ext4` via `scripts/docker-to-rootfs.sh`. -5. Applies config overlays with precedence: - - `configs/-/` (tenant-specific) - - then `configs//` (shared) +5. Selects one config overlay directory (if present): + - `configs/-/` (tenant-specific, checked first) + - `configs//` (shared, used as fallback when no tenant-specific overlay exists) 6. Uploads resulting `*-rootfs.ext4` artifacts to S3. From b27830c68d6599679267a463182c20695c89ec27 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Sun, 7 Jun 2026 17:55:44 +0200 Subject: [PATCH 2/5] fix: apply shared and tenant-specific config overlays in layers Previously the workflow selected only one overlay directory (tenant-specific if present, otherwise shared), so shared configs were silently skipped whenever a tenant-specific overlay existed. Change docker-to-rootfs.sh to accept a colon-separated list of overlay directories and apply them in order (later entries override earlier ones). Update the workflow to build the colon-separated path when both a shared and a tenant-specific overlay exist, so shared configs serve as the baseline and tenant-specific configs override them. Also update the README to reflect the correct layered behavior. Co-Authored-By: Claude Sonnet 4.6 --- .github/workflows/build-images.yaml | 7 +++++-- README.md | 6 +++--- scripts/docker-to-rootfs.sh | 23 +++++++++++++++-------- 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/.github/workflows/build-images.yaml b/.github/workflows/build-images.yaml index 3387359..885350f 100644 --- a/.github/workflows/build-images.yaml +++ b/.github/workflows/build-images.yaml @@ -82,9 +82,12 @@ jobs: size_mb="$(yq '.rootfs_size_mb // 512' "$svc_file")" output="${tenant_id}-${base_name}-rootfs.ext4" - # Config overlay: tenant-specific overlay takes precedence. + # Config overlays: shared baseline applied first, tenant-specific on top. + # Pass as a colon-separated list so docker-to-rootfs.sh applies them in order. overlay_arg="" - if [ -d "configs/${tenant_id}-${base_name}" ]; then + if [ -d "configs/${base_name}" ] && [ -d "configs/${tenant_id}-${base_name}" ]; then + overlay_arg="configs/${base_name}:configs/${tenant_id}-${base_name}" + elif [ -d "configs/${tenant_id}-${base_name}" ]; then overlay_arg="configs/${tenant_id}-${base_name}" elif [ -d "configs/${base_name}" ]; then overlay_arg="configs/${base_name}" diff --git a/README.md b/README.md index 3014c25..5960363 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ The `build-images` workflow does the following on relevant pushes: 2. Iterates over `tenants/*/*.yaml`. 3. Reads `source_image` and optional `rootfs_size_mb` from each tenant file. 4. Builds `--rootfs.ext4` via `scripts/docker-to-rootfs.sh`. -5. Selects one config overlay directory (if present): - - `configs/-/` (tenant-specific, checked first) - - `configs//` (shared, used as fallback when no tenant-specific overlay exists) +5. Applies config overlays in order (shared baseline first, tenant-specific on top): + - `configs//` (shared baseline, applied first if present) + - `configs/-/` (tenant-specific, applied on top if present, overrides shared) 6. Uploads resulting `*-rootfs.ext4` artifacts to S3. diff --git a/scripts/docker-to-rootfs.sh b/scripts/docker-to-rootfs.sh index 8ea19f7..04dc4ce 100755 --- a/scripts/docker-to-rootfs.sh +++ b/scripts/docker-to-rootfs.sh @@ -4,10 +4,11 @@ # # Usage: ./scripts/docker-to-rootfs.sh [size_mb] [overlay_dir] [fc_init_bin] # -# overlay_dir Optional directory whose contents are copied into the rootfs, -# mirroring the guest filesystem layout. For example, placing a -# file at overlay_dir/usr/share/elasticsearch/config/elasticsearch.yml -# overwrites that path in the rootfs. +# overlay_dir Optional colon-separated list of directories whose contents are +# copied into the rootfs in order, mirroring the guest filesystem +# layout. Later directories override earlier ones, so pass the shared +# baseline first and tenant-specific overlay second. For example: +# configs/elasticsearch:configs/tenant-1-elasticsearch # fc_init_bin Optional path to a prebuilt linux/arm64 fc-init binary. # If omitted, the script tries: # 1) FC_INIT_BIN env var @@ -164,10 +165,16 @@ RUNTIME_WRITABLE_PATHS_JSON="$(jq -cn \ | unique ')" -# Apply config overlay if provided. -if [ -n "$OVERLAY_DIR" ] && [ -d "$OVERLAY_DIR" ]; then - echo "==> Applying config overlay from $OVERLAY_DIR" - cp -r "$OVERLAY_DIR/." "$ROOTFS/" +# Apply config overlays in order. OVERLAY_DIR may be a colon-separated list; +# later entries override earlier ones (shared baseline first, tenant-specific second). +if [ -n "$OVERLAY_DIR" ]; then + IFS=: read -ra overlay_dirs <<< "$OVERLAY_DIR" + for dir in "${overlay_dirs[@]}"; do + [ -n "$dir" ] || continue + [ -d "$dir" ] || continue + echo "==> Applying config overlay from $dir" + cp -r "$dir/." "$ROOTFS/" + done fi # Some upstream images carry runtime-generated files that should not be baked From cf584b6a81f4ed0d53b82f3b1191aca2e5b4e9c4 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Tue, 16 Jun 2026 20:10:22 +0200 Subject: [PATCH 3/5] Optimize CI --- .github/workflows/build-images.yaml | 85 +++----------- Makefile | 7 ++ scripts/build-images.sh | 172 ++++++++++++++++++++++++++++ scripts/push-images.sh | 18 +++ 4 files changed, 210 insertions(+), 72 deletions(-) create mode 100644 Makefile create mode 100644 scripts/build-images.sh create mode 100644 scripts/push-images.sh diff --git a/.github/workflows/build-images.yaml b/.github/workflows/build-images.yaml index 885350f..72fe16f 100644 --- a/.github/workflows/build-images.yaml +++ b/.github/workflows/build-images.yaml @@ -7,6 +7,15 @@ on: - "tenants/**" - "configs/**" - "scripts/**" + - "Makefile" + - ".github/workflows/build-images.yaml" + pull_request: + paths: + - "tenants/**" + - "configs/**" + - "scripts/**" + - "Makefile" + - ".github/workflows/build-images.yaml" jobs: build: @@ -30,85 +39,17 @@ jobs: sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_arm64 sudo chmod +x /usr/local/bin/yq - - name: Resolve fc-init + - name: Build per-tenant rootfs images env: FC_INIT_VERSION: ${{ vars.FC_INIT_VERSION }} FIREWORK_GITHUB_TOKEN: ${{ secrets.FIREWORK_GITHUB_TOKEN }} - run: | - set -euo pipefail - mkdir -p .cache/bin - - if [ -n "${FC_INIT_VERSION:-}" ]; then - tag="${FC_INIT_VERSION#v}" - tag="v${tag}" - url="https://github.com/artemnikitin/firework/releases/download/${tag}/fc-init-linux-arm64" - echo "Downloading fc-init from release ${tag}" - curl -fsSL "$url" -o .cache/bin/fc-init - chmod +x .cache/bin/fc-init - else - echo "FC_INIT_VERSION not set; building fc-init from firework@main" - if [ -n "${FIREWORK_GITHUB_TOKEN:-}" ]; then - git config --global url."https://x-access-token:${FIREWORK_GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/" - export GOPRIVATE=github.com/artemnikitin/* - fi - - if ! GOBIN="$PWD/.cache/bin" GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \ - go install github.com/artemnikitin/firework/cmd/fc-init@main; then - echo "::warning::Failed to build fc-init from firework@main. Falling back to bundled source." - GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \ - go build -ldflags "-s -w" -o .cache/bin/fc-init ./scripts/fc-init/main.go - fi - fi - - - name: Build per-tenant rootfs images - run: | - chmod +x scripts/docker-to-rootfs.sh - - for tenant_dir in tenants/*/; do - [ -d "$tenant_dir" ] || continue - tenant_id="$(basename "$tenant_dir")" - echo "::group::Tenant: $tenant_id" - - for svc_file in "${tenant_dir}"*.yaml "${tenant_dir}"*.yml; do - [ -f "$svc_file" ] || continue - base_name="$(basename "${svc_file%.*}")" # e.g. "kibana" - - source_image="$(yq '.source_image // ""' "$svc_file")" - if [ -z "$source_image" ]; then - echo "Skipping ${tenant_id}-${base_name} — no source_image" - continue - fi - - size_mb="$(yq '.rootfs_size_mb // 512' "$svc_file")" - output="${tenant_id}-${base_name}-rootfs.ext4" - - # Config overlays: shared baseline applied first, tenant-specific on top. - # Pass as a colon-separated list so docker-to-rootfs.sh applies them in order. - overlay_arg="" - if [ -d "configs/${base_name}" ] && [ -d "configs/${tenant_id}-${base_name}" ]; then - overlay_arg="configs/${base_name}:configs/${tenant_id}-${base_name}" - elif [ -d "configs/${tenant_id}-${base_name}" ]; then - overlay_arg="configs/${tenant_id}-${base_name}" - elif [ -d "configs/${base_name}" ]; then - overlay_arg="configs/${base_name}" - fi - - echo "Building $output from $source_image" - ./scripts/docker-to-rootfs.sh "$source_image" "$output" "$size_mb" \ - "${overlay_arg:-}" ".cache/bin/fc-init" - done - echo "::endgroup::" - done + run: make build - name: Upload images to S3 + if: github.event_name == 'push' && github.ref == 'refs/heads/main' env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_REGION: ${{ vars.AWS_REGION }} S3_IMAGES_BUCKET: ${{ vars.S3_IMAGES_BUCKET }} - run: | - for ext4 in *-rootfs.ext4; do - [ -f "$ext4" ] || continue - echo "Uploading $ext4 to s3://${S3_IMAGES_BUCKET}/${ext4}" - aws s3 cp "$ext4" "s3://${S3_IMAGES_BUCKET}/${ext4}" - done + run: make push diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..43d1f60 --- /dev/null +++ b/Makefile @@ -0,0 +1,7 @@ +.PHONY: build push + +build: + bash ./scripts/build-images.sh + +push: + bash ./scripts/push-images.sh diff --git a/scripts/build-images.sh b/scripts/build-images.sh new file mode 100644 index 0000000..7505466 --- /dev/null +++ b/scripts/build-images.sh @@ -0,0 +1,172 @@ +#!/usr/bin/env bash + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +cd "$REPO_ROOT" + +CACHE_BIN_DIR="${CACHE_BIN_DIR:-.cache/bin}" +if [[ "$CACHE_BIN_DIR" != /* ]]; then + CACHE_BIN_DIR="$REPO_ROOT/$CACHE_BIN_DIR" +fi + +FC_INIT_BIN="${FC_INIT_BIN:-$CACHE_BIN_DIR/fc-init}" +if [[ "$FC_INIT_BIN" != /* ]]; then + FC_INIT_BIN="$REPO_ROOT/$FC_INIT_BIN" +fi + +configure_tool_paths() { + if command -v mkfs.ext4 >/dev/null 2>&1; then + return + fi + + if command -v brew >/dev/null 2>&1; then + local e2fsprogs_prefix + e2fsprogs_prefix="$(brew --prefix e2fsprogs 2>/dev/null || true)" + if [ -n "$e2fsprogs_prefix" ]; then + export PATH="${e2fsprogs_prefix}/sbin:${e2fsprogs_prefix}/bin:${PATH}" + fi + fi +} + +install_fc_init_from_main() { + local go_path="$REPO_ROOT/.cache/go" + local installed_bin + + mkdir -p "$go_path" "$(dirname "$FC_INIT_BIN")" + + GOPATH="$go_path" GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \ + go install github.com/artemnikitin/firework/cmd/fc-init@main + + installed_bin="$go_path/bin/fc-init" + if [ ! -f "$installed_bin" ]; then + installed_bin="$go_path/bin/linux_arm64/fc-init" + fi + + if [ ! -f "$installed_bin" ]; then + echo "ERROR: go install completed, but fc-init was not found in $go_path/bin" >&2 + return 1 + fi + + cp "$installed_bin" "$FC_INIT_BIN" + chmod +x "$FC_INIT_BIN" +} + +resolve_fc_init() { + mkdir -p "$CACHE_BIN_DIR" + + if [ -n "${FC_INIT_VERSION:-}" ]; then + local tag + tag="${FC_INIT_VERSION#v}" + tag="v${tag}" + + local url + url="https://github.com/artemnikitin/firework/releases/download/${tag}/fc-init-linux-arm64" + echo "Downloading fc-init from release ${tag}" + curl -fsSL "$url" -o "$FC_INIT_BIN" + chmod +x "$FC_INIT_BIN" + return + fi + + echo "FC_INIT_VERSION not set; building fc-init from firework@main" + if [ -n "${FIREWORK_GITHUB_TOKEN:-}" ]; then + export GOPRIVATE='github.com/artemnikitin/*' + export GIT_CONFIG_COUNT=1 + export GIT_CONFIG_KEY_0="url.https://x-access-token:${FIREWORK_GITHUB_TOKEN}@github.com/.insteadOf" + export GIT_CONFIG_VALUE_0="https://github.com/" + fi + + if ! install_fc_init_from_main; then + echo "::warning::Failed to build fc-init from firework@main. Falling back to bundled source." + GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \ + go build -ldflags "-s -w" -o "$FC_INIT_BIN" ./scripts/fc-init/main.go + fi +} + +overlay_arg_for() { + local tenant_id="$1" + local base_name="$2" + + if [ -d "configs/${base_name}" ] && [ -d "configs/${tenant_id}-${base_name}" ]; then + printf '%s\n' "configs/${base_name}:configs/${tenant_id}-${base_name}" + elif [ -d "configs/${tenant_id}-${base_name}" ]; then + printf '%s\n' "configs/${tenant_id}-${base_name}" + elif [ -d "configs/${base_name}" ]; then + printf '%s\n' "configs/${base_name}" + fi +} + +yaml_value() { + local file="$1" + local key="$2" + local default_value="$3" + + if command -v yq >/dev/null 2>&1; then + yq ".${key} // \"${default_value}\"" "$file" + return + fi + + if command -v ruby >/dev/null 2>&1; then + ruby -ryaml -e ' + file, key, default_value = ARGV + data = YAML.load_file(file) || {} + value = data[key] + print(value.nil? ? default_value : value) + ' "$file" "$key" "$default_value" + return + fi + + awk -v key="$key" -v default_value="$default_value" ' + BEGIN { value = default_value } + $0 ~ "^[[:space:]]*" key ":" { + sub("^[[:space:]]*" key ":[[:space:]]*", "") + gsub(/^"|"$/, "") + value = $0 + exit + } + END { print value } + ' "$file" +} + +build_images() { + for tenant_dir in tenants/*/; do + [ -d "$tenant_dir" ] || continue + local tenant_id + tenant_id="$(basename "$tenant_dir")" + echo "::group::Tenant: $tenant_id" + + for svc_file in "${tenant_dir}"*.yaml "${tenant_dir}"*.yml; do + [ -f "$svc_file" ] || continue + + local base_name + base_name="$(basename "${svc_file%.*}")" + + local source_image + source_image="$(yaml_value "$svc_file" source_image "")" + if [ -z "$source_image" ]; then + echo "Skipping ${tenant_id}-${base_name} - no source_image" + continue + fi + + local size_mb + size_mb="$(yaml_value "$svc_file" rootfs_size_mb 512)" + + local output + output="${tenant_id}-${base_name}-rootfs.ext4" + + local overlay_arg + overlay_arg="$(overlay_arg_for "$tenant_id" "$base_name")" + + echo "Building $output from $source_image" + bash ./scripts/docker-to-rootfs.sh "$source_image" "$output" "$size_mb" \ + "${overlay_arg:-}" "$FC_INIT_BIN" + done + + echo "::endgroup::" + done +} + +configure_tool_paths +resolve_fc_init +build_images diff --git a/scripts/push-images.sh b/scripts/push-images.sh new file mode 100644 index 0000000..8d079c9 --- /dev/null +++ b/scripts/push-images.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +cd "$REPO_ROOT" + +if [ -z "${S3_IMAGES_BUCKET:-}" ]; then + echo "ERROR: S3_IMAGES_BUCKET is required" >&2 + exit 1 +fi + +for ext4 in *-rootfs.ext4; do + [ -f "$ext4" ] || continue + echo "Uploading $ext4 to s3://${S3_IMAGES_BUCKET}/${ext4}" + aws s3 cp "$ext4" "s3://${S3_IMAGES_BUCKET}/${ext4}" +done From 5661d4ee11eb602c35dda42e5665494c1aee3396 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Tue, 16 Jun 2026 20:10:25 +0200 Subject: [PATCH 4/5] Update AGENTS.md --- AGENTS.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index f87c6e9..c82f293 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,9 +7,12 @@ This is the example GitOps input repo for Firework. It defines tenant service YA ## Layout - `defaults.yaml`: global service defaults consumed by Firework enricher; changes here affect every service on the next enricher run. +- `Makefile`: image pipeline entrypoints used by CI (`build` and `push`). - `tenants//.yaml`: tenant service specs. - `configs//` and `configs/-/`: rootfs overlays; tenant-specific overlays take precedence. +- `scripts/build-images.sh`: resolves `fc-init` and builds all tenant rootfs images. - `scripts/docker-to-rootfs.sh`: converts Docker images into ext4 rootfs images. +- `scripts/push-images.sh`: uploads generated rootfs images to S3. - `scripts/fc-init/`: fallback bundled `fc-init` source for CI. ## Conventions @@ -27,6 +30,6 @@ For image pipeline changes, validate: - `shellcheck scripts/docker-to-rootfs.sh` - A targeted local rootfs build when Docker, `jq`, `mkfs.ext4`, and a linux/arm64 `fc-init` are available. -For CI-equivalent validation, refer to `.github/workflows/build-images.yaml`, but skip the step with uploading files to S3. +For CI-equivalent validation, run `make build`, but skip `make push`. Do not upload to S3 or run cloud-mutating commands unless explicitly requested. From c1d1f75bf8e3f2cd27a0f0dbaded19b68d0b7dc2 Mon Sep 17 00:00:00 2001 From: Artem Nikitin Date: Tue, 16 Jun 2026 20:10:27 +0200 Subject: [PATCH 5/5] Update README.md --- README.md | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 5960363..f1ae039 100644 --- a/README.md +++ b/README.md @@ -36,13 +36,24 @@ flowchart LR ## CI Image Pipeline -The `build-images` workflow does the following on relevant pushes: - -1. Resolves `fc-init` (release asset, `go install`, or bundled fallback build). -2. Iterates over `tenants/*/*.yaml`. -3. Reads `source_image` and optional `rootfs_size_mb` from each tenant file. -4. Builds `--rootfs.ext4` via `scripts/docker-to-rootfs.sh`. -5. Applies config overlays in order (shared baseline first, tenant-specific on top): +The `build-images` workflow installs CI dependencies, then delegates image work to +the Makefile. The Makefile is a thin entrypoint that calls the shell scripts in +`scripts/`: + +On pull requests, CI runs `make build` only. On pushes to `main`, CI runs both +`make build` and `make push`. + +1. `make build` calls `scripts/build-images.sh`. +2. `scripts/build-images.sh` resolves `fc-init` (release asset, `go install`, + or bundled fallback build). +3. `scripts/build-images.sh` iterates over `tenants/*/*.yaml`. +4. `scripts/build-images.sh` reads `source_image` and optional `rootfs_size_mb` + from each tenant file. +5. `scripts/build-images.sh` creates `--rootfs.ext4` via + `scripts/docker-to-rootfs.sh`. +6. `scripts/build-images.sh` applies config overlays in order (shared baseline + first, tenant-specific on top): - `configs//` (shared baseline, applied first if present) - `configs/-/` (tenant-specific, applied on top if present, overrides shared) -6. Uploads resulting `*-rootfs.ext4` artifacts to S3. +7. `make push` calls `scripts/push-images.sh` to upload resulting + `*-rootfs.ext4` artifacts to S3.