Skip to content

Commit d7be4bf

Browse files
artemnikitinclaude
andauthored
Update CI (#7)
## Summary - Apply shared and tenant-specific config overlays in deterministic order: `configs/<service>/` first, then `configs/<tenant>-<service>/` on top. - Move the image pipeline logic out of GitHub Actions into reusable shell scripts with a thin `Makefile` entrypoint. - Run `make build` on pull requests, and run both `make build` and `make push` only for pushes to `main`. - Update README and AGENTS guidance to match the new pipeline and validation flow. ## Details - `scripts/build-images.sh` resolves `fc-init`, reads tenant service YAML, computes overlay arguments, and builds all tenant rootfs images. - `scripts/push-images.sh` uploads generated `*-rootfs.ext4` artifacts to S3. - `scripts/docker-to-rootfs.sh` now accepts colon-separated overlay directories and applies them in order. - The workflow now delegates build/upload behavior to `make build` and `make push` instead of carrying the shell logic inline. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent dca8cab commit d7be4bf

7 files changed

Lines changed: 250 additions & 88 deletions

File tree

.github/workflows/build-images.yaml

Lines changed: 13 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,15 @@ on:
77
- "tenants/**"
88
- "configs/**"
99
- "scripts/**"
10+
- "Makefile"
11+
- ".github/workflows/build-images.yaml"
12+
pull_request:
13+
paths:
14+
- "tenants/**"
15+
- "configs/**"
16+
- "scripts/**"
17+
- "Makefile"
18+
- ".github/workflows/build-images.yaml"
1019

1120
jobs:
1221
build:
@@ -30,82 +39,17 @@ jobs:
3039
sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_arm64
3140
sudo chmod +x /usr/local/bin/yq
3241
33-
- name: Resolve fc-init
42+
- name: Build per-tenant rootfs images
3443
env:
3544
FC_INIT_VERSION: ${{ vars.FC_INIT_VERSION }}
3645
FIREWORK_GITHUB_TOKEN: ${{ secrets.FIREWORK_GITHUB_TOKEN }}
37-
run: |
38-
set -euo pipefail
39-
mkdir -p .cache/bin
40-
41-
if [ -n "${FC_INIT_VERSION:-}" ]; then
42-
tag="${FC_INIT_VERSION#v}"
43-
tag="v${tag}"
44-
url="https://github.com/artemnikitin/firework/releases/download/${tag}/fc-init-linux-arm64"
45-
echo "Downloading fc-init from release ${tag}"
46-
curl -fsSL "$url" -o .cache/bin/fc-init
47-
chmod +x .cache/bin/fc-init
48-
else
49-
echo "FC_INIT_VERSION not set; building fc-init from firework@main"
50-
if [ -n "${FIREWORK_GITHUB_TOKEN:-}" ]; then
51-
git config --global url."https://x-access-token:${FIREWORK_GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/"
52-
export GOPRIVATE=github.com/artemnikitin/*
53-
fi
54-
55-
if ! GOBIN="$PWD/.cache/bin" GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \
56-
go install github.com/artemnikitin/firework/cmd/fc-init@main; then
57-
echo "::warning::Failed to build fc-init from firework@main. Falling back to bundled source."
58-
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \
59-
go build -ldflags "-s -w" -o .cache/bin/fc-init ./scripts/fc-init/main.go
60-
fi
61-
fi
62-
63-
- name: Build per-tenant rootfs images
64-
run: |
65-
chmod +x scripts/docker-to-rootfs.sh
66-
67-
for tenant_dir in tenants/*/; do
68-
[ -d "$tenant_dir" ] || continue
69-
tenant_id="$(basename "$tenant_dir")"
70-
echo "::group::Tenant: $tenant_id"
71-
72-
for svc_file in "${tenant_dir}"*.yaml "${tenant_dir}"*.yml; do
73-
[ -f "$svc_file" ] || continue
74-
base_name="$(basename "${svc_file%.*}")" # e.g. "kibana"
75-
76-
source_image="$(yq '.source_image // ""' "$svc_file")"
77-
if [ -z "$source_image" ]; then
78-
echo "Skipping ${tenant_id}-${base_name} — no source_image"
79-
continue
80-
fi
81-
82-
size_mb="$(yq '.rootfs_size_mb // 512' "$svc_file")"
83-
output="${tenant_id}-${base_name}-rootfs.ext4"
84-
85-
# Config overlay: tenant-specific overlay takes precedence.
86-
overlay_arg=""
87-
if [ -d "configs/${tenant_id}-${base_name}" ]; then
88-
overlay_arg="configs/${tenant_id}-${base_name}"
89-
elif [ -d "configs/${base_name}" ]; then
90-
overlay_arg="configs/${base_name}"
91-
fi
92-
93-
echo "Building $output from $source_image"
94-
./scripts/docker-to-rootfs.sh "$source_image" "$output" "$size_mb" \
95-
"${overlay_arg:-}" ".cache/bin/fc-init"
96-
done
97-
echo "::endgroup::"
98-
done
46+
run: make build
9947

10048
- name: Upload images to S3
49+
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
10150
env:
10251
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
10352
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
10453
AWS_REGION: ${{ vars.AWS_REGION }}
10554
S3_IMAGES_BUCKET: ${{ vars.S3_IMAGES_BUCKET }}
106-
run: |
107-
for ext4 in *-rootfs.ext4; do
108-
[ -f "$ext4" ] || continue
109-
echo "Uploading $ext4 to s3://${S3_IMAGES_BUCKET}/${ext4}"
110-
aws s3 cp "$ext4" "s3://${S3_IMAGES_BUCKET}/${ext4}"
111-
done
55+
run: make push

AGENTS.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,12 @@ This is the example GitOps input repo for Firework. It defines tenant service YA
77
## Layout
88

99
- `defaults.yaml`: global service defaults consumed by Firework enricher; changes here affect every service on the next enricher run.
10+
- `Makefile`: image pipeline entrypoints used by CI (`build` and `push`).
1011
- `tenants/<tenant>/<service>.yaml`: tenant service specs.
1112
- `configs/<service>/` and `configs/<tenant>-<service>/`: rootfs overlays; tenant-specific overlays take precedence.
13+
- `scripts/build-images.sh`: resolves `fc-init` and builds all tenant rootfs images.
1214
- `scripts/docker-to-rootfs.sh`: converts Docker images into ext4 rootfs images.
15+
- `scripts/push-images.sh`: uploads generated rootfs images to S3.
1316
- `scripts/fc-init/`: fallback bundled `fc-init` source for CI.
1417

1518
## Conventions
@@ -27,6 +30,6 @@ For image pipeline changes, validate:
2730
- `shellcheck scripts/docker-to-rootfs.sh`
2831
- A targeted local rootfs build when Docker, `jq`, `mkfs.ext4`, and a linux/arm64 `fc-init` are available.
2932

30-
For CI-equivalent validation, refer to `.github/workflows/build-images.yaml`, but skip the step with uploading files to S3.
33+
For CI-equivalent validation, run `make build`, but skip `make push`.
3134

3235
Do not upload to S3 or run cloud-mutating commands unless explicitly requested.

Makefile

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.PHONY: build push
2+
3+
build:
4+
bash ./scripts/build-images.sh
5+
6+
push:
7+
bash ./scripts/push-images.sh

README.md

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -36,13 +36,24 @@ flowchart LR
3636

3737
## CI Image Pipeline
3838

39-
The `build-images` workflow does the following on relevant pushes:
40-
41-
1. Resolves `fc-init` (release asset, `go install`, or bundled fallback build).
42-
2. Iterates over `tenants/*/*.yaml`.
43-
3. Reads `source_image` and optional `rootfs_size_mb` from each tenant file.
44-
4. Builds `<tenant>-<service>-rootfs.ext4` via `scripts/docker-to-rootfs.sh`.
45-
5. Applies config overlays with precedence:
46-
- `configs/<tenant>-<service>/` (tenant-specific)
47-
- then `configs/<service>/` (shared)
48-
6. Uploads resulting `*-rootfs.ext4` artifacts to S3.
39+
The `build-images` workflow installs CI dependencies, then delegates image work to
40+
the Makefile. The Makefile is a thin entrypoint that calls the shell scripts in
41+
`scripts/`:
42+
43+
On pull requests, CI runs `make build` only. On pushes to `main`, CI runs both
44+
`make build` and `make push`.
45+
46+
1. `make build` calls `scripts/build-images.sh`.
47+
2. `scripts/build-images.sh` resolves `fc-init` (release asset, `go install`,
48+
or bundled fallback build).
49+
3. `scripts/build-images.sh` iterates over `tenants/*/*.yaml`.
50+
4. `scripts/build-images.sh` reads `source_image` and optional `rootfs_size_mb`
51+
from each tenant file.
52+
5. `scripts/build-images.sh` creates `<tenant>-<service>-rootfs.ext4` via
53+
`scripts/docker-to-rootfs.sh`.
54+
6. `scripts/build-images.sh` applies config overlays in order (shared baseline
55+
first, tenant-specific on top):
56+
- `configs/<service>/` (shared baseline, applied first if present)
57+
- `configs/<tenant>-<service>/` (tenant-specific, applied on top if present, overrides shared)
58+
7. `make push` calls `scripts/push-images.sh` to upload resulting
59+
`*-rootfs.ext4` artifacts to S3.

scripts/build-images.sh

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
7+
cd "$REPO_ROOT"
8+
9+
CACHE_BIN_DIR="${CACHE_BIN_DIR:-.cache/bin}"
10+
if [[ "$CACHE_BIN_DIR" != /* ]]; then
11+
CACHE_BIN_DIR="$REPO_ROOT/$CACHE_BIN_DIR"
12+
fi
13+
14+
FC_INIT_BIN="${FC_INIT_BIN:-$CACHE_BIN_DIR/fc-init}"
15+
if [[ "$FC_INIT_BIN" != /* ]]; then
16+
FC_INIT_BIN="$REPO_ROOT/$FC_INIT_BIN"
17+
fi
18+
19+
configure_tool_paths() {
20+
if command -v mkfs.ext4 >/dev/null 2>&1; then
21+
return
22+
fi
23+
24+
if command -v brew >/dev/null 2>&1; then
25+
local e2fsprogs_prefix
26+
e2fsprogs_prefix="$(brew --prefix e2fsprogs 2>/dev/null || true)"
27+
if [ -n "$e2fsprogs_prefix" ]; then
28+
export PATH="${e2fsprogs_prefix}/sbin:${e2fsprogs_prefix}/bin:${PATH}"
29+
fi
30+
fi
31+
}
32+
33+
install_fc_init_from_main() {
34+
local go_path="$REPO_ROOT/.cache/go"
35+
local installed_bin
36+
37+
mkdir -p "$go_path" "$(dirname "$FC_INIT_BIN")"
38+
39+
GOPATH="$go_path" GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \
40+
go install github.com/artemnikitin/firework/cmd/fc-init@main
41+
42+
installed_bin="$go_path/bin/fc-init"
43+
if [ ! -f "$installed_bin" ]; then
44+
installed_bin="$go_path/bin/linux_arm64/fc-init"
45+
fi
46+
47+
if [ ! -f "$installed_bin" ]; then
48+
echo "ERROR: go install completed, but fc-init was not found in $go_path/bin" >&2
49+
return 1
50+
fi
51+
52+
cp "$installed_bin" "$FC_INIT_BIN"
53+
chmod +x "$FC_INIT_BIN"
54+
}
55+
56+
resolve_fc_init() {
57+
mkdir -p "$CACHE_BIN_DIR"
58+
59+
if [ -n "${FC_INIT_VERSION:-}" ]; then
60+
local tag
61+
tag="${FC_INIT_VERSION#v}"
62+
tag="v${tag}"
63+
64+
local url
65+
url="https://github.com/artemnikitin/firework/releases/download/${tag}/fc-init-linux-arm64"
66+
echo "Downloading fc-init from release ${tag}"
67+
curl -fsSL "$url" -o "$FC_INIT_BIN"
68+
chmod +x "$FC_INIT_BIN"
69+
return
70+
fi
71+
72+
echo "FC_INIT_VERSION not set; building fc-init from firework@main"
73+
if [ -n "${FIREWORK_GITHUB_TOKEN:-}" ]; then
74+
export GOPRIVATE='github.com/artemnikitin/*'
75+
export GIT_CONFIG_COUNT=1
76+
export GIT_CONFIG_KEY_0="url.https://x-access-token:${FIREWORK_GITHUB_TOKEN}@github.com/.insteadOf"
77+
export GIT_CONFIG_VALUE_0="https://github.com/"
78+
fi
79+
80+
if ! install_fc_init_from_main; then
81+
echo "::warning::Failed to build fc-init from firework@main. Falling back to bundled source."
82+
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \
83+
go build -ldflags "-s -w" -o "$FC_INIT_BIN" ./scripts/fc-init/main.go
84+
fi
85+
}
86+
87+
overlay_arg_for() {
88+
local tenant_id="$1"
89+
local base_name="$2"
90+
91+
if [ -d "configs/${base_name}" ] && [ -d "configs/${tenant_id}-${base_name}" ]; then
92+
printf '%s\n' "configs/${base_name}:configs/${tenant_id}-${base_name}"
93+
elif [ -d "configs/${tenant_id}-${base_name}" ]; then
94+
printf '%s\n' "configs/${tenant_id}-${base_name}"
95+
elif [ -d "configs/${base_name}" ]; then
96+
printf '%s\n' "configs/${base_name}"
97+
fi
98+
}
99+
100+
yaml_value() {
101+
local file="$1"
102+
local key="$2"
103+
local default_value="$3"
104+
105+
if command -v yq >/dev/null 2>&1; then
106+
yq ".${key} // \"${default_value}\"" "$file"
107+
return
108+
fi
109+
110+
if command -v ruby >/dev/null 2>&1; then
111+
ruby -ryaml -e '
112+
file, key, default_value = ARGV
113+
data = YAML.load_file(file) || {}
114+
value = data[key]
115+
print(value.nil? ? default_value : value)
116+
' "$file" "$key" "$default_value"
117+
return
118+
fi
119+
120+
awk -v key="$key" -v default_value="$default_value" '
121+
BEGIN { value = default_value }
122+
$0 ~ "^[[:space:]]*" key ":" {
123+
sub("^[[:space:]]*" key ":[[:space:]]*", "")
124+
gsub(/^"|"$/, "")
125+
value = $0
126+
exit
127+
}
128+
END { print value }
129+
' "$file"
130+
}
131+
132+
build_images() {
133+
for tenant_dir in tenants/*/; do
134+
[ -d "$tenant_dir" ] || continue
135+
local tenant_id
136+
tenant_id="$(basename "$tenant_dir")"
137+
echo "::group::Tenant: $tenant_id"
138+
139+
for svc_file in "${tenant_dir}"*.yaml "${tenant_dir}"*.yml; do
140+
[ -f "$svc_file" ] || continue
141+
142+
local base_name
143+
base_name="$(basename "${svc_file%.*}")"
144+
145+
local source_image
146+
source_image="$(yaml_value "$svc_file" source_image "")"
147+
if [ -z "$source_image" ]; then
148+
echo "Skipping ${tenant_id}-${base_name} - no source_image"
149+
continue
150+
fi
151+
152+
local size_mb
153+
size_mb="$(yaml_value "$svc_file" rootfs_size_mb 512)"
154+
155+
local output
156+
output="${tenant_id}-${base_name}-rootfs.ext4"
157+
158+
local overlay_arg
159+
overlay_arg="$(overlay_arg_for "$tenant_id" "$base_name")"
160+
161+
echo "Building $output from $source_image"
162+
bash ./scripts/docker-to-rootfs.sh "$source_image" "$output" "$size_mb" \
163+
"${overlay_arg:-}" "$FC_INIT_BIN"
164+
done
165+
166+
echo "::endgroup::"
167+
done
168+
}
169+
170+
configure_tool_paths
171+
resolve_fc_init
172+
build_images

scripts/docker-to-rootfs.sh

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,11 @@
44
#
55
# Usage: ./scripts/docker-to-rootfs.sh <docker-image> <output.ext4> [size_mb] [overlay_dir] [fc_init_bin]
66
#
7-
# overlay_dir Optional directory whose contents are copied into the rootfs,
8-
# mirroring the guest filesystem layout. For example, placing a
9-
# file at overlay_dir/usr/share/elasticsearch/config/elasticsearch.yml
10-
# overwrites that path in the rootfs.
7+
# overlay_dir Optional colon-separated list of directories whose contents are
8+
# copied into the rootfs in order, mirroring the guest filesystem
9+
# layout. Later directories override earlier ones, so pass the shared
10+
# baseline first and tenant-specific overlay second. For example:
11+
# configs/elasticsearch:configs/tenant-1-elasticsearch
1112
# fc_init_bin Optional path to a prebuilt linux/arm64 fc-init binary.
1213
# If omitted, the script tries:
1314
# 1) FC_INIT_BIN env var
@@ -164,10 +165,16 @@ RUNTIME_WRITABLE_PATHS_JSON="$(jq -cn \
164165
| unique
165166
')"
166167
167-
# Apply config overlay if provided.
168-
if [ -n "$OVERLAY_DIR" ] && [ -d "$OVERLAY_DIR" ]; then
169-
echo "==> Applying config overlay from $OVERLAY_DIR"
170-
cp -r "$OVERLAY_DIR/." "$ROOTFS/"
168+
# Apply config overlays in order. OVERLAY_DIR may be a colon-separated list;
169+
# later entries override earlier ones (shared baseline first, tenant-specific second).
170+
if [ -n "$OVERLAY_DIR" ]; then
171+
IFS=: read -ra overlay_dirs <<< "$OVERLAY_DIR"
172+
for dir in "${overlay_dirs[@]}"; do
173+
[ -n "$dir" ] || continue
174+
[ -d "$dir" ] || continue
175+
echo "==> Applying config overlay from $dir"
176+
cp -r "$dir/." "$ROOTFS/"
177+
done
171178
fi
172179
173180
# Some upstream images carry runtime-generated files that should not be baked

scripts/push-images.sh

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
7+
cd "$REPO_ROOT"
8+
9+
if [ -z "${S3_IMAGES_BUCKET:-}" ]; then
10+
echo "ERROR: S3_IMAGES_BUCKET is required" >&2
11+
exit 1
12+
fi
13+
14+
for ext4 in *-rootfs.ext4; do
15+
[ -f "$ext4" ] || continue
16+
echo "Uploading $ext4 to s3://${S3_IMAGES_BUCKET}/${ext4}"
17+
aws s3 cp "$ext4" "s3://${S3_IMAGES_BUCKET}/${ext4}"
18+
done

0 commit comments

Comments
 (0)