fix(tenant-2): prepend http:// to ELASTICSEARCH_HOSTS in kibana config - #14
Closed
artemnikitin wants to merge 1 commit into
Closed
fix(tenant-2): prepend http:// to ELASTICSEARCH_HOSTS in kibana config#14artemnikitin wants to merge 1 commit into
artemnikitin wants to merge 1 commit into
Conversation
cross_node_links injects bare `host:port` (no protocol). Kibana 9
requires a full URI for elasticsearch.hosts. Add a tenant-2-kibana
overlay that prepends http:// to ${ELASTICSEARCH_HOSTS}.
Also remove the unsupported `protocol` field from the cross_node_links
spec — the controller ignores it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
artemnikitin
added a commit
that referenced
this pull request
Aug 8, 2026
…ntics (#28) Switches AWS image publishing to x86_64 and brings the whole repository in line with it. Refs [artemnikitin/firework-deployment-example#13](artemnikitin/firework-deployment-example#13) and [#14](artemnikitin/firework-deployment-example#14). ## Why `firework-deployment-example` switched the AWS data plane default from bare-metal Graviton (`c6g.metal`) to x86_64 instances using nested virtualization. GCP has always been x86_64. Both providers therefore consume **amd64** rootfs images. CI already built amd64. What it did not do was publish it to S3: the amd64 build resolved its S3 target from `S3_IMAGES_BUCKET_AMD64` only, that variable is not configured, so the amd64 build silently logged `Skipping S3 upload; no bucket configured` and exited 0. The only images ever uploaded to S3 were arm64. That combination is live right now and it takes the whole AWS deployment down. An x86_64 node running an arm64 rootfs boots its kernel and mounts the root filesystem fine, then panics executing init: ``` Run /sbin/fc-init as init process Kernel panic - not syncing: Requested init /sbin/fc-init failed (error -8). ``` `-8` is `ENOEXEC`. Firecracker then exits `exit_code=0`, so nothing on the host looks failed — nodes, agent, Traefik and ALB targets are all healthy — and it surfaces only as every tenant service failing its health check. ## What changed **Bucket resolution** (`.github/workflows/build-images.yaml`) — the legacy `S3_IMAGES_BUCKET` now means the amd64 S3 bucket, mirroring how `GCS_IMAGES_BUCKET` already means the amd64 GCS bucket. The amd64 build falls back to it; arm64 publishing becomes opt-in via `S3_IMAGES_BUCKET_ARM64`. With the repository variables exactly as they are today: | Build | S3 bucket | GCS bucket | | --- | --- | --- | | amd64 | `artemnikitin-firework-images` | `firework-images` | | arm64 | skipped (warns) | skipped (warns) | The amd64 images land in the **existing** bucket under the **same object names**, replacing the arm64 objects in place. No new bucket, no node IAM change, no `s3_images_bucket_id` repoint. **Explicit push backend** — `push-images.sh` inferred its backend from whichever bucket variable was set, checking GCS first. Until now exactly one was ever set per matrix leg, so the precedence never mattered. Resolving both buckets for the amd64 build exposed it: `make push-s3` would have run `gcloud storage cp` against the GCS bucket and left S3 untouched, so the change above would not actually have fixed anything. The script now takes an explicit `s3`/`gcs` argument; inference errors out when both buckets are set. Caught by codex in review — thank you. **Regression check** — `scripts/test-push-images.sh` runs the real Makefile targets with `aws` and `gcloud` stubbed and asserts the chosen backend for both-buckets-set, single-bucket, and missing-bucket cases. It fails against the previous implementation. Runs in `validate-config`. **Loud no-op** — a build that resolves no bucket at all now emits a `::warning::`. A silent skip is precisely what let this mismatch reach a running deployment. **Architecture-gated publish steps** — the GCP credentials step was already gated on the matrix arch having a configured bucket; the two upload steps were not. The arm64 leg was running an `Upload images to S3` step on every main build and receiving `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` for an upload that could only no-op. All three steps are now gated identically. **Local defaults** — `TARGET_PLATFORM` flipped to `linux/amd64` in `Makefile`, `scripts/build-images.sh`, and `scripts/docker-to-rootfs.sh`. Local pushes bypass the workflow's arch resolution, so leaving these at arm64 would let `make build && make push-s3` push arm64 images into what is now the amd64 bucket and reproduce the outage. **Docs** — `docs/ci-pipeline.md` rewritten for the new semantics with a resolution table, the backend contract, and an explicit migration note that `S3_IMAGES_BUCKET` changed meaning. `README.md` and `AGENTS.md` corrected: they claimed the repo publishes both architectures, when it builds both and publishes amd64 by default. ## Trade-off Anyone still on Graviton nodes must set `S3_IMAGES_BUCKET_ARM64` and point `s3_images_bucket_id` at it. Documented, and the build now warns rather than skipping silently. This is the right default because both provider data planes are x86_64. ## Validation - `shellcheck scripts/*.sh` clean - `bash scripts/test-push-images.sh` passes, and fails against the pre-fix script - workflow YAML parses; all three publish-step conditions verified - bucket resolution simulated against the real repository variables for both arches, including the arm64 opt-in path - repository swept for remaining arm64 defaults; none left Merging is enough to republish: this PR touches four `GLOBAL_PIPELINE_PATHS` entries, which forces a full rebuild of every tenant image rather than letting change detection skip them. Not yet done: the images in S3 are still arm64 until this merges and the build workflow runs on `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cross_node_linksinjects barehost:port(no scheme). Kibana 9 strictly requires a full URI (http://host:port) forelasticsearch.hosts. So tenant-2-kibana was gettingELASTICSEARCH_HOSTS=10.30.0.3:9200and logging[FATAL]: expected URI with scheme [http|https].Fix
Add a
configs/tenant-2-kibana/rootfs overlay that overrides the sharedkibana.ymlconfig with:The build script applies the shared
configs/kibana/overlay first, thenconfigs/tenant-2-kibana/on top, so tenant-1 is unaffected (it useslinkswhich already injectshttp://...).Also removes the
protocol: "http"field from thecross_node_linksspec — the controller ignores unknown fields and it was misleading.Test plan
tenant-2-kibana-rootfs.ext4and uploads to GCSkubectl rollout restart deployment/firework-events -n fireworkhttps://tenant-2.gcp.artemnikitin.com/returns 200