Multi-stage Backstage template chain (VM → Ansible → Flux) in backstage-template-execution

[patrick-hermann-sva]

Goal

Extend the backstage-template-execution workflow to chain multiple Backstage scaffolder templates in one Dapr workflow instance, with three target shapes:

• a) create-terraform-vm only
• b) create-terraform-vm → ansible-provisioning
• c) create-terraform-vm → ansible-provisioning → flux-bootstrap

A single Go workflow handles all three; the shape is selected by the input file (stages: [...]).

Design

Input schema

{
  "backstageURL": "...",
  "stages": [
    { "name": "vm",      "templateRef": "...", "values": {...},
      "watch": { "kind": "pr", "workflowFile": "pr-vm-deploy.yaml",
                 "branch": "...", "merge": {"enabled": true} },
      "exports": { "vm_ip": "fromGithubJobOutput:deploy.vm_ip" } },
    { "name": "ansible", "templateRef": "...",
      "values": { "targets": ["${stages.vm.vm_ip}"], ... },
      "watch": { "kind": "pr", "workflowFile": "pr-ansible.yaml",
                 "branch": "...", "merge": {"enabled": true} } },
    { "name": "flux",    "templateRef": "...",
      "values": { "kube_config": "secrets/kubeconfigs/<name>.yaml", ... },
      "watch": { "kind": "dispatch", "workflowFile": "dispatch-flux-bootstrap.yaml" } }
  ]
}

Trigger kinds

• pr — current behavior. Backstage opens a PR; we poll runs filtered by head branch; optional auto-merge.
• dispatch — Backstage calls github:actions:dispatch. No PR, no merge. Run identification: time-based filter on event=workflow_dispatch + created>=<dispatchedAt> (strategy "a"). Upgrade to a correlation_id workflow input (strategy "b") if races bite.

Data hand-off

• Stage 1 → 2 (VM IP): pr-vm-deploy.yaml exposes vm_ip as a job output. New activity FetchGitHubJobOutputs reads it; resolver replaces ${stages.vm.vm_ip} in stage 2's values.
• Stage 2 → 3 (kubeconfig path): deterministic from vm_name — no runtime fetch, just a literal secrets/kubeconfigs/<name>.yaml in stage 3's values.

Code changes (main.go, ~150 LOC)

• Replace top-level templateRef/values/watch with Stages []Stage.
• Lift current scaffolder→poll→watch→merge body into runStage(...).
• Outer loop over stages; export map keyed <stageName>.<key>.
• resolveValues(values, exports) — recursive ${stages.X.Y} replacement.
• FetchGitHubJobOutputs activity (gated on exports != nil).
• watch.kind discriminator: pr keeps existing logic; dispatch uses time-window filter and skips merge.
• Hard-cut input shape (no compat layer for the old single-template format).

External changes

• stuttgart-things/stuttgart-things — pr-vm-deploy.yaml: upload stage-outputs artifact containing outputs.json with vm_ip (and any other stage-2 inputs). Original plan said "job output", corrected — see comment on this issue.

Plan

• 0. Baseline real run: single-stage create-terraform-vm with current code (no refactor) to validate tokens/network/PR-watch/merge end-to-end. Done 2026-04-25, ~28 min, PR #2092 merged in stuttgart-things/stuttgart-things.
• 1. Refactor main.go to stages: [...] + watch.kind discriminator, strategy (a) for dispatch correlation. Done — commit 15b6d63 on feat/multi-stage-backstage-chain.
• 1b. Re-run case (a) against the refactored code. Skipped full E2E in favor of unit tests (4 pass: interpolation, type preservation, nested resolveValues, no-exports passthrough) + live worker boot + structural smoke. Burning a real VM cycle for this would have been wasteful.
• 2. Add stage-outputs artifact upload to pr-vm-deploy.yaml (separate PR in stuttgart-things/stuttgart-things); implement FetchStageExports activity to download + parse it. Done — stuttgart-things/blueprints v1.83.0 writes outputs.json; stuttgart-things/stuttgart-things workflow change merged; activity verified end-to-end on 2026-04-26 ({"vm_ips":["10.31.101.109"]} retrieved + injected into stage 2).
• 3. Verify case (b) VM + ansible. Framework parts validated end-to-end on 2026-04-26 (instance run-1777182898): blueprint outputs.json, artifact upload + download, FetchStageExports, type-aware ${stages.vm.vm_ips} list passthrough into Backstage's array-typed targets, stage 1 PR auto-merge (#2096), stage 2 PR open + GH run watch + failure detected (no auto-merge on conclusion=failure). Underlying ansible playbook fails at kind create cluster with permission denied on /var/run/docker.sock — newly-added docker group not picked up by same SSH session. Out of scope for this issue (lives in stuttgart-things/ansible roles); 219/220 ansible tasks succeeded.
• 4. Verify case (c) — flux-bootstrap with kind: dispatch watch (the only framework piece still untested). Will use action: check against an existing cluster to keep it lab-light.
• 5. If concurrency becomes a problem: upgrade dispatch correlation to strategy (b) (workflow-input correlation_id + templated run-name).

Out of scope

• Replacing the in-repo SOPS kubeconfig pattern with a runtime kubeconfig fetch.
• Per-stage retry/backoff policy (today: stop on first failure).
• Fixing the kind create cluster permission failure surfaced in step 3 — it's a stuttgart-things ansible playbook ordering bug (docker install adds user to docker group, but the same SSH session doesn't see it). Track separately.

[patrick-hermann-sva]

Step 0 complete (baseline single-stage VM run)

• Instance: run-1777146004
• Backstage task: 5bbc4aca-7047-4403-a7ba-7870d05b9d57
• GH run: https://github.com/stuttgart-things/stuttgart-things/actions/runs/24939040429 (success)
• Merged: https://github.com/stuttgart-things/stuttgart-things/pull/2092 (c0f33bc, squash)
• VM: dapr-vm-20260425 in LabUL
• End-to-end duration: ~28 min (Dagger Deploy VMs with Dagger (with retry) step alone took ~24 min — slow but not stuck)

Drift fix needed before any future run: input.json had stale enums for ansible_collections (versions) and sthings_collections (URL); patched to match the live template enums in input.json.

Tokens, network, PR-watch logic, and auto-merge all verified working. Proceeding to step 1 (refactor to stages: [...]).

[patrick-hermann-sva]

Step 1 done — multi-stage refactor.

Design correction: the original plan said pr-vm-deploy.yaml would expose vm_ip as a job output and we'd read it via FetchGitHubJobOutputs. GitHub REST does not expose jobs.<job>.outputs externally — they're only readable inside the same workflow via needs. Switching the planned transport to artifact-based: each watched workflow uploads an artifact (e.g. stage-outputs containing outputs.json with a flat map[string]string); the FetchStageExports activity downloads + parses it. The exports syntax becomes { "<key>": "<artifactName>.<jsonKey>" }.

Code changes (workflows/backstage-template-execution/main.go):

• Input.Stages []Stage (hard cut from old single-template shape).
• Each stage has name, templateRef, values, dryRun, watch, exports.
• StageWatch.Kind discriminator: pr (existing logic) or dispatch (new).
• dispatch correlation: time-window filter (strategy "a") — CallScaffolder records dispatchedAt = time.Now().UTC(), FetchGitHubRun returns the newest workflow_dispatch run created on/after that timestamp (with 30 s clock-skew buffer).
• Outer per-stage loop in BackstageTemplateWorkflow; helper watchGHRun extracted.
• resolveValues walks stage values and replaces ${stages.<name>.<key>} from the cumulative exports map.
• FetchStageExports activity registered but currently returns not implemented — wiring deferred until step 2 (artifact format + GH workflow change).

Other changes:

• input.json rewritten as single-stage stages: [...] (case (a) shape) — same VM payload as the step-0 baseline.
• run.sh banner now lists stages.

Smoke check: go build and go vet pass; binary boots and registers all 5 activities cleanly.

Worker note: the existing dev worker (started for step 0) is running the OLD code. To pick up the refactor it needs a restart (Ctrl+C in shell 1, then re-run the dapr run -- go run . one-liner).

[patrick-hermann-sva]

Step 2 — Dapr-side implementation done (commit 15b6d63 on feat/multi-stage-backstage-chain).

FetchStageExports activity wired to GitHub artifacts:

1. List artifacts on the watched run via REST.
2. For each exports expression <artifactName>.<jsonKey>: download the named artifact, unzip in memory, read outputs.json (flat JSON object), pull the requested key.
3. Group exports by artifact name → each artifact downloaded at most once even if many keys are pulled.
4. Errors carry context: includes available artifact names when a lookup fails.

Resolver type rules (designed to support the multi-VM IP case):

• A value that is exactly ${stages.X.Y} → replaced with the raw exported value, preserving its native JSON type. Lets a list export populate a list-typed Backstage parameter directly.
• A placeholder embedded in literal text → fmt %v stringification.

This means the consumer side has zero awareness of whether an export is a string or a list — the resolver hands it through correctly. Confirmed by 4 unit tests.

Producer side (out of this repo) — change needed in stuttgart-things/blueprints/vm:
local.go:194-203 already calls OutputTerraformRunWithCreds (i.e. terraform output --json) and stores it in tfOutput, then parses IPs via ParseIPsFromTfOutput(tfOutput) for ansible inventory. Right after the existing WithNewFile("inventory.yaml", ...) (~line 220), add:

ips, _ := ParseIPsFromTfOutput(tfOutput)
outputs := map[string]any{"vm_ips": ips}
outputsBytes, _ := json.Marshal(outputs)
terraformDirResult = terraformDirResult.WithNewFile("outputs.json", string(outputsBytes))

…and in stuttgart-things/.github/workflows/pr-vm-deploy.yaml's Deploy-VMs job (after the deploy step succeeds):

- name: Upload stage outputs
  if: hashFiles('output/outputs.json') != ''
  uses: actions/upload-artifact@v4
  with:
    name: stage-outputs
    path: output/outputs.json
    retention-days: 7

The if guard makes the step a no-op for older blueprint versions that don't emit outputs.json — backwards-safe.

Single-VM case ends up as {"vm_ips":["10.1.2.3"]} — consumer that wants a single IP can index in their template, but typically they'll want the whole list anyway (ansible targets is an array).

Decisions baked in:

• Emit vm_ips (list, always) — no vm_ip shortcut for the single-VM case.
• No indexing syntax in placeholders (no ${stages.vm.vm_ips[0]}); add only if it bites.
• outputs.json is map[string]any — same mechanism extends to future non-IP outputs without further design.