[bajtos]

Written by Claude.

Move staging to piri-0, piri-1 and piri-2 behind its one Ingot, as dev runs them, at piri-<n>.eu-central-3.staging.fil-forge.com.

The bare-metal host keeps one data disk, so each Piri gets a directory on it, /mnt/data/fil-one/data/piri-<n>, and a loopback port, 15100 to 15102. A volume per Piri needs the host reconfigured and is left for later.

The host Caddy snippet, the DNS records and postgres-init follow the same names. The host Caddy reload and the host Alloy edits are manual.

Migrating the staging node

This PR merges in the middle of the migration, after the node and central are reset and every Piri is onboarded. If it merged earlier, the reconcile timer would start three Piris with no keys and no onboarding. Dev moved the same way on 2026-10-16, in #170.

The node loses its Ingot and Piri databases and data. Central's staging stage loses its providers, tenants and Piri DIDs. These stay:

• OpenBao and the unseal token;
• every key and wallet;
• hilt's delegation to Ingot.

The existing Piri keeps its DID and funded wallet as piri-0. Its old on-chain data set faults and is left to fault, stranding about 0.9 USDFC of the payer's lockup. The node takes no uploads from step 1 to step 10, and the smoke test alerts in that window.

Commands run in four places:

• the staging host: ssh root@23.83.66.244, then cd /root/fil-one/infra-nodes. The login shell is Fish, and every command below works in it.
• an infra-central checkout on main, with the filone-sandbox AWS profile.
• an infra-nodes checkout, with the same profile.
• a fil-one checkout, with credentials for FilOne's AWS account.

Before the window

This procedure assumes two infra-central changes have landed:

• Promote the provision Lambda to staging infra-central#265 promotes the provision Lambda to staging with the reset-apps phase from Add the reset-apps phase to the provision Lambda infra-central#259. Step 2 needs that phase.
• Point Hilt's revocations at its own stage's Swarf infra-central#264 points Hilt's revocations at each stage's own Swarf. It reaches staging with the next apply of terraform/envs/staging/apps, and step 11's E2E cleanup needs it.

Then, before the window:

• Merge the open infra-central staging promotion, Promote dev's images to staging infra-central#247 or whatever replaced it. Its apply brings #264's Hilt settings to staging.
• After that, merge no other infra-central staging promotion until step 10. Each one applies terraform/envs/staging/apps, which sets every service's desired count back to 1 and can bring a service up while step 2 has it at zero. Pushes to infra-central's main apply only dev and can go ahead.
• Merge Promote dev's images to staging #176, the infra-nodes staging promotion, or whichever version of it is open. Staging pins piri sha256:1e91500 and ingot sha256:f629535, while dev runs the three-Piri layout on piri sha256:7feed0e and ingot sha256:1ef08c4. Promoting first makes staging's first three-Piri deploy run digests already proven on that layout. Hold any later promotion until step 10.
• Mark this PR ready: gh pr ready 171 --repo fil-forge/infra-nodes.
• Leave dev's image bumps running. They pin dev only, and step 1 stops staging's timer.
• Tell #filone-alerts that eu-central-3 is down for the window.

1. Reset the node (staging host)

git fetch origin
git checkout --detach origin/main
systemctl stop filone-reconcile.timer
scripts/host/reset-apps.sh

Check the plan it prints before you type reset-staging. It should list the containers filone-piri and filone-ingot, the databases piri and ingot, and data directories under /mnt/data/fil-one/data. The script ends with "apps reset; they stay stopped". Skip its "Next" line, which is for a plain reset.

2. Reset central (infra-central)

make reset-apps STAGE=staging

It asks for reset-staging. Emptying sprue's agent-message and delegation buckets takes minutes, and the script prints a running count while it does (fil-forge/infra-central#262). It ends with Done. once every service is back up. Skip its "Next" line too.

3. DNS (infra-nodes, on this branch)

git fetch origin
git checkout --detach origin/miroslav/fil-1503-staging
tofu -chdir=terraform/envs/staging/eu-central-3 init
tofu -chdir=terraform/envs/staging/eu-central-3 plan

The plan should create three A records, piri-0 to piri-2.eu-central-3.staging.fil-forge.com, all pointing at 23.83.66.244. It should destroy piri-0.staging.fil-forge.com and change nothing else. Then apply it and check that each name resolves:

tofu -chdir=terraform/envs/staging/eu-central-3 apply
dig +short piri-0.eu-central-3.staging.fil-forge.com
dig +short piri-1.eu-central-3.staging.fil-forge.com
dig +short piri-2.eu-central-3.staging.fil-forge.com

4. Keys (staging host, on this branch)

git fetch origin
git checkout --detach origin/miroslav/fil-1503-staging
install -d -m 0755 /mnt/data/fil-one/data/piri-0 /mnt/data/fil-one/data/piri-1 /mnt/data/fil-one/data/piri-2
scripts/host/keygen.sh

The host was bootstrapped with one piri directory, so the install -d creates the three new ones. keygen.sh must report that it copied OpenBao path piri to piri-0 and generated piri-1 and piri-2. If it generates a new piri-0 instead, stop: piri-0 would get a new DID and an unfunded wallet.

5. Host Caddy (staging host)

The host owns Caddy. Its /root/storacha/caddy/Caddyfile imports this node's snippet from the checkout. On this branch the snippet serves the three new Piri hostnames on ports 15100 to 15102 and no longer serves piri-0.staging.fil-forge.com. With the timer stopped, nothing reloads Caddy on its own, so reload it by hand.

First check that the import still points at the checkout:

grep -n 'infra-nodes/nodes' /root/storacha/caddy/Caddyfile

Then check that nothing else in the host's file names the old hostname:

grep -n 'piri-0.staging.fil-forge.com' /root/storacha/caddy/Caddyfile

Change any match from the second grep to the new hostnames, then validate and reload:

scripts/host/reload-staging-host-caddy.sh

The script validates the combined configuration before it reloads caddy-guppy, so a broken edit stops there rather than taking down the host's other sites. Caddy then requests certificates for the three new names, which needs step 3's DNS to resolve. Each name should serve the status document over HTTPS with HTTP 200, even before its Piri runs:

curl -sS -o /dev/null -w '%{http_code}\n' https://piri-0.eu-central-3.staging.fil-forge.com/.well-known/filone-node-status.json

Repeat for piri-1 and piri-2. A TLS error means Caddy has no certificate for that name yet.

6. Host Alloy (staging host)

Make the three edits to /etc/alloy/config.alloy that the RUNBOOK's "The eu-central-3 staging appliance on Servers.com" shows:

• Docker logs: the rule that sets instance from filone-apps;(piri-[0-9]+);
• prometheus.relabel "host_caddy": the host regex piri-[0-9]+\.eu-central-3\.staging\.fil-forge\.com;
• prometheus.relabel "filone_forge": the rule that maps Piri's service.instance.id to instance.

Then validate and restart:

alloy validate /etc/alloy/config.alloy
systemctl restart alloy

Restart rather than reload: Alloy v1.17 keeps the Docker log tailers, and their old labels, across a reload.

7. Onboarding and wallets (staging host, then infra-central)

scripts/host/onboarding-request.sh

It prints three make onboard-appliance STAGE=staging REGION=eu-central-3 … commands and the three owner wallet addresses. Run the three commands in infra-central. Each run logs "returning the delegation issued earlier". All three write the same ingot-proof.txt, which is hilt's delegation to the region's one Ingot. The node still holds it, so there is nothing to store with store-hilt-proof.sh.

Each Piri's first start registers it on chain and spends 5 tFIL from its own wallet. Before step 10, look up each address on a Calibration block explorer. Any wallet holding less than 6 tFIL needs funding from a faucet.

8. FilOne's tenants (fil-one)

Run the dry run first:

node bin/reset-region-provisioning.ts --stage staging --region eu-central-3 --dry-run

Then the real run, typing yes at the prompt:

node bin/reset-region-provisioning.ts --stage staging --region eu-central-3

Every non-production FilOne stage offers eu-central-3. Repeat for any personal or preview stage that has tenants there.

9. Merge (infra-nodes, the stack's checkout)

gh stack merge 172 --yes --squash

This merges this PR and #172.

10. First deploy (staging host)

git fetch origin
git checkout --detach origin/main
scripts/host/deploy-platform.sh
scripts/host/provision-apps.sh
systemctl start filone-reconcile.timer

deploy-platform.sh runs postgres-init, which creates piri_0 to piri_2. provision-apps.sh removes the old filone-piri container and starts the Piris one after another. It prints their logs, prefixed piri-<n> |, while each first start runs piri init, registers on chain and creates a proof set. That takes minutes per Piri, and a 403 means that Piri's onboarding didn't complete. The script ends with acceptance checks for OpenBao, the three Piris' /readyz, Ingot's /health, and every hostname's certificate and status document. The timer is still enabled from before, so start is enough, and its first pass should report "nothing to deploy".

11. Verify

• Node: on the host, run scripts/ci/smoke-test.sh staging/eu-central-3 and systemctl list-timers | grep filone.

• Central: in infra-central, run make smoke STAGE=staging.

• End to end: from a checkout of Test each node end to end after its smoke test #154 until it merges, load the partner key into the shell without printing it, run the test, clean up, and drop the key:

  export HILT_PARTNER_KEY="$(AWS_REGION=us-east-2 aws ssm get-parameter --name /forge-central/staging/hilt/partner-key --with-decryption --query Parameter.Value --output text)"
  E2E_RUN_ID=migration-1 scripts/ci/e2e-test.sh staging/eu-central-3
  E2E_RUN_ID=migration-1 scripts/ci/e2e-test.sh staging/eu-central-3 --cleanup
  unset HILT_PARTNER_KEY

  On dev, the final tenant delete in --cleanup failed with HTTP 500. Hilt on dev and staging has no revocation settings, so it sent revocations to prod's Swarf, which answered 404. Point Hilt's revocations at its own stage's Swarf infra-central#264 fixes that, and it reaches staging with the promotion merged before the window.

• Logs: in Grafana, {service_name="appliance-staging-eu-central-3-piri"} |= "/blob/accept" should show the test's blobs under instance values staging/eu-central-3/piri-0 to piri-2. That also confirms step 6's Alloy edits.

• FilOne: run gh workflow run e2e-staging.yaml --repo fil-one/fil-one.

• Proofs: over the next day, each Piri should submit its first proof for its new proof set.

12. Clean up (staging host)

reset-apps.sh recreated an empty piri database, which the new layout doesn't use. Confirm it has no tables:

docker exec filone-platform-postgres-1 psql -U admin -d piri -c '\dt'

If it reports no relations, drop it and remove the old data directory. rmdir refuses a directory that isn't empty.

docker exec filone-platform-postgres-1 psql -U admin -d admin -c 'DROP DATABASE piri'
rmdir /mnt/data/fil-one/data/piri

Refs FIL-1503

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-20T10:12:06.047013Z	114a9e3	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

🔵 Needs a closer look

Although the committed diff is clean and consistent, it is one step of a coordinated live staging migration involving DNS teardown/recreate, a Postgres database-layout change, on-chain wallet implications, and manual multi-repo/host steps that warrant human sign-off.

No open issues 🎉

What changed in this PR

This PR moves the staging eu-central-3 bare-metal appliance from a single Piri to three (piri-0, piri-1, piri-2) behind its one Ingot, mirroring the layout dev already adopted in #170. Because the host keeps a single data disk, each Piri gets a directory under /mnt/data/fil-one/data/piri-<n> and a loopback port (15100–15102) rather than a dedicated volume. The committed changes are the declarative pieces of a larger, carefully ordered live migration (the manual host Caddy/Alloy edits and docs land out-of-band, per the PR description and #172).

Changes:

• Define three Piri services from a shared x-piri-base anchor, each with its own hostname, loopback port, Postgres database (piri_0..2), key/wallet, data directory, and service.instance.id OTEL attribute; start them one-at-a-time via depends_on.
• Serve each Piri hostname through a new (filone-piri) host-Caddy snippet, create per-Piri DNS A records (piri-<n>.eu-central-3.staging.fil-forge.com), and split postgres-init into ensure_role/ensure_database to create one role and three databases.
• Update node.env to a PIRI_HOSTNAME_SUFFIX (dropping PIRI_HOSTNAME/PIRI_PUBLIC_URL) and provision three per-Piri data directories at bootstrap.

File	Description
terraform/​envs/​staging/​eu-central-3/​main.tf	Replaces the single piri A record with a for_each creating piri-0..2.eu-central-3.staging.fil-forge.com.
scripts/​host/​bootstrap-staging-eu-central-3.sh	Creates piri-0/1/2 data directories instead of a single piri directory.
nodes/​staging/​eu-central-3/​platform/​config/​caddy/​host-caddy.caddy	Adds a reusable (filone-piri) snippet and routes three hostnames to ports 15100–15102.
nodes/​staging/​eu-central-3/​platform/​compose.yml	Splits postgres-init into role/database helpers and creates piri_0..2 for the one piri role.
nodes/​staging/​eu-central-3/​node.env	Switches to PIRI_HOSTNAME_SUFFIX, removing PIRI_HOSTNAME/PIRI_PUBLIC_URL.
nodes/​staging/​eu-central-3/​apps/​compose.yml	Defines piri-0/1/2 from a shared base anchor with per-Piri port, URL, DB, secrets, data dir, and OTEL instance id.

I verified the changes are internally consistent (ports, hostnames, databases, and data directories all align across the six files), that the removed PIRI_HOSTNAME/PIRI_PUBLIC_URL are only consumed as fallbacks when PIRI_HOSTNAME_SUFFIX is unset (scripts/host/lib.sh, scripts/ci/smoke-test.sh, .github/workflows/smoke.yml), that ${FILONE_NODE} is defined in node.env, that the {args[0]} snippet syntax matches the existing dev Caddyfile convention, and that the smoke test auto-enumerates all three Piris. The diff faithfully follows the already-merged dev pattern (#170), and I found no concrete code defects to comment on.

_{Review effort: Balanced}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 114a9e3801

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".