[bajtos]

Written by Claude.

Test each dev and staging node end to end after its smoke test. The smoke test proves the services answer and run the pinned images. This proves a node can serve a customer.

Fixes FIL-1180.

The test

scripts/ci/e2e-test.sh <node> checks each node on every push to main and on the hourly schedule. It does six things:

• creates a tenant through central's Hilt;
• mints an S3 key for the tenant that expires after 2 hours;
• creates a bucket on the node's Ingot;
• uploads a 4 KiB object with PutObject and a 16 MiB one with a multipart upload, checking that the ETag shows two parts;
• downloads both and compares their SHA-256 sums;
• deletes everything it created, in a separate --cleanup run.

The test does not retry: a new tenant on every run means each run tests tenant creation, and the next hourly run tests again.

Cleanup runs even after a failed test. It deletes only what the test recorded creating, in the order Hilt requires: objects, then the bucket, then it disables the tenant and deletes it. Each delete is retried 3 times. If one still fails, cleanup stops and keeps the tenant, because Hilt cannot recover the storage of a tenant deleted with data in it.

The Slack alert now tells four cases apart: the node never reached the commit, it failed the smoke test, it failed the end-to-end test (naming the failed operation), or cleanup left something behind (naming the tenant and bucket, with a link to the manual cleanup in the runbook).

The credential

The test calls Hilt with FilOne's partner key for the stage, hilt/partner-key in central's SSM, copied into the HILT_PARTNER_KEY secret. That key can manage every tenant on its Hilt (FIL-1489), so it lives in a GitHub Environment per stage (dev, staging) that admits only main. As a result, this workflow can no longer be dispatched from a branch. Every run also adds a deployment record to the repository's Deployments view.

Production stays out of the matrix until FIL-1491 finds a credential that cannot reach customer tenants.

The script never puts the partner key on a command line, and it masks the minted S3 secret before doing anything else with it. Each new tenant leaves records in Sprue and the PLC directory that Hilt does not delete yet (FIL-1488).

Before merging

Create the dev and staging Environments here, limited to main, then set HILT_PARTNER_KEY in each. The runbook section "Setting up the end-to-end test's credential" has the steps. A workflow that names a missing Environment creates it with no branch rule, so this has to happen first.

Testing

• make check-shell passes.
• The script has not run against a real Hilt or node yet. I ran it against a fake Hilt and a fake aws CLI, which covered:
  • a clean run;
  • a tenant that already exists, which is left untouched;
  • a failed key creation, after which cleanup deletes only the tenant;
  • corrupted downloads, caught by the comparison;
  • a bucket that cannot be deleted, after which cleanup tries 4 times, keeps the tenant and names both leftovers;
  • Hilt failing twice before deleting the tenant, which the retries absorb.
• Before merging, run the test and its cleanup by hand against dev, as the runbook's "Run the end-to-end test by hand" shows. Then confirm that GET /tenants/<name> on Hilt answers 404.
• After merging, watch the first push run and one hourly run on both nodes.

[bajtos]

I ran the E2E tests against dev and already discovered a misconfiguration. Fixed here:

• Point Hilt's revocations at its own stage's Swarf infra-central#264
• fix(config): require the revocation service settings hilt#99