Repository navigation
Conversation
Replace smelt's single-VM Docker Compose deployment for the five central services with Terraform-managed infrastructure: ECS/Fargate behind an ALB, one RDS Postgres instance with a database per service, and OpenBao as the secrets manager. Supports multiple stages through a `stage` variable. The substantial change from smelt is where secrets come from. smelt mints keys on an operator's laptop and stores them in 1Password; here a Go Lambda runs inside the VPC, writes private material straight to SSM Parameter Store, and returns only DIDs, wallet addresses and database names. No private key reaches Terraform state or a local disk. - Port smelt's keygen: Ed25519 identities, secp256k1 wallets with the three serializations each consumer expects, and the UCAN startup proofs, now issued in-process through ucantool's new ucandelegate API rather than by shelling out to its CLI. - Never regenerate an existing secret. Writes use Overwrite=false, so re-applying cannot abandon a funded wallet or change a registered DID. - Namespace parameters per service, so a task execution role reads only /forge/<stage>/<service>/* and a compromised task cannot reach another service's credentials. - Store OpenBao's data in Postgres and seal it with KMS. Fargate has no durable local disk, and KMS removes the unseal key, the 1Password item holding it, and the sidecar that applied it. - Give hilt an AppRole scoped to forge/hilt/data/tenant/* instead of the Vault root token, which smelt tracks as debt. - Expose OpenBao publicly, since fil-one/RFC#21 makes it the transit seal regional appliances authenticate against to unseal at boot. - Add a fund phase that signs the three FilecoinPay transactions inside AWS, so the payer key never leaves the account. Terraform never invokes it; `make fund-payer` shows a plan and requires confirmation. - Pin images by manifest digest rather than git SHA, which names the last commit instead of the code just built and collides against a dirty tree. Use ucantool's identity helpers rather than libforge's: libforge formats the signer into its error text with %s, and an ed25519 signer is a []byte with no String method, so a marshalling failure renders the raw private key into a logged error. Reported separately; sprue, hilt and swarf still depend on it. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
bajtos
requested review from
frrist
and
a lite review from Copilot
and removed request for
Copilot
August 11, 2026 19:24
bajtos
requested review from
frrist and
hannahhoward
and removed request for
frrist and
hannahhoward
August 11, 2026 19:26
ECR repositories are regional and Lambda pulls an image only from ECR in the same region as the function, so a stage outside us-east-2 cannot reuse the one repository the bootstrap workspace created. The workspace name was hardcoded in the cloud block, leaving no way to stand up a second one. Move the repository into a modules/provision-ecr module and reduce the env directory to a per-region wrapper naming its own workspace and provider region. Adding a region is now a copy of that directory with two names changed. The HCP cloud block cannot interpolate a workspace name, and the tag-based alternative moves the region out of the repo into workspace variables where a reviewer cannot see it. A directory per region keeps it explicit and matches how the stages are already laid out. The us-east-2 workspace has to be renamed to infra-central-bootstrap-us-east-2 in HCP Terraform before the next apply, which preserves its state. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5[1m]
The five root modules floored at 1.9, several releases behind. 1.15 is the current stable line; 1.16 is still in beta, so it would force pre-release builds on everyone. Nothing in the tree needs a 1.15-only feature yet. The point is to keep the floor close to what the workspaces will actually run, so a constraint failure surfaces at init rather than mid-apply. Every root module plans and applies through HCP Terraform. Pin each workspace to 1.15.x once it exists, instead of leaving it tracking latest, so the 1.16 upgrade is a scheduled change. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The cloud blocks named "fil-forge", which is the GitHub org, not the HCP one. No workspace would have resolved under that name. The two tfe_outputs data sources in the apps modules name the org too, so they move with the cloud blocks. Had they been left behind, both apps workspaces would fail to read the platform outputs they build on. The remaining fil-forge references in the tree are GitHub: Go module paths and two README links. Those stay. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The provision Lambda image lived in `forge-provision`, a flat name with no room for the service images that will move off GHCR. It is now `forge-central/provision`: one repository per image under a shared prefix, which is what makes per-image push permissions, lifecycle policies and tag mutability possible. A single repository holding every service, distinguished by tag, forfeits all three. The module is `modules/ecr` rather than `modules/provision-ecr`, so the repositories to come have somewhere to live, and its output is namespaced as `provision_repository_url`. `make publish` also wrote a `sha-<short>` tag that nothing read, since Terraform deploys the Lambda by digest. It now pushes by digest with no tag at all, which lets the repository be IMMUTABLE. That change makes the lifecycle rule dangerous. It expired untagged images after 14 days, where untagged used to mean orphaned by a re-pushed tag. Untagged now means every image, including the digest a stage pins, and a Lambda whose image is deleted from ECR stops working and cannot be updated. ECR's rule language cannot express "keep anything a stage pins", so the rule is gone and pruning is manual. The images are a few tens of MB. Applying this replaces the repository, and Terraform cannot destroy one that holds images. Drop the two old resources from the bootstrap state, apply, delete `forge-provision` with `--force`, then `make publish` and repoint each stage's provision_image_repository_url. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Nothing in the config said which account a workspace applies into. An
apply under the wrong AWS_PROFILE succeeded, created a real stage in the
wrong place, and recorded it in HCP state. Every provider block now
carries allowed_account_ids, so a mismatch fails at plan time and names
the account it found.
The ids live in modules/constants, a module that creates nothing and
returns literals. Root modules cannot share a variable, and a provider
block may reference module outputs as long as they do not come from a
resource.
provision_image_repository_url was the second place each account id was
written, by hand, into terraform.tfvars. Both stages now derive it from
their account, their region and the repository name, all three from
modules/constants, and the variable is gone. That is the same
derivation `make publish` already does from the calling identity, and it
states the invariant that made copying error-prone: a Lambda pulls only
from its own account's registry in its own region, so those two values
are the whole address. Prod's REPLACE_ME resolves as a side effect.
One bootstrap workspace per region cannot serve both accounts for the
same reason, so bootstrap/<region> becomes bootstrap/<account>/<region>
with a workspace and an account guard each.
In HCP Terraform: create infra-central-bootstrap-{nonprod,prod}-us-east-2
with credentials for their accounts, delete
infra-central-bootstrap-us-east-2, and drop any workspace variable named
provision_image_repository_url, which is now undeclared.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The HCP workspace names and the Project default tag both derive from the repository name, which is becoming forge-central. The workspaces have to be renamed in HCP Terraform as well; Terraform does not create them and a plan against a missing workspace fails. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The dev platform stage plans in HCP, which sees only what is in version control, so a digest written to a gitignored file on the operator's machine was never applied anywhere. Track image.auto.tfvars instead and say so in the runbook: `make publish` writes it, the operator commits it, the HCP run picks it up. Promotion to prod stays a deliberate copy of the digest into the prod terraform.tfvars. Also make `make publish` create a docker-container builder on first use. Docker Desktop's default builder cannot push by digest and cannot cross-build for arm64, so the target failed on a stock install. Document how each part is deployed today, now that there is no CI: the bootstrap workspaces and the image are handled by an operator locally, and the platform and apps paths are still open. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The ecs-service module decided whether to create the task policy by comparing the policy JSON against null. OpenBao's seal policy names the KMS key created in the same apply, so that JSON is unknown while planning, and a count that depends on it cannot be evaluated: the first plan of a stage failed with "Invalid count argument". Take a map keyed by policy name instead. for_each needs its keys known at plan time but tolerates unknown values, so a service gets its policy even when the permissions name resources the same apply creates. Also drop failure_threshold from health_check_custom_config. AWS ignores the value and always applies 1, so it only produced a deprecation warning. The block itself stays: ECS reports task health through it, and it cannot be added to an existing Cloud Map service. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5[1m]
Both dev workspaces track main in HCP with auto-apply on, so a merge reaches the stage without anyone running Terraform, and a run trigger keeps apps behind the platform outputs it reads. The runbook still told operators to run terraform apply from a laptop, and the deployment table still called the path unsettled. Record what a new stage's workspaces need, since none of it is visible in the repository: remote execution, the OIDC variable set, trigger patterns covering terraform/modules, and the run trigger on platform. Prod is described in one sentence and in the future tense. It does not exist yet and its process is likely to change. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5[1m]
Dev followed rolling tags, which was defensible when a deploy was an operator running apply. It is not now: HCP applies dev on every commit to main, so with a tag what dev runs depends on when a task last restarted rather than on what was merged. Take a digest per service instead, validated to start with sha256: so a tag pasted here fails at plan time rather than at task start. The dev stage pins what each repository's main tag pointed at today; all six are multi-arch indexes carrying the linux/arm64 the tasks run on. envs/prod/apps/terraform.tfvars loses its pinned tags without gaining digests. The tags named a scheme the service repos no longer publish, and the digests that replace them have to come from real images. Prod is not deployed, so a plan that fails for want of them is the right outcome. Also record two runbook steps that the move to remote runs broke: rotating an identity and rotating hilt's OpenBao credential both need a provision phase re-run, and neither trigger is reachable from a VCS-driven run. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5[1m]
The stage prefixed its AWS resources with forge-, which says nothing about which Forge deployment they belong to. This repository is the central one, as against the regional nodes, so the names should say so. Resource names take fc- rather than the full forge-central. A target group name is capped at 32 characters and has to hold <prefix>-<stage>-signing-service, which forge-central-staging- already overflows. Paths and namespaces are nowhere near a limit, so they spell forge-central out, matching the ECR repository that already did. - resources: forge-<stage>-* -> fc-<stage>-* - parameters: /forge/<stage>/* -> /forge-central/<stage>/* - log groups: /forge/<stage>/<svc> -> /forge-central/<stage>/<svc> - namespace: forge.internal -> forge-central.internal - KV mount: forge/hilt -> forge-central/hilt - command: cmd/forge-provision -> cmd/provision The KV mount moves on both sides at once, in vaultinit and in hilt's HILT_VAULT_HASHICORP_MOUNT, since hilt's path builder is mount-relative. Applying this re-mints a stage. The parameter path is the identity of every secret, so seed finds an empty namespace and generates fresh keys, DIDs and wallets; the old payer keeps its funds at its old address and the new one needs funding from the faucet. Parameters under the old path are not Terraform-managed and outlive the apply, so retire them once the stage is healthy. Only dev is affected: prod has never been applied. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
A dev stage sets protect_stateful_resources = false so that it can be destroyed, but three things ignored the flag and blocked or outlived the destroy. S3 refused to delete a bucket holding objects, the delegator's tables paid for continuous backups a disposable stage does not want, and the KMS key sat in PendingDeletion for 30 days afterwards. Versioning goes with them. It guards against losing something irreplaceable, and nothing in these buckets is: sprue writes the objects and can write them again. Keeping it would mean a lifecycle rule to stop noncurrent versions accumulating, and it made a bucket undeletable until every version had been purged a page at a time. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5[1m]
openbao crash looped on every task start, three lines per attempt and a new attempt every seventy seconds: mkdir: can't create directory '/run/forge': Permission denied /bin/sh: can't create /run/forge/openbao.hcl: nonexistent directory error loading configuration from /run/forge/openbao.hcl The ephemeral volume behind secret_dir was created only for a service that declares secret_files, and openbao declares none: its config is a template around the Postgres DSN rather than a secret in its own right, so its own shell_command assembles the file. With no mount, secret_dir belongs to the image, the task is not root, and the write is denied. The mount is what a service needs, and writing the files is only one reason to want it. writable_secret_dir asks for it directly, so the requirement is visible at the call site rather than implied by an empty map somewhere else. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Mounting the ephemeral volume let openbao create the directory and then denied the write into it: /bin/sh: can't create /run/forge/openbao.hcl: Permission denied A Fargate ephemeral volume mounts root-owned and takes no uid or gid option, so no mount can be written by a container that is not root. That is not an openbao quirk. The delegator and the signing service both end their images with USER nobody and would have failed the same way on the first deploy that exercised their secret_files; sprue works only because its image never sets USER and runs as root. So drop the volume and write into /tmp, the one directory an image is relied on to leave world-writable. The container filesystem carries the same AES-256 encryption and dies with the task, so the volume was holding a guarantee that /tmp already gives. umask 077 and chmod 400 still keep every file readable only by the process that wrote it. This replaces writable_secret_dir, added an hour ago for openbao. The knob asked for a mount, and a mount is the thing that cannot work. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The first apply of a stage waited four minutes and then timed out
against a server that was behaving correctly:
openbao is not serving yet attempt=30
reason="openbao is sealed; check that the KMS seal key is reachable"
An OpenBao that has never been initialised reports itself sealed,
because there is no barrier to unseal yet. WaitForUnsealed read that as
a reason to keep waiting, and the call that would have initialised it
runs after the wait returns, so the phase waited on a state only it
could produce.
Uninitialised now counts as ready. The sealed branch keeps its KMS
message, which is accurate once initialised is true: a barrier that
exists and stays shut means the seal key did not open it.
That moves the wait rather than removing it. sys/init returns once the
barrier is written and the seal opens it a moment later, so the phase
now waits for the barrier it just created before it mounts anything.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The provision Lambda could not reach OpenBao. Its packets were dropped rather than refused: dial tcp 10.20.33.229:8200: connect: connection timed out aws_security_group.service declares its rules inline, and inline blocks are the authoritative set for a group: anything added by a separate aws_security_group_rule is removed the next time the group is applied. So the standalone rule and the group have been undoing each other on every run, which is what the recurring "aws_security_group.service: Plan to update" in each plan was. Whichever ran last decided whether the Lambda could reach port 8200, and this time it was the removal. Moving the rule inline settles it. The comment on the old resource said a separate rule avoided a cycle between the two group definitions, but there is no cycle to avoid: the lambda group holds one egress rule to 0.0.0.0/0 and refers to no other group. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Replacing a Cloud Map service deadlocks against the task it serves. Terraform destroys the old service before it updates the ECS service that points at it, so the registration is still there when the delete runs, and Cloud Map refuses: ResourceInUse: Service contains registered instances; delete the instances before deleting the service Scaling the ECS service to zero clears it, but only until something starts a task again, and the apply itself is what brings one back. Three runs failed this way on the same service id. force_destroy has the provider deregister the instances as part of the delete, so the replacement no longer depends on catching an empty window. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Every plan since 66c8db2 has scheduled a replacement of the Cloud Map service, and each replacement landed in the state that forces the next one. The service reads back with no HealthCheckCustomConfig at all while the configuration declares the block, and the block is ForceNew, so the only move Terraform has is to recreate. An empty block is the cause. Without failure_threshold the provider sends no custom health config on CreateService, AWS records none, and the read finds none. Stating the value makes the block real. It is inert either way, since AWS always applies 1 and the attribute is deprecated, so the cost is one deprecation warning per plan against a service that stops being recreated on every apply. One more replacement follows this, which force_destroy now lets finish. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
OpenBao 2.x refuses audit devices created over the API, so the vault phase failed with a 400 as soon as it reached the stdout device. The device has to come from an audit stanza in the server config instead, which needs stdout confirmed as a sink that cannot block a Fargate task and a check that 2.6.0 applies the stanza at first boot. Both are written up under Planned work; until then OpenBao runs without an audit log. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Two settings decide whether the apps workspace works, and neither was in the stage checklist. Auto-apply run triggers is independent of auto-apply, so a workspace with auto-apply on still parks a triggered run in front of a confirm button. That is the platform-only change, which is the case nobody is watching. The state read needs a token, not a sharing grant. Remote state access controls do not cover tfe_outputs: it goes through the API as the tfe provider, so the workspace needs TFE_TOKEN with read access to the platform workspace's outputs. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The tfe_outputs data source marks its whole `values` map sensitive, and the taint spreads to everything derived from it, so the apps workspace could not export service_urls or service_dids without declaring them sensitive. Nothing the platform workspace exports there is a secret: ARNs, subnet ids, hostnames. Read `nonsensitive_values` instead of hiding two useful outputs behind `sensitive = true`. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
An apply wires up everything the central services need, so it reads as though a stage is complete once both workspaces are green. The first Piri/Ingot appliance pointed at one needs five things nothing here provides: three registration writes against the delegator, sprue and hilt, and two UCAN proofs that cannot be minted from nothing because each depends on a key that does not exist until the appliance does. Written down now rather than during the first onboarding, together with the two constraints that shape the answer: there is no shell to run sprue's and hilt's admin CLIs in, and the proof traffic runs in both directions, so onboarding is a request carrying the appliance's DIDs rather than another idempotent phase. Assisted-by: Claude:claude-opus-5[1m] Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Nothing said whether an apply worked. Confirming a stage meant opening five hostnames by hand, and the check that matters most is the one an operator is least likely to think of: sprue, hilt and swarf mint an ephemeral identity when no key is supplied and report themselves healthy either way, so a service whose DID has silently changed underneath everything registered against it looks exactly like a working one. Two checks per public service, over public HTTPS. A 200 from the health path covers the whole ingress route at once, and the did.json check compares the published id against did:web:<hostname>, which is what proves the key arrived from SSM. Needs neither AWS credentials nor a TFE token: the hostname suffix comes from the stage's committed platform/terraform.tfvars, and the expected DID is built from the hostname. Services are probed concurrently, since a task that accepts the connection and never replies waits out the whole timeout. Two gaps the script names in its own output rather than passing over. plc has no public hostname, so nothing here reaches it. And piri-signing-service serves no document at the did:web it was given, so its DID resolves nowhere; the service_dids description said otherwise and is corrected here. Assisted-by: Claude:claude-opus-5[1m] Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
The delegator has never started. Every task died before its process ran, with runc reporting an invalid environment variable containing a nul byte and naming nothing else. Both of its startup proofs are bare DAG-CBOR delegations, which is what its delegation.Decode reads, and binary. Every proof reaches its consumer as an SSM-injected environment variable that the entrypoint wrapper writes to a file, and an environment variable cannot carry a NUL. The value never got that far intact anyway: PutParameter goes over JSON, which replaces invalid UTF-8 with U+FFFD. So a binary proof is stored base64-encoded and decoded on the way to the file. hilt's proof is a base64+gzip container, already text, and is still stored as ucantool writes it, trailing newline included. A stage seeded before this needs both parameters deleted and the seed phase re-run, because an existing proof is left alone. The README says which two and why. Assisted-by: Claude:claude-opus-5[1m] Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
sprue could not start: creating its agent-message bucket failed on a DNS lookup of fc-dev-agent-message-<account>.minio, a host that exists in smelt's compose file and nowhere in the VPC. Its storage.s3.endpoint carries a default of http://minio:9000, so leaving the variable out does not select real S3, it selects a MinIO that was never deployed. Setting it empty is what reaches AWS, and the empty value survives because sprue calls viper's AllowEmptyEnv. Nothing else changes: the buckets exist, the task role already grants the ListBucket that sprue's HeadBucket probe needs, and CreateBucket is reached only when that probe fails. Assisted-by: Claude:claude-opus-5[1m] Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
plc crashed on startup with no pg_hba.conf entry for host "10.20.54.142", user "plc", database "plc", no encryption Its credentials are a JSON blob rather than a URL, and the blob carried no sslmode, so plc built a plaintext connection string. The instance runs Postgres 16, whose default parameter group sets rds.force_ssl, and RDS rejects an unencrypted connection with a pg_hba error that names no other cause. Every other service reaches the same instance over a DSN with sslmode=require. no-verify rather than require because plc connects through node-postgres, which verifies the server certificate against Node's own trust store. That store carries no Amazon RDS root and the image has no bundle to point NODE_EXTRA_CA_CERTS at, so require fails the handshake instead. The connection is encrypted either way, which is what force_ssl asks for. No migration: db-creds-json is rewritten on every seed run. Assisted-by: Claude:claude-opus-5[1m] Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Publishes the provision image carrying the base64 proof encoding and plc's sslmode, and forces the seed phase to run with it. Both are needed. aws_lambda_invocation re-invokes only when its input changes, and the function's image is not part of that input, so a new digest on its own would leave the seed phase untouched. The delegator's two proof parameters were deleted by hand beforehand. The seed phase leaves an existing proof alone, so it re-issues them, this time base64-encoded. plc's db-creds-json is rewritten every run and needed nothing. Assisted-by: Claude:claude-opus-5[1m] Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Its store describes both tables before serving, and with no endpoint override a failure there is fatal: an absent table means the wrong account, so it refuses to create one and stops. The task role granted only item-level actions, so the check failed with AccessDenied and the service never finished wiring its dependency graph. Assisted-by: Claude:claude-opus-5[1m] Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Every module sat directly under terraform/modules, so a workspace could not name its dependencies in a trigger pattern without matching terraform/modules/**/* and planning both workspaces on every module change. Modules used only by the platform workspace now live under modules/platform, and the two that cross the boundary (ecs-service, constants) live under modules/shared. Each workspace can point its triggers at its own composite module plus shared. Assisted-by: Claude:claude-opus-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Every plan warns that health_check_custom_config.failure_threshold is deprecated. The warning is expected: AWS fixed the Cloud Map custom health check wait at one 30-second interval and deprecated the parameter, provider v6.0.0 mirrored that, and 1ccea93 already found that omitting the value makes the provider create the service with no custom health config, forcing a replacement on every plan. Record the upstream state in the routing.tf comment and a README sharp edge, with links to the issues that will eventually retire the warning: - hashicorp/terraform-provider-aws#44285 asks the provider to stop warning on the value AWS forces anyway - hashicorp/terraform-provider-aws#43428 lets the argument be removed without replacing the service - hashicorp/terraform-provider-aws#44291 got the deprecation documented Assisted-by: Claude:claude-fable-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Sections had accumulated in the order they were written, so standing up a stage meant jumping between the Stages section and the Runbook, and the Terraform module tree was printed in both. Concepts now come first and every procedure lives in the Runbook in the order an operator meets it. "Adding a stage" and "A personal sandbox stage" move there; the two "Why..." notes on UCAN proofs become subsections of the identity rotation they explain. Corrections along the way: - the intro counted five services where four later passages counted six - Related described smelt#11 as the Compose deployment being replaced, which is the repository, not that PR - "Prod is not set up yet" contradicted envs/prod/ in the layout; the config is committed, the HCP workspaces are not created - indexer and etracker appeared once with no introduction - Planned work claimed every item needs a change outside this repo Added a table of contents, a prerequisites list covering cast and buildx, the CloudWatch log group names, the HCP organization and project, and Planned work entries for RDS restore, alerting and cost. Assisted-by: Claude:claude-fable-5 Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
This was referenced Aug 13, 2026
bajtos
added a commit
to fil-one/RFC
that referenced
this pull request
Aug 17, 2026
<!-- rfc-preview-link:start --> [📖 Preview](https://github.com/fil-one/RFC/blob/forge-central-deployment-strategy/rfcs/2026-08-forge-central-deployment-strategy.md) <!-- rfc-preview-link:end --> ## Summary Forge consists of two independent sets of services: Forge-Central contains centrally managed set of services operated by the FilOne team, FilOne Appliance contains the software stack powering regional nodes. This pull request proposes deployment strategy for the Central stack. It's a complement for [rfc#19](#19), which describes the deployment strategy for FilOne Appliance. ## Proofs of Concept - Terraform version: fil-forge/infra-central#1 - Pulumi version: fil-forge/infra-central#23
Member
Author
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace smelt's single-VM Docker Compose deployment for the five central services with Terraform-managed infrastructure: ECS/Fargate behind an ALB, one RDS Postgres instance with a database per service, and OpenBao as the secrets manager. Multiple stages are supported through a
stagevariable.This pull request is intended as the first step toward the future staging & production deployments. It enables continuous deployment to a dev environment and is designed to support multiple stages, including production running on a different AWS account. I expect that we will incrementally improve this setup until it's ready to support our staging & production deployments too.
Architecture
Cloud/AWS native, but using non-AWS infra dependencies wherever possible - Postgres, OpenBao.
All services are running on ECS and can be scaled independently.
We are using as few AWS-specific components as possible, to make it easy to migrate to a different infrastructure provider in the future if desired.
delegatordoes not support Postgres and requires DynamoDB instead. It should be easy to add support for Postgres, but not worth it now.spruerequires an S3-compatible storage, and we use AWS S3 for simplicity. This is easy to swap for a different S3-compatible object storage if we migrate off AWS.Secrets never touch a laptop or Terraform state
In fil-forge/smelt#11, I proposed minting keys on a developer's or operator's machine and storing them in 1Password. Here, a Go Lambda (
cmd/forge-provision) runs inside the VPC, writes private material straight to SSM Parameter Store, and returns only DIDs, wallet addresses and database names.Platform
forge/hilt/data/tenant/*instead of the Vault root token, which smelt tracks as debt.Funding the wallet
A fund phase signs the three FilecoinPay transactions inside AWS, so the payer key never leaves the account. Terraform never invokes it;
make fund-payershows a plan and requires confirmation.Start reading here: README.md