From 1fc944396d2f5ec95df83daadc1e741b9fde475e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miroslav=20Bajto=C5=A1?= Date: Thu, 13 Aug 2026 18:50:28 +0200 Subject: [PATCH 1/5] rfc: Forge-Central Deployment Strategy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Miroslav Bajtoš --- ...26-08-forge-central-deployment-strategy.md | 167 ++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 rfcs/2026-08-forge-central-deployment-strategy.md diff --git a/rfcs/2026-08-forge-central-deployment-strategy.md b/rfcs/2026-08-forge-central-deployment-strategy.md new file mode 100644 index 0000000..be954ee --- /dev/null +++ b/rfcs/2026-08-forge-central-deployment-strategy.md @@ -0,0 +1,167 @@ +# RFC: Forge-Central Deployment Strategy + +**Status:** Proposal + +## Authors + +- [Miroslav Bajtoš](https://github.com/bajtos) + +## Motivation + +Forge consists of two independent sets of services: Forge-Central contains centrally managed set of +services operated by the FilOne team, FilOne Appliance contains the software stack powering regional +nodes. + +This pull request proposes deployment strategy for the Central stack. It's a complement for +[rfc#19](https://github.com/fil-one/RFC/pull/19), which describes the deployment strategy for FilOne +Appliance. + +## Requirements & Constraints + +1. All aspects of the deployment are managed using infrastructure-as-code, version-tracked in Git. + +1. Continuous delivery for the dev environment. Every commit landed on `main` in any of the service + repositories is eventually deployed to the dev env with no manual steps involved. + +1. We can promote a given combination of service versions & configurations from dev env to other + environments (staging, production) as a single git commit. + +### Non-requirements + +The following requirements are out of the scope of the initial version: + +1. Zero-downtime upgrades +2. High availability / no single points of failures. +3. Multi-region deployment. +4. Local Lotus node providing Filecoin RPC + +## Components & Dependencies + +Forge Central contains the following services: + +| Service | Dependencies | Interacts with | +| -------------------- | ------------------------------- | ----------------- | +| sprue | Postgres, 3 S3 buckets | plc | +| hilt | Postgres, OpenBao | sprue, plc, swarf | +| swarf | Postgres | plc | +| delegator | 2 DynamoDB tables, Filecoin RPC | | +| piri-signing-service | Filecoin RPC | | +| plc | Postgres | | + +## Proposal + +Adopt a cloud-native architecture that allows us to push a lot of the operational burden to AWS +while preventing a vendor lock-in that would make it difficult to migrate to a different cloud +provider or onprem. + +- Run all Forge services as idenpendent OCI containers on AWS ECS Fargate +- A single Postgres on AWS RDS, shared by all services, with one database per service for isolation +- OpenBao running on AWS ECS Fargate, unsealed automatically using AWS KMS +- AWS S3 for sprue +- AWS DynamoDB for delegator +- Manage all resources using Terraform + +### High-level Layout + +**Terraform Workspaces** + +Split the components into three Terraform workspaces: + +1. Bootstrap: ECR repository per AWS Account & Region. + +2. Platform: 3rd-party dependencies + + - Postgres + - OpenBao + - AWS KMS + - AWS DynamoDB & S3 + - AWS ALB + - AWS VPC & NAT + +3. Apps: Forge services + - sprue + - hilt + - swarf + - delegator + - signing-service + - plc + +This allows us to scope changes and limit their blast radius - a mistake in apps configuration will not accidentally drop or recreate the DB server. + +**Terraform Modules** + +Define the infrastructure resources in modules shared by all environments. + +Each environment is described by a new set of Terraform files that define only the parameters like +multi-AZ database, deletion protection and different VM sizing. + +### Secret & Key Generation + +When provisioning a new environment, we need to generate several secrets - wallet private keys and +identity private keys. These secrets must not leak and must not be stored anywhere outside of AWS +SSM or OpenBao, in particular they must not be written to local filesystem or make it to the +Terraform state. + +We will achieve this by implementing a `provision` Lambda function (written in Go) that will +generate all secrets and write them to AWS SSM. The Terraform will invoke this Lambda function when +provisioning a new deployment. + +### Continuous Delivery + +Terraform files for Forge Central will live in a dedicated GitHub repository - https://github.com/fil-forge/infra-central. + +Every commit landed to the `main` branch in this repository will trigge `terraform plan` and +`terraform apply` for the dev workspaces (platform, apps). + +We will create GitHub Actions-based pipeline to automatically promote changes from individual +service repositories: when a new image version is pushed to GHCR in a service repository, a pull +request will be automatically opened in the infra-central repository to bump the pinned digest. This +pull request will be automatically merged after it passes CI checks. + +## Alternatives Considered + +**[SST](https://sst.dev/)** + +Rejected: SST's main focus is on serverless projects written in TypeScript. + +**[Storoku](https://github.com/storacha/storoku)** + +Rejected: + +- Feature gaps: + - Storoku models one public HTTP service per repository, with its own VPC, ECS cluster, ALB, and so on. Our project is seven services sharing one stage. + - We need to fetch secrets from SSM and write them to config files when services start. Storoku has no entryPoint/command override; nothing writes secrets to files. + - Storoku hardcodes the healtcheck enpoint, while Forge services use multiple different paths + - No general-purpose Lambda we could use to provision secrets + - Autoscaling with blue/green deployments is not compatible with our DB migrations that require `desired_count = 1` +- Storoku adds another layer of abstraction that both humans and coding agents would need to learn. +- Coding agents can handle the verbosity of Terraform for us, there is less need for prefabricated resource bundles. +- Storoku is not maintaned. The only bugfixes and improvements we will get are those we make ourselves. + +However, we will review the best practices and security hardening that went into Storoku and apply +it to our Terraform projects. + +**[Pulumi](https://www.pulumi.com/)** + +Rejected? Discussing this on Slack. + +Pulumi has two main advantages: + +- We can use our language of choice (Go) instead of HCL, which makes it easier to create reusable components. +- We can write unit test for our infrastructure definitions and run them as part of regular "go test" workflow. + +It also comes with downsides: + +- We are already using Terraform to manage FilOne Cloudflare domains; we will end up having both Terraform and Pulumi in our toolbelt. + +- Pulumi is primarily TypeScript. Some plugins like awsx require Node.js runtime even if we describe + our Pulumi resources in Go. Either we will be limited to core features only, or we have to add + Node.js to our dependencies. (Adding Node.js is not a big deal to me, but it's a bit ugly.) + +- If we want to use Pulumi Cloud, it's yet another service to procure, and they charge per resource + managed (not per-seat/per-run as Hashicorp), which get expensive as the stack grows. (I am not + sure if this is applicable to us, though.) The alternative is DIY backend on S3 + KMS. We will + lose RBAC beyond S3 bucket IAM, no web UI, no deployment history view. + +- Pulumi's AWS provider is largely bridged from the Terraform provider, so coverage is equivalent, + but bug fixes occasionally arrive with a lag, and error messages sometimes leak Terraform-isms. From df4a17964a46a64ad50a952a1889ac86e713d928 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miroslav=20Bajto=C5=A1?= Date: Thu, 13 Aug 2026 18:57:35 +0200 Subject: [PATCH 2/5] fixup! rework reasoning for platform+apps split MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Miroslav Bajtoš --- rfcs/2026-08-forge-central-deployment-strategy.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rfcs/2026-08-forge-central-deployment-strategy.md b/rfcs/2026-08-forge-central-deployment-strategy.md index be954ee..224f1c4 100644 --- a/rfcs/2026-08-forge-central-deployment-strategy.md +++ b/rfcs/2026-08-forge-central-deployment-strategy.md @@ -86,7 +86,8 @@ Split the components into three Terraform workspaces: - signing-service - plc -This allows us to scope changes and limit their blast radius - a mistake in apps configuration will not accidentally drop or recreate the DB server. +This gives us faster deployments for typical changes that touch only the apps, as Terraform planner +does not need to check ~60 platform resources. **Terraform Modules** From 727422a443c350276637229fb7621cdcce07e902 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miroslav=20Bajto=C5=A1?= Date: Fri, 14 Aug 2026 09:54:33 +0200 Subject: [PATCH 3/5] Fix grammar Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- ...026-08-forge-central-deployment-strategy.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/rfcs/2026-08-forge-central-deployment-strategy.md b/rfcs/2026-08-forge-central-deployment-strategy.md index 224f1c4..202d152 100644 --- a/rfcs/2026-08-forge-central-deployment-strategy.md +++ b/rfcs/2026-08-forge-central-deployment-strategy.md @@ -31,7 +31,7 @@ Appliance. The following requirements are out of the scope of the initial version: 1. Zero-downtime upgrades -2. High availability / no single points of failures. +2. High availability / no single points of failure. 3. Multi-region deployment. 4. Local Lotus node providing Filecoin RPC @@ -54,7 +54,7 @@ Adopt a cloud-native architecture that allows us to push a lot of the operationa while preventing a vendor lock-in that would make it difficult to migrate to a different cloud provider or onprem. -- Run all Forge services as idenpendent OCI containers on AWS ECS Fargate +- Run all Forge services as independent OCI containers on AWS ECS Fargate - A single Postgres on AWS RDS, shared by all services, with one database per service for isolation - OpenBao running on AWS ECS Fargate, unsealed automatically using AWS KMS - AWS S3 for sprue @@ -83,7 +83,7 @@ Split the components into three Terraform workspaces: - hilt - swarf - delegator - - signing-service + - piri-signing-service - plc This gives us faster deployments for typical changes that touch only the apps, as Terraform planner @@ -111,7 +111,7 @@ provisioning a new deployment. Terraform files for Forge Central will live in a dedicated GitHub repository - https://github.com/fil-forge/infra-central. -Every commit landed to the `main` branch in this repository will trigge `terraform plan` and +Every commit landed to the `main` branch in this repository will trigger `terraform plan` and `terraform apply` for the dev workspaces (platform, apps). We will create GitHub Actions-based pipeline to automatically promote changes from individual @@ -130,14 +130,14 @@ Rejected: SST's main focus is on serverless projects written in TypeScript. Rejected: - Feature gaps: - - Storoku models one public HTTP service per repository, with its own VPC, ECS cluster, ALB, and so on. Our project is seven services sharing one stage. + - Storoku models one public HTTP service per repository, with its own VPC, ECS cluster, ALB, and so on. Our project is six services sharing one stage. - We need to fetch secrets from SSM and write them to config files when services start. Storoku has no entryPoint/command override; nothing writes secrets to files. - - Storoku hardcodes the healtcheck enpoint, while Forge services use multiple different paths + - Storoku hardcodes the healthcheck endpoint, while Forge services use multiple different paths. - No general-purpose Lambda we could use to provision secrets - Autoscaling with blue/green deployments is not compatible with our DB migrations that require `desired_count = 1` - Storoku adds another layer of abstraction that both humans and coding agents would need to learn. - Coding agents can handle the verbosity of Terraform for us, there is less need for prefabricated resource bundles. -- Storoku is not maintaned. The only bugfixes and improvements we will get are those we make ourselves. +- Storoku is not maintained. The only bugfixes and improvements we will get are those we make ourselves. However, we will review the best practices and security hardening that went into Storoku and apply it to our Terraform projects. @@ -149,7 +149,7 @@ Rejected? Discussing this on Slack. Pulumi has two main advantages: - We can use our language of choice (Go) instead of HCL, which makes it easier to create reusable components. -- We can write unit test for our infrastructure definitions and run them as part of regular "go test" workflow. +- We can write unit tests for our infrastructure definitions and run them as part of regular "go test" workflow. It also comes with downsides: @@ -160,7 +160,7 @@ It also comes with downsides: Node.js to our dependencies. (Adding Node.js is not a big deal to me, but it's a bit ugly.) - If we want to use Pulumi Cloud, it's yet another service to procure, and they charge per resource - managed (not per-seat/per-run as Hashicorp), which get expensive as the stack grows. (I am not + managed (not per-seat/per-run as HashiCorp), which get expensive as the stack grows. (I am not sure if this is applicable to us, though.) The alternative is DIY backend on S3 + KMS. We will lose RBAC beyond S3 bucket IAM, no web UI, no deployment history view. From 3f134957e6e533cc4ba5e4af72997060e916e264 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miroslav=20Bajto=C5=A1?= Date: Fri, 14 Aug 2026 09:55:09 +0200 Subject: [PATCH 4/5] more grammar fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Miroslav Bajtoš --- rfcs/2026-08-forge-central-deployment-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/2026-08-forge-central-deployment-strategy.md b/rfcs/2026-08-forge-central-deployment-strategy.md index 202d152..409b43e 100644 --- a/rfcs/2026-08-forge-central-deployment-strategy.md +++ b/rfcs/2026-08-forge-central-deployment-strategy.md @@ -160,7 +160,7 @@ It also comes with downsides: Node.js to our dependencies. (Adding Node.js is not a big deal to me, but it's a bit ugly.) - If we want to use Pulumi Cloud, it's yet another service to procure, and they charge per resource - managed (not per-seat/per-run as HashiCorp), which get expensive as the stack grows. (I am not + managed (not per-seat/per-run as HashiCorp), which gets expensive as the stack grows. (I am not sure if this is applicable to us, though.) The alternative is DIY backend on S3 + KMS. We will lose RBAC beyond S3 bucket IAM, no web UI, no deployment history view. From eb741012d8f5a9a58c68f222695a94da82b44b68 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miroslav=20Bajto=C5=A1?= Date: Mon, 17 Aug 2026 13:57:22 +0200 Subject: [PATCH 5/5] final touches MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Miroslav Bajtoš --- rfcs/2026-08-forge-central-deployment-strategy.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/rfcs/2026-08-forge-central-deployment-strategy.md b/rfcs/2026-08-forge-central-deployment-strategy.md index 409b43e..0009afa 100644 --- a/rfcs/2026-08-forge-central-deployment-strategy.md +++ b/rfcs/2026-08-forge-central-deployment-strategy.md @@ -144,16 +144,17 @@ it to our Terraform projects. **[Pulumi](https://www.pulumi.com/)** -Rejected? Discussing this on Slack. +Rejected. Pulumi has two main advantages: - We can use our language of choice (Go) instead of HCL, which makes it easier to create reusable components. - We can write unit tests for our infrastructure definitions and run them as part of regular "go test" workflow. -It also comes with downsides: +However, the downsides outweigh them: -- We are already using Terraform to manage FilOne Cloudflare domains; we will end up having both Terraform and Pulumi in our toolbelt. +- We are already using Terraform to manage FilOne Cloudflare domains; we will end up having both + Terraform and Pulumi in our toolbelt. - Pulumi is primarily TypeScript. Some plugins like awsx require Node.js runtime even if we describe our Pulumi resources in Go. Either we will be limited to core features only, or we have to add