Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,7 @@ All host-side ports live in a dedicated `15XXX` range to avoid collision with co
| ingot | 15130 | S3/HTTP | S3 gateway over Forge |
| ingot-postgres | 15131 | PostgreSQL | Ingot registry/metadata |
| ingot-openbao | 15132 | HTTP | Region KEK vault for ingot (OpenBao, raft storage) |
| central-openbao | 15210 | HTTP | Seal root for appliance OpenBaos (transit auto-unseal) |
| swarf | 15140 | HTTP/UCAN | UCAN revocation service |
| swarf-postgres | 15141 | PostgreSQL | Swarf revocation store |
| guppy | (none) | CLI | Client container |
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ All host ports live in a dedicated `15XXX` range to avoid collision with common
| ingot | 15130 | S3 gateway over Forge |
| ingot-postgres | 15131 | Postgres for ingot's registry/metadata |
| ingot-openbao | 15132 | Region KEK vault for ingot (OpenBao, raft storage) |
| central-openbao | 15210 | Seal root for appliance OpenBaos (transit auto-unseal) |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if we really need to exercise OpenBao unlocking in the ephemeral dev deployments created by Smelt. It feels like unneeded complexity to me.

The real dev/staging/prod deployment will not use Smelt, so we are gaining very little by adding the central OpenBao here.

The real process for setting up "local OpenBao unsealed by central OpenBao" is a bit complex, see https://github.com/fil-forge/infra-central/blob/main/docs/decisions/2026-08-region-onboarding.md, fil-forge/infra-central#50 and fil-forge/infra-central#49. (Note: I haven't exercised this process yet; the final version may be slightly different if I need to fix any bugs.)

@bajtos bajtos Aug 27, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My proposal: keep it simple in Smelt, let Ingot's OpenBao unseal automatically using a locally stored secret.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It just means it's impossible to test that revoked central access means an appliance can no longer read/serve encrypted data or that adding an appliance to the network in the first instance is blocked by it being authorised by the central vault.

Perhaps that is the useful functionality and tests we should add to this PR to make it worthwhile?

@bajtos bajtos Aug 27, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point.

I am not sure if this is the right place for testing these two features. In the current direction, Smelt is used for ephemeral deployments only. Long-living deployments will use infra-central & infra-nodes. (Why? The first nudge towards that direction comes from #11 (comment). More importantly, Smelt deploys both the central and the regional services together, while the real deployments need to manage them differently.)

Do we need to periodically test whether "local OpenBao unsealed via central OpenBao" still works as a concept? I don't see much value in that.

Also: Revoking the unsealing key in the central OpenBao is not enough to retire a node - the node can keep operating as long as it does not restart. See https://linear.app/filecoin-foundation/issue/FIL-1090 for more details.

adding an appliance to the network in the first instance is blocked by it being authorised by the central vault.

It all depends on how the appliance is configured. In particular, how the node operator configures their OpenBao instance.

  • The "canonical" recipe lives in infra-nodes, not Smelt.
  • Unless we have full control of the entire appliance stack, including all dependencies like OpenBao, we cannot rely just on the fact that our canonical recipe gives us this guarantee. We would need to implement some sort of audit to verify that the node is using the central OpenBao for unsealing.

Anyhow, I don't feel too strongly about this. If you see value in adding the central vault to Smelt deployments, then go for it.

| swarf | 15140 | UCAN revocation service |
| swarf-postgres | 15141 | Postgres for swarf's revocation store |
| guppy | — | CLI client for uploads (no exposed port) |
Expand Down
1 change: 1 addition & 0 deletions compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ include:
- path: systems/plc/compose.yml
- path: systems/swarf/compose.yml
- path: systems/hilt/compose.yml
- path: systems/central-openbao/compose.yml
- path: systems/ingot/compose.yml

networks:
Expand Down
6 changes: 4 additions & 2 deletions docs/SNAPSHOTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,8 +125,10 @@ generated/snapshots/<name>/
├── plc-postgres-data.tar.gz # DID PLC registry
├── ingot-data.tar.gz # ingot node state
├── ingot-postgres-data.tar.gz # ingot registry
├── ingot-openbao-data.tar.gz # ingot's region KEK (OpenBao raft storage)
├── ingot-openbao-init.tar.gz # its unseal share (dev-only custody)
├── ingot-openbao-data.tar.gz # ingot's region KEK (OpenBao raft, sealed by central)
├── ingot-openbao-init.tar.gz # its recovery share + root token (dev-only custody)
├── central-openbao-data.tar.gz # appliance seal keys (OpenBao raft storage)
├── central-openbao-init.tar.gz # its unseal share (dev-only custody)
├── piri-postgres-data.tar.gz # only when topology uses postgres
└── piri-minio-data.tar.gz # only when topology uses S3
```
Expand Down
4 changes: 4 additions & 0 deletions embed.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ import "embed"

//go:embed systems/common/compose.yml

//go:embed systems/central-openbao/compose.yml
//go:embed systems/central-openbao/config.hcl
//go:embed systems/central-openbao/init.sh

//go:embed systems/blockchain/compose.yml
//go:embed systems/blockchain/state/deployed-addresses.json
//go:embed systems/blockchain/state/anvil-state.json
Expand Down
24 changes: 13 additions & 11 deletions pkg/snapshot/volumes.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,17 +31,19 @@ func resolveVolumes(m *manifest.Manifest) ([]string, error) {
// stack delta" framing — a user who logged in and created spaces before
// saving will find them after load.
vols := []string{
"minio-data", // systems/common — upload's S3 backend
"ipni-data", // systems/indexing/ipni — content discovery
"dynamodb-data", // systems/common — delegator allow list, upload registry
"guppy-data", // systems/guppy — client's login/space state
"swarf-postgres-data", // systems/swarf — revocation records
"hilt-postgres-data", // systems/hilt — tenant state
"plc-postgres-data", // systems/plc — DID PLC registry
"ingot-data", // systems/ingot — node state
"ingot-postgres-data", // systems/ingot — registry
"ingot-openbao-data", // systems/ingot — region KEK (raft storage)
"ingot-openbao-init", // systems/ingot — unseal share for the above; restored together
"minio-data", // systems/common — upload's S3 backend
"ipni-data", // systems/indexing/ipni — content discovery
"dynamodb-data", // systems/common — delegator allow list, upload registry
"guppy-data", // systems/guppy — client's login/space state
"swarf-postgres-data", // systems/swarf — revocation records
"hilt-postgres-data", // systems/hilt — tenant state
"plc-postgres-data", // systems/plc — DID PLC registry
"ingot-data", // systems/ingot — node state
"ingot-postgres-data", // systems/ingot — registry
"ingot-openbao-data", // systems/ingot — region KEK (raft storage), sealed by central-openbao
"ingot-openbao-init", // systems/ingot — recovery share + root token for the above
"central-openbao-data", // systems/central-openbao — appliance seal keys (raft storage)
"central-openbao-init", // systems/central-openbao — unseal share for the above; restored together
}

// Per-piri data volumes.
Expand Down
2 changes: 2 additions & 0 deletions pkg/stack/stack_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@ func TestEmbeddedFilesExist(t *testing.T) {
"systems/upload/register-providers.sh",
"systems/ingot/openbao/config.hcl",
"systems/ingot/openbao/init.sh",
"systems/central-openbao/config.hcl",
"systems/central-openbao/init.sh",
}

for _, f := range files {
Expand Down
1 change: 1 addition & 0 deletions pkg/stack/testmode_env.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ func testModeEnv(nodes []manifest.ResolvedPiriNode) map[string]string {
"SMELT_INGOT_PORT": "9000",
"SMELT_INGOT_POSTGRES_PORT": "5432",
"SMELT_INGOT_OPENBAO_PORT": "8200",
"SMELT_CENTRAL_OPENBAO_PORT": "8200",
"SMELT_SWARF_PORT": "80",
"SMELT_SWARF_POSTGRES_PORT": "5432",

Expand Down
2 changes: 1 addition & 1 deletion snapshots/3-piri-postgres-s3/blockchain/anvil-state.json

Large diffs are not rendered by default.

12 changes: 11 additions & 1 deletion snapshots/3-piri-postgres-s3/manifest.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "3-piri-postgres-s3",
"created_at": "2026-08-26T06:48:09.354132Z",
"created_at": "2026-08-26T14:51:11.130639Z",
"volumes": [
"minio-data",
"ipni-data",
Expand All @@ -13,6 +13,8 @@
"ingot-postgres-data",
"ingot-openbao-data",
"ingot-openbao-init",
"central-openbao-data",
"central-openbao-init",
"piri-0-data",
"piri-1-data",
"piri-2-data",
Expand Down Expand Up @@ -75,6 +77,14 @@
"tag": "ghcr.io/fil-forge/filecoin-localdev:main",
"digest": "ghcr.io/fil-forge/filecoin-localdev@sha256:96e72624eb71efb1c15de1d0729b780955e5ece700e840dd56291f5c522b4bee"
},
"central-openbao": {
"tag": "openbao/openbao:2.6",
"digest": "openbao/openbao@sha256:11fd73a2102cda9c55d5d881a8c3210303146a7ec1e8ac76f526e175c6d24641"
},
"central-openbao-init": {
"tag": "openbao/openbao:2.6",
"digest": "openbao/openbao@sha256:11fd73a2102cda9c55d5d881a8c3210303146a7ec1e8ac76f526e175c6d24641"
},
"delegator": {
"tag": "ghcr.io/fil-forge/delegator:main",
"digest": "ghcr.io/fil-forge/delegator@sha256:b83f860bf91b27d33673db4d6f2458902abef16aa4a7c567ab4e60fb6dd65966"
Expand Down
Binary file not shown.
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/dynamodb-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/hilt-postgres-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/ingot-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/ingot-openbao-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/ingot-openbao-init.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/ingot-postgres-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/ipni-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/minio-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/piri-0-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/piri-1-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/piri-2-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/piri-minio-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/piri-postgres-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/plc-postgres-data.tar.gz
Binary file not shown.
Binary file modified snapshots/3-piri-postgres-s3/volumes/swarf-postgres-data.tar.gz
Binary file not shown.
79 changes: 79 additions & 0 deletions systems/central-openbao/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Central OpenBao (appliance seal root)

The central OpenBao of the
[regional security and key management RFC](https://github.com/fil-one/RFC/pull/21).
Each appliance runs its own OpenBao ([ingot-openbao](../ingot/) here) whose
storage is sealed by a transit key held on this instance. At boot the
appliance presents its seal token, central unwraps the appliance's barrier
key, and the appliance unseals. Revoking that token here is the RFC's
startup-kill lever: the appliance never starts again, and everything on its
disk stays unreadable.

This instance is deliberately separate from [hilt-vault](../hilt/). It is
the one Fil One endpoint that operator-controlled hardware talks to, so it
holds nothing else: no tenant keys, no Hilt state.

## Services

- **central-openbao** — `openbao/openbao:2.6` in server mode, raft storage
on `central-openbao-data`.
- **central-openbao-init** — one-shot: initializes and unseals
`central-openbao`, then provisions the `seal/` transit engine, the
appliance seal key, policy, and token (`init.sh`).

## Ports

| Host Port | Container Port | Service | Description |
|-----------|----------------|---------|-------------|
| 15210 | 8200 | central-openbao | OpenBao HTTP API |

## What init does

On every `make up`:

1. First boot: `bao operator init` with one unseal share. The share and the
root token are written to the `central-openbao-init` volume (`/init`).
Dev-only custody; production roots this instance in an HSM or KMS.
2. Every boot: unseal if sealed; enable transit at `seal/` if missing;
create `seal/keys/ingot-openbao` (`aes256-gcm96`, non-derived,
`exportable=false`) if missing; write the `appliance-ingot-openbao`
policy (`update` on `seal/encrypt/ingot-openbao` and
`seal/decrypt/ingot-openbao`, nothing else); recreate the appliance's
seal token: orphan, periodic (24h, renewed by the appliance's seal), fixed
id `dev-ingot-openbao-seal-token` (override with
`INGOT_OPENBAO_SEAL_TOKEN=... make up`; `systems/ingot/compose.yml` reads
the same variable).

`make clean` removes both volumes together, which keeps the unseal share and
the storage it opens in step.

## The kill lever

```bash
# Root token for admin actions (dev custody on the init volume; not printed here)
ROOT=$(docker run --rm -v smelt_central-openbao-init:/init alpine cat /init/root-token)

# Revoke the appliance: it fails to unseal on its next start
docker compose exec -e BAO_TOKEN="$ROOT" central-openbao bao token revoke dev-ingot-openbao-seal-token
docker compose restart ingot-openbao # stays sealed / exits with a seal error

# Reinstate: re-run the provisioner, then start the appliance again
docker compose up central-openbao-init
docker compose up -d ingot-openbao
```

## Production notes

Per the RFC, the production shape adds: TLS on the listener, an appliance
token bound to the region's egress CIDR (and single-use where the deployment
permits), an HSM or KMS root for this instance, and an operator for central
(an open question in the RFC). None of those change the appliance side.

## Volumes

- `central-openbao-data` - raft storage (appliance seal keys)
- `central-openbao-init` - unseal share + root token (dev-only custody)

## Used By

- ingot-openbao (`seal "transit"` in `systems/ingot/openbao/config.hcl`)
66 changes: 66 additions & 0 deletions systems/central-openbao/compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Central OpenBao System - seal root for the appliance OpenBaos
#
# The "central OpenBao" of the regional security RFC. It holds one transit
# key per appliance under `seal/`; each appliance's local OpenBao
# (ingot-openbao) is configured with `seal "transit"` against it and presents
# a scoped token at boot to unwrap its barrier key and unseal. Revoking that
# token here means the appliance never starts again (the RFC's startup-kill
# lever). Deliberately separate from hilt-vault: this is the one Fil One
# endpoint operator-controlled hardware talks to, so it holds nothing else.
# Used by: ingot-openbao (seal), and central-openbao-init (provisioning).

services:
central-openbao:
image: openbao/openbao:2.6
command: ["server"]
ports:
- "${SMELT_CENTRAL_OPENBAO_PORT:-15210:8200}"
environment:
# For `docker compose exec central-openbao bao ...` (the CLI defaults to https)
- BAO_ADDR=http://127.0.0.1:8200
volumes:
- ./config.hcl:/openbao/config/config.hcl:ro
- central-openbao-data:/openbao/file
# Healthy means listening: a fresh server is uninitialized and a
# restarted one is sealed until central-openbao-init runs again (which
# `make up` does, since one-shot services re-run on every up).
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:8200/v1/sys/health?sealedcode=200&uninitcode=200"]
start_interval: 1s
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
restart: unless-stopped
networks:
- forge-network

# One-shot initializer: `operator init` on first boot (one unseal share +
# root token on the central-openbao-init volume, dev-only custody), unseal
# on every boot, then make sure the `seal/` transit engine, the appliance
# seal key, the appliance policy (encrypt/decrypt on that key only), and the
# appliance's periodic seal token exist. See init.sh.
central-openbao-init:
image: openbao/openbao:2.6
user: "0:0" # the image runs as `openbao`; the init volume is root-owned
entrypoint: ["sh", "/init.sh"]
environment:
- BAO_ADDR=http://central-openbao:8200
- APPLIANCE_NAME=ingot-openbao
# Token ingot-openbao presents to unseal; local-dev value, override with
# INGOT_OPENBAO_SEAL_TOKEN. Must match the BAO_TOKEN ingot-openbao gets
# in systems/ingot/compose.yml.
- APPLIANCE_SEAL_TOKEN=${INGOT_OPENBAO_SEAL_TOKEN:-dev-ingot-openbao-seal-token}
volumes:
- ./init.sh:/init.sh:ro
- central-openbao-init:/init
depends_on:
central-openbao:
condition: service_healthy
restart: "no"
networks:
- forge-network

volumes:
central-openbao-data: # raft storage: appliance seal keys
central-openbao-init: # unseal share + root token (dev-only custody)
27 changes: 27 additions & 0 deletions systems/central-openbao/config.hcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# central-openbao server config (local dev).
#
# The central OpenBao of the regional security RFC: it holds the transit
# keys that seal each appliance's local OpenBao (ingot-openbao here). At
# boot an appliance presents its token, central unwraps the appliance's
# barrier key, and the appliance unseals; revoking that token at central is
# the startup-kill lever. This instance holds nothing else: hilt's tenant
# keys stay in hilt-vault.
#
# Production differs in the root of trust for this instance (HSM/KMS unseal
# instead of a stored share), TLS on the listener, and CIDR-bound appliance
# tokens. All of those are changes to this file or to init.sh.

ui = false

api_addr = "http://central-openbao:8200"
cluster_addr = "http://central-openbao:8201"

listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}

storage "raft" {
path = "/openbao/file"
node_id = "central-openbao"
}
Loading