From e98c7bc148fe890e7d0782afbcc8ffa0c8ade237 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 29 Jun 2026 07:23:47 +0000 Subject: [PATCH 1/2] Consolidate docs: delete architecture/ and old docs/, rename docs-site/ to docs/ The Astro Starlight site (formerly docs-site/) is now the single home for documentation, published to GitHub Pages. Remove the superseded legacy hand-built architecture/ HTML site and the old docs/ folder (ADRs, design docs, documentation roadmap), and rename docs-site/ to docs/. - Delete architecture/ (legacy site) and docs/ (adr, design, roadmap). - git mv docs-site/ -> docs/. - Workflows: drop obsolete pages.yml (architecture/ deploy); repoint docs-site.yml and docs-ci.yml at docs/ and remove the architecture-legacy bundling/triggers; strip architecture/ from doc-update and ai-review paths-ignore. - .ai-reviewer.yaml: remove the architecture/ static_docs_dirs and the source_to_docs_mapping pointing at deleted architecture/*.html pages. - README + docs site content: repoint docs-site/ references to docs/ and drop links to the removed ADR page, architecture-legacy site, and roadmap. - Source doc-comments: strip now-dead references to architecture/*.html and docs/adr/*.md; point the SDK migrations note at the published docs page. Verified: `npm run check` (astro build + internal link check) passes. --- .ai-reviewer.yaml | 45 +- .github/workflows/ai-review.yaml | 3 +- .github/workflows/doc-update.yaml | 1 - .github/workflows/docs-ci.yml | 8 +- .github/workflows/docs-site.yml | 25 +- .github/workflows/pages.yml | 32 - README.md | 4 +- architecture/abi-conformance.html | 163 -- architecture/app-lifecycle.html | 189 -- architecture/auto-follow.html | 817 ------- architecture/concepts.html | 171 -- architecture/config-reference.html | 693 ------ architecture/crates/auth.html | 347 --- architecture/crates/context.html | 691 ------ architecture/crates/dag.html | 173 -- architecture/crates/network.html | 500 ----- architecture/crates/node.html | 561 ----- architecture/crates/runtime.html | 296 --- architecture/crates/sdk.html | 703 ------ architecture/crates/server.html | 547 ----- architecture/crates/store.html | 592 ----- architecture/crates/sync.html | 783 ------- architecture/crates/tools.html | 598 ----- architecture/dependency-explorer.html | 465 ---- architecture/error-flows.html | 706 ------ architecture/example-chat.html | 156 -- architecture/example-docs.html | 182 -- architecture/getting-started.html | 66 - architecture/glossary.html | 946 -------- architecture/index.html | 195 -- architecture/local-governance.html | 641 ------ architecture/membership-and-leave.html | 1941 ----------------- architecture/metrics-reference.html | 510 ----- architecture/migrations.html | 968 -------- architecture/nav.js | 452 ---- architecture/protocol/concepts.html | 128 -- architecture/protocol/execution.html | 226 -- architecture/protocol/governance.html | 204 -- architecture/protocol/identities.html | 79 - architecture/protocol/index.html | 152 -- architecture/protocol/networking.html | 176 -- architecture/protocol/operations.html | 264 --- architecture/protocol/projection.html | 223 -- architecture/protocol/receive-path.html | 79 - architecture/protocol/storage.html | 114 - architecture/protocol/sync.html | 190 -- architecture/protocol/write-path.html | 82 - architecture/release.html | 198 -- architecture/seq-diagram.js | 133 -- architecture/sequence-diagrams.html | 1090 --------- architecture/storage-schema.html | 724 ------ architecture/styles.css | 824 ------- architecture/system-overview.html | 716 ------ architecture/tee-fleet-ha.html | 406 ---- architecture/tee-mode.html | 126 -- .../unified-causal-log-cutover-plan.html | 212 -- architecture/wire-protocol.html | 971 --------- crates/client/src/client/group.rs | 6 +- crates/context/src/auto_follow.rs | 3 +- crates/context/src/handlers/leave_context.rs | 2 - crates/context/src/handlers/leave_group.rs | 4 +- .../context/src/handlers/leave_namespace.rs | 4 +- crates/context/src/lib.rs | 2 +- crates/context/src/self_purge.rs | 3 - crates/governance-store/src/op_events.rs | 3 +- crates/governance-types/src/lib.rs | 2 - crates/sdk/AGENTS.md | 6 +- crates/storage/src/rotation_log.rs | 3 +- crates/store/src/key/group/mod.rs | 3 - docs-site/README.md | 32 - .../src/content/docs/contribute/adrs.mdx | 39 - {docs-site => docs}/.gitignore | 0 {docs-site => docs}/CODE-DOC-GAPS.md | 0 {docs-site => docs}/IMPROVEMENTS.md | 5 +- docs/README.md | 26 + ...0001-shared-storage-concurrent-rotation.md | 195 -- docs/adr/0002-fleet-tee-leave-protocol.md | 133 -- {docs-site => docs}/astro.config.mjs | 3 +- .../2230-shared-wrapper-verification.md | 196 -- docs/design/calimero-components.md | 575 ----- docs/design/shared-collection-guarding.md | 131 -- .../design/unified-causal-log-cutover-plan.md | 305 --- docs/design/unified-causal-log-flip-plan.md | 118 - .../design/unified-causal-log-p5-decisions.md | 261 --- docs/documentation-roadmap.md | 337 --- {docs-site => docs}/package-lock.json | 0 {docs-site => docs}/package.json | 2 +- {docs-site => docs}/public/favicon.svg | 0 {docs-site => docs}/public/robots.txt | 0 {docs-site => docs}/scripts/check-links.mjs | 0 {docs-site => docs}/src/assets/logo.svg | 0 .../src/components/ArchMap.astro | 0 .../src/components/Figure.astro | 0 .../src/components/SeqDiagram.astro | 0 .../src/components/diagrams/AbiBoundary.astro | 0 .../src/components/diagrams/ActorModel.astro | 0 .../src/components/diagrams/CausalDag.astro | 0 .../components/diagrams/ContextInMemory.astro | 0 .../src/components/diagrams/CrateStack.astro | 0 .../src/components/diagrams/DagVsState.astro | 0 .../components/diagrams/EntityStorage.astro | 0 .../components/diagrams/FoldPipeline.astro | 0 .../src/components/diagrams/KeyLayers.astro | 0 .../src/components/diagrams/Libp2pStack.astro | 0 .../diagrams/MerkleTreeMemory.astro | 0 .../components/diagrams/ScopeHierarchy.astro | 0 .../src/components/diagrams/ScopeTree.astro | 0 .../components/diagrams/StateProduction.astro | 0 .../src/components/diagrams/StoreLayout.astro | 0 .../src/components/diagrams/TreeMerge.astro | 0 .../src/components/diagrams/Verdict.astro | 0 {docs-site => docs}/src/content.config.ts | 0 .../src/content/docs/build/advanced-sdk.mdx | 0 .../src/content/docs/build/app-abi.mdx | 0 .../src/content/docs/build/collections.mdx | 0 .../src/content/docs/build/error-handling.mdx | 0 .../src/content/docs/build/examples.mdx | 0 .../src/content/docs/build/gotchas.mdx | 0 .../docs/build/guides/access-control.mdx | 0 .../src/content/docs/build/guides/blobs.mdx | 0 .../content/docs/build/guides/collections.mdx | 0 .../docs/build/guides/cross-context.mdx | 0 .../src/content/docs/build/guides/events.mdx | 0 .../src/content/docs/build/guides/index.mdx | 0 .../src/content/docs/build/host-functions.mdx | 0 .../src/content/docs/build/index.mdx | 0 .../src/content/docs/build/migrations.mdx | 0 .../content/docs/build/packaging-signing.mdx | 0 .../docs/build/permissioned-storage.mdx | 3 +- .../src/content/docs/build/quickstart.mdx | 0 .../src/content/docs/build/sdk-macros.mdx | 0 .../src/content/docs/build/state-modeling.mdx | 0 .../content/docs/build/storage-complexity.mdx | 0 .../src/content/docs/build/testing.mdx | 0 .../src/content/docs/build/tutorial.mdx | 0 .../content/docs/contribute/architecture.mdx | 6 +- .../content/docs/contribute/crate-guide.mdx | 0 .../content/docs/contribute/development.mdx | 8 +- .../src/content/docs/contribute/docs.mdx | 8 +- .../src/content/docs/contribute/index.mdx | 6 - .../src/content/docs/contribute/testing.mdx | 0 .../src/content/docs/index.mdx | 0 .../src/content/docs/journeys.mdx | 0 .../src/content/docs/operate/admin-api.mdx | 0 .../src/content/docs/operate/auth.mdx | 0 .../src/content/docs/operate/config.mdx | 0 .../src/content/docs/operate/deployment.mdx | 0 .../src/content/docs/operate/index.mdx | 0 .../src/content/docs/operate/install.mdx | 0 .../src/content/docs/operate/meroctl.mdx | 0 .../src/content/docs/operate/merod.mdx | 0 .../src/content/docs/operate/networking.mdx | 0 .../content/docs/operate/observability.mdx | 0 .../src/content/docs/operate/runbooks.mdx | 0 .../src/content/docs/operate/security.mdx | 0 .../content/docs/operate/troubleshooting.mdx | 0 .../content/docs/protocol/applications.mdx | 0 .../src/content/docs/protocol/blobs.mdx | 0 .../docs/protocol/capability-inheritance.mdx | 0 .../src/content/docs/protocol/concepts.mdx | 0 .../docs/protocol/context-lifecycle.mdx | 0 .../content/docs/protocol/crdt-internals.mdx | 0 .../content/docs/protocol/data-anatomy.mdx | 0 .../docs/protocol/divergence-recovery.mdx | 0 .../src/content/docs/protocol/encryption.mdx | 0 .../src/content/docs/protocol/execution.mdx | 0 .../src/content/docs/protocol/glossary.mdx | 0 .../docs/protocol/governance-edge-cases.mdx | 0 .../src/content/docs/protocol/governance.mdx | 0 .../src/content/docs/protocol/hlc.mdx | 0 .../src/content/docs/protocol/identities.mdx | 0 .../content/docs/protocol/key-rotation.mdx | 0 .../src/content/docs/protocol/limits.mdx | 0 .../src/content/docs/protocol/networking.mdx | 0 .../src/content/docs/protocol/operations.mdx | 0 .../src/content/docs/protocol/overview.mdx | 0 .../src/content/docs/protocol/projection.mdx | 0 .../content/docs/protocol/receive-path.mdx | 0 .../content/docs/protocol/security-model.mdx | 0 .../src/content/docs/protocol/storage.mdx | 0 .../content/docs/protocol/sync-internals.mdx | 0 .../src/content/docs/protocol/sync.mdx | 0 .../content/docs/protocol/tee-attestation.mdx | 0 .../src/content/docs/protocol/upgrades.mdx | 0 .../src/content/docs/protocol/write-path.mdx | 0 .../src/content/docs/protocol/xcall.mdx | 0 .../src/content/docs/topics.mdx | 0 .../src/content/docs/using-the-docs.mdx | 0 {docs-site => docs}/src/middleware.ts | 0 {docs-site => docs}/src/pages/llms.txt.ts | 0 .../src/scripts/diagrams.client.ts | 0 {docs-site => docs}/src/styles/theme.css | 0 {docs-site => docs}/tsconfig.json | 0 193 files changed, 68 insertions(+), 24849 deletions(-) delete mode 100644 .github/workflows/pages.yml delete mode 100644 architecture/abi-conformance.html delete mode 100644 architecture/app-lifecycle.html delete mode 100644 architecture/auto-follow.html delete mode 100644 architecture/concepts.html delete mode 100644 architecture/config-reference.html delete mode 100644 architecture/crates/auth.html delete mode 100644 architecture/crates/context.html delete mode 100644 architecture/crates/dag.html delete mode 100644 architecture/crates/network.html delete mode 100644 architecture/crates/node.html delete mode 100644 architecture/crates/runtime.html delete mode 100644 architecture/crates/sdk.html delete mode 100644 architecture/crates/server.html delete mode 100644 architecture/crates/store.html delete mode 100644 architecture/crates/sync.html delete mode 100644 architecture/crates/tools.html delete mode 100644 architecture/dependency-explorer.html delete mode 100644 architecture/error-flows.html delete mode 100644 architecture/example-chat.html delete mode 100644 architecture/example-docs.html delete mode 100644 architecture/getting-started.html delete mode 100644 architecture/glossary.html delete mode 100644 architecture/index.html delete mode 100644 architecture/local-governance.html delete mode 100644 architecture/membership-and-leave.html delete mode 100644 architecture/metrics-reference.html delete mode 100644 architecture/migrations.html delete mode 100644 architecture/nav.js delete mode 100644 architecture/protocol/concepts.html delete mode 100644 architecture/protocol/execution.html delete mode 100644 architecture/protocol/governance.html delete mode 100644 architecture/protocol/identities.html delete mode 100644 architecture/protocol/index.html delete mode 100644 architecture/protocol/networking.html delete mode 100644 architecture/protocol/operations.html delete mode 100644 architecture/protocol/projection.html delete mode 100644 architecture/protocol/receive-path.html delete mode 100644 architecture/protocol/storage.html delete mode 100644 architecture/protocol/sync.html delete mode 100644 architecture/protocol/write-path.html delete mode 100644 architecture/release.html delete mode 100644 architecture/seq-diagram.js delete mode 100644 architecture/sequence-diagrams.html delete mode 100644 architecture/storage-schema.html delete mode 100644 architecture/styles.css delete mode 100644 architecture/system-overview.html delete mode 100644 architecture/tee-fleet-ha.html delete mode 100644 architecture/tee-mode.html delete mode 100644 architecture/unified-causal-log-cutover-plan.html delete mode 100644 architecture/wire-protocol.html delete mode 100644 docs-site/README.md delete mode 100644 docs-site/src/content/docs/contribute/adrs.mdx rename {docs-site => docs}/.gitignore (100%) rename {docs-site => docs}/CODE-DOC-GAPS.md (100%) rename {docs-site => docs}/IMPROVEMENTS.md (99%) create mode 100644 docs/README.md delete mode 100644 docs/adr/0001-shared-storage-concurrent-rotation.md delete mode 100644 docs/adr/0002-fleet-tee-leave-protocol.md rename {docs-site => docs}/astro.config.mjs (98%) delete mode 100644 docs/design/2230-shared-wrapper-verification.md delete mode 100644 docs/design/calimero-components.md delete mode 100644 docs/design/shared-collection-guarding.md delete mode 100644 docs/design/unified-causal-log-cutover-plan.md delete mode 100644 docs/design/unified-causal-log-flip-plan.md delete mode 100644 docs/design/unified-causal-log-p5-decisions.md delete mode 100644 docs/documentation-roadmap.md rename {docs-site => docs}/package-lock.json (100%) rename {docs-site => docs}/package.json (82%) rename {docs-site => docs}/public/favicon.svg (100%) rename {docs-site => docs}/public/robots.txt (100%) rename {docs-site => docs}/scripts/check-links.mjs (100%) rename {docs-site => docs}/src/assets/logo.svg (100%) rename {docs-site => docs}/src/components/ArchMap.astro (100%) rename {docs-site => docs}/src/components/Figure.astro (100%) rename {docs-site => docs}/src/components/SeqDiagram.astro (100%) rename {docs-site => docs}/src/components/diagrams/AbiBoundary.astro (100%) rename {docs-site => docs}/src/components/diagrams/ActorModel.astro (100%) rename {docs-site => docs}/src/components/diagrams/CausalDag.astro (100%) rename {docs-site => docs}/src/components/diagrams/ContextInMemory.astro (100%) rename {docs-site => docs}/src/components/diagrams/CrateStack.astro (100%) rename {docs-site => docs}/src/components/diagrams/DagVsState.astro (100%) rename {docs-site => docs}/src/components/diagrams/EntityStorage.astro (100%) rename {docs-site => docs}/src/components/diagrams/FoldPipeline.astro (100%) rename {docs-site => docs}/src/components/diagrams/KeyLayers.astro (100%) rename {docs-site => docs}/src/components/diagrams/Libp2pStack.astro (100%) rename {docs-site => docs}/src/components/diagrams/MerkleTreeMemory.astro (100%) rename {docs-site => docs}/src/components/diagrams/ScopeHierarchy.astro (100%) rename {docs-site => docs}/src/components/diagrams/ScopeTree.astro (100%) rename {docs-site => docs}/src/components/diagrams/StateProduction.astro (100%) rename {docs-site => docs}/src/components/diagrams/StoreLayout.astro (100%) rename {docs-site => docs}/src/components/diagrams/TreeMerge.astro (100%) rename {docs-site => docs}/src/components/diagrams/Verdict.astro (100%) rename {docs-site => docs}/src/content.config.ts (100%) rename {docs-site => docs}/src/content/docs/build/advanced-sdk.mdx (100%) rename {docs-site => docs}/src/content/docs/build/app-abi.mdx (100%) rename {docs-site => docs}/src/content/docs/build/collections.mdx (100%) rename {docs-site => docs}/src/content/docs/build/error-handling.mdx (100%) rename {docs-site => docs}/src/content/docs/build/examples.mdx (100%) rename {docs-site => docs}/src/content/docs/build/gotchas.mdx (100%) rename {docs-site => docs}/src/content/docs/build/guides/access-control.mdx (100%) rename {docs-site => docs}/src/content/docs/build/guides/blobs.mdx (100%) rename {docs-site => docs}/src/content/docs/build/guides/collections.mdx (100%) rename {docs-site => docs}/src/content/docs/build/guides/cross-context.mdx (100%) rename {docs-site => docs}/src/content/docs/build/guides/events.mdx (100%) rename {docs-site => docs}/src/content/docs/build/guides/index.mdx (100%) rename {docs-site => docs}/src/content/docs/build/host-functions.mdx (100%) rename {docs-site => docs}/src/content/docs/build/index.mdx (100%) rename {docs-site => docs}/src/content/docs/build/migrations.mdx (100%) rename {docs-site => docs}/src/content/docs/build/packaging-signing.mdx (100%) rename {docs-site => docs}/src/content/docs/build/permissioned-storage.mdx (99%) rename {docs-site => docs}/src/content/docs/build/quickstart.mdx (100%) rename {docs-site => docs}/src/content/docs/build/sdk-macros.mdx (100%) rename {docs-site => docs}/src/content/docs/build/state-modeling.mdx (100%) rename {docs-site => docs}/src/content/docs/build/storage-complexity.mdx (100%) rename {docs-site => docs}/src/content/docs/build/testing.mdx (100%) rename {docs-site => docs}/src/content/docs/build/tutorial.mdx (100%) rename {docs-site => docs}/src/content/docs/contribute/architecture.mdx (96%) rename {docs-site => docs}/src/content/docs/contribute/crate-guide.mdx (100%) rename {docs-site => docs}/src/content/docs/contribute/development.mdx (91%) rename {docs-site => docs}/src/content/docs/contribute/docs.mdx (91%) rename {docs-site => docs}/src/content/docs/contribute/index.mdx (89%) rename {docs-site => docs}/src/content/docs/contribute/testing.mdx (100%) rename {docs-site => docs}/src/content/docs/index.mdx (100%) rename {docs-site => docs}/src/content/docs/journeys.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/admin-api.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/auth.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/config.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/deployment.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/index.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/install.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/meroctl.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/merod.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/networking.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/observability.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/runbooks.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/security.mdx (100%) rename {docs-site => docs}/src/content/docs/operate/troubleshooting.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/applications.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/blobs.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/capability-inheritance.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/concepts.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/context-lifecycle.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/crdt-internals.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/data-anatomy.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/divergence-recovery.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/encryption.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/execution.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/glossary.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/governance-edge-cases.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/governance.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/hlc.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/identities.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/key-rotation.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/limits.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/networking.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/operations.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/overview.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/projection.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/receive-path.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/security-model.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/storage.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/sync-internals.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/sync.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/tee-attestation.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/upgrades.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/write-path.mdx (100%) rename {docs-site => docs}/src/content/docs/protocol/xcall.mdx (100%) rename {docs-site => docs}/src/content/docs/topics.mdx (100%) rename {docs-site => docs}/src/content/docs/using-the-docs.mdx (100%) rename {docs-site => docs}/src/middleware.ts (100%) rename {docs-site => docs}/src/pages/llms.txt.ts (100%) rename {docs-site => docs}/src/scripts/diagrams.client.ts (100%) rename {docs-site => docs}/src/styles/theme.css (100%) rename {docs-site => docs}/tsconfig.json (100%) diff --git a/.ai-reviewer.yaml b/.ai-reviewer.yaml index 43d535657a..b7791e27a3 100644 --- a/.ai-reviewer.yaml +++ b/.ai-reviewer.yaml @@ -14,50 +14,17 @@ anthropic: default_model: claude-sonnet-4-6 enable_prompt_caching: true -# Doc auto-update — opens a draft PR on merges to master with proposed -# changes to the architecture site (and any Markdown docs that go stale). -# Disabled by default upstream; enabled here so doc-update.yaml has work to do. -# Disabled at the docs cutover: the published docs are now the hand-authored -# Starlight site in docs-site/, not the auto-generated architecture/ HTML. -# Re-enable only if the bot is repointed at generating reference *data* for the -# new site (see the docs-generation plan), not free-form HTML. +# Doc auto-update — historically opened a draft PR on merges to master with +# proposed changes to the generated architecture site. Disabled at the docs +# cutover: the published docs are now the hand-authored Starlight site in +# docs/, not auto-generated HTML, and the legacy architecture/ site has been +# removed. Re-enable only if the bot is repointed at generating reference +# *data* for the new site, not free-form HTML. doc_generation: enabled: false model: claude-sonnet-4-6 # Sonnet: Haiku failed the HTML find/replace patch task max_files: 15 - static_docs_dirs: - - architecture/ # GitHub Pages source for the architecture site pr_labels: - automated-docs - documentation pr_draft: true - -# Source → docs routing. A change whose files match a glob routes deterministically -# to the listed page(s) with no model guess (the fast mapping-hit path); unmapped -# crates fall back to model routing. Globs are fnmatch-style (`*` spans `/`). -documentation: - source_to_docs_mapping: - # Per-crate internals pages (architecture/crates/*.html) - "crates/auth/**": ["architecture/crates/auth.html"] - "crates/node/**": ["architecture/crates/node.html", "architecture/crates/sync.html"] - "crates/context/**": ["architecture/crates/context.html", "architecture/membership-and-leave.html"] - "crates/network/**": ["architecture/crates/network.html", "architecture/wire-protocol.html"] - "crates/server/**": ["architecture/crates/server.html"] - "crates/runtime/**": ["architecture/crates/runtime.html", "architecture/app-lifecycle.html"] - "crates/dag/**": ["architecture/crates/dag.html"] - "crates/sdk/**": ["architecture/crates/sdk.html"] - "crates/meroctl/**": ["architecture/crates/tools.html"] - "crates/merod/**": ["architecture/crates/tools.html"] - # Storage - "crates/store/**": ["architecture/crates/store.html", "architecture/storage-schema.html"] - "crates/storage/**": ["architecture/storage-schema.html", "architecture/migrations.html"] - "crates/storage-macros/**": ["architecture/storage-schema.html"] - # Governance & op-log - "crates/governance-store/**": ["architecture/local-governance.html", "architecture/auto-follow.html"] - "crates/governance-types/**": ["architecture/local-governance.html"] - "crates/authz/**": ["architecture/local-governance.html"] - "crates/op/**": ["architecture/crates/dag.html", "architecture/auto-follow.html"] - "crates/op-adapter/**": ["architecture/crates/dag.html"] - # Subsystems - "crates/config/**": ["architecture/config-reference.html"] - "crates/tee-attestation/**": ["architecture/tee-mode.html"] diff --git a/.github/workflows/ai-review.yaml b/.github/workflows/ai-review.yaml index 8830c2c073..e2d018f4a4 100644 --- a/.github/workflows/ai-review.yaml +++ b/.github/workflows/ai-review.yaml @@ -6,13 +6,12 @@ on: # cost — a review fires on PR open / reopen / draft→ready, not on each # push. Re-trigger a review by toggling draft or via workflow_dispatch. types: [opened, reopened, ready_for_review] - # Skipped only when ALL changed files are under docs/architecture/docs-static — + # Skipped only when ALL changed files are under docs/docs-static — # mixed PRs still trigger a review (standard GitHub Actions paths-ignore semantics). # Pure-doc PRs are owned by the `Doc Auto-Update` workflow; this avoids paying # twice for the same diff. paths-ignore: - 'docs/**' - - 'architecture/**' - 'docs-static/**' workflow_dispatch: inputs: diff --git a/.github/workflows/doc-update.yaml b/.github/workflows/doc-update.yaml index fa59abac0b..41be9092fc 100644 --- a/.github/workflows/doc-update.yaml +++ b/.github/workflows/doc-update.yaml @@ -22,7 +22,6 @@ on: # infinite loop. Mirrors the paths-ignore in ai-review.yaml. paths-ignore: - 'docs/**' - - 'architecture/**' - 'docs-static/**' workflow_dispatch: inputs: diff --git a/.github/workflows/docs-ci.yml b/.github/workflows/docs-ci.yml index 797944ff17..06d04e1055 100644 --- a/.github/workflows/docs-ci.yml +++ b/.github/workflows/docs-ci.yml @@ -4,10 +4,10 @@ name: Docs CI # touches it, so the docs can't silently break. Content-only; no deploy. on: pull_request: - paths: ['docs-site/**'] + paths: ['docs/**'] push: branches: [master] - paths: ['docs-site/**'] + paths: ['docs/**'] permissions: contents: read @@ -21,13 +21,13 @@ jobs: runs-on: ubuntu-latest defaults: run: - working-directory: docs-site + working-directory: docs steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 cache: npm - cache-dependency-path: docs-site/package-lock.json + cache-dependency-path: docs/package-lock.json - run: npm ci - run: npm run check # astro build + internal link check diff --git a/.github/workflows/docs-site.yml b/.github/workflows/docs-site.yml index f5beb12c93..d009db8a84 100644 --- a/.github/workflows/docs-site.yml +++ b/.github/workflows/docs-site.yml @@ -1,16 +1,12 @@ name: Deploy Docs Site (Starlight) -# Builds the Astro Starlight documentation site (docs-site/) and deploys it to -# GitHub Pages — this is now the PUBLISHED docs (it replaced the legacy -# architecture/ site). It publishes Starlight at the Pages root and preserves -# the old hand-built site under /architecture-legacy/ so nothing is lost. -# Rollback: run the (now manual-only) `pages.yml` to republish architecture/. +# Builds the Astro Starlight documentation site (docs/) and deploys it to +# GitHub Pages — this is the published documentation. on: push: branches: [master] paths: - - 'docs-site/**' - - 'architecture/**' + - 'docs/**' - '.github/workflows/docs-site.yml' workflow_dispatch: @@ -19,7 +15,6 @@ permissions: pages: write id-token: write -# Shares the `pages` concurrency group with pages.yml so the two never race. concurrency: group: pages cancel-in-progress: true @@ -37,25 +32,19 @@ jobs: with: node-version: 20 cache: npm - cache-dependency-path: docs-site/package-lock.json + cache-dependency-path: docs/package-lock.json - name: Install - working-directory: docs-site + working-directory: docs run: npm ci - name: Build - working-directory: docs-site + working-directory: docs run: npm run build - # Keep the legacy hand-built site reachable during migration. - - name: Bundle legacy architecture site - run: | - mkdir -p docs-site/dist/architecture-legacy - cp -r architecture/* docs-site/dist/architecture-legacy/ - - uses: actions/configure-pages@v5 - uses: actions/upload-pages-artifact@v3 with: - path: docs-site/dist + path: docs/dist - id: deployment uses: actions/deploy-pages@v4 diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml deleted file mode 100644 index 148e523c75..0000000000 --- a/.github/workflows/pages.yml +++ /dev/null @@ -1,32 +0,0 @@ -name: Deploy Architecture Site (legacy — manual rollback only) - -# Demoted to manual-only: the Starlight site (docs-site.yml) is now the -# published documentation. Keep this workflow as an emergency rollback — run it -# via "Run workflow" to republish the legacy architecture/ site at the Pages -# root. -on: - workflow_dispatch: - -permissions: - contents: read - pages: write - id-token: write - -concurrency: - group: pages - cancel-in-progress: true - -jobs: - deploy: - environment: - name: github-pages - url: ${{ steps.deployment.outputs.page_url }} - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/configure-pages@v5 - - uses: actions/upload-pages-artifact@v3 - with: - path: architecture - - id: deployment - uses: actions/deploy-pages@v4 diff --git a/README.md b/README.md index 180dcc8dec..d7987b345c 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,7 @@ Peer-to-peer application platform with local-first governance, CRDT state sync, | Work on Calimero Core itself | [Contribute](https://calimero-network.github.io/core/contribute/) — architecture, crate guide, dev workflow | | Look up a term | [Glossary](https://calimero-network.github.io/core/protocol/glossary/) | -The docs live in [`docs-site/`](docs-site/) (Astro Starlight). The previous -hand-built site is preserved at -[`/architecture-legacy/`](https://calimero-network.github.io/core/architecture-legacy/). +The docs live in [`docs/`](docs/) (Astro Starlight). ## Related Repositories diff --git a/architecture/abi-conformance.html b/architecture/abi-conformance.html deleted file mode 100644 index 2e5c0b1c13..0000000000 --- a/architecture/abi-conformance.html +++ /dev/null @@ -1,163 +0,0 @@ - - - - -ABI Conformance — Calimero Core Architecture - - - -
-
-
Home/ABI Conformance
- -

ABI Conformance

-

AuthoredMap, AuthoredVector, and the full CRDT collection coverage gate

- -
-

What changed

-

The AbiState struct inside the abi_conformance app gained two new fields:

-
    -
  • authored_counters: AuthoredMap<String, LwwRegister<u32>>
    • -
  • authored_log: AuthoredVector<LwwRegister<UserId32>>
    • -
-

The golden file abi.expected.json was updated to include the corresponding ABI entries for both fields. With this change the conformance app now emits all 11 CrdtCollectionType variants, closing the last gap in the cross-repo coverage gate.

-
- -
-

Why these two types were missing

-

The cross-repo coverage gate requires every schema-declared CrdtType to be exercised by at least one corpus ABI. Before this change, authored_map and authored_vector were never emitted by any conformance corpus, so they had to be listed in an explicit allowlist exemption inside the gate script.

-
    -
  • Allowlist exemptions are fragile — a new variant added to the schema would silently pass the gate unless the exemption list was manually updated.
    • -
  • Coverage should be structural — the conformance app is the designated place to lock every marker; the exemption mechanism should remain empty in steady state.
    • -
  • Adding the two fields removes both exemptions and makes the gate self-maintaining: any future CrdtCollectionType that lacks a matching conformance field will now fail the gate rather than slip through silently.
    • -
-
- -
-

The full set of 11 CrdtCollectionType variants

-

After this change the conformance app covers every declared variant. The table below lists them all for reference:

-
    -
  1. lww_register
    2. -
  2. counter
    3. -
  3. set
    4. -
  4. map
    5. -
  5. vector
    6. -
  6. priority_queue
    7. -
  7. queue
    8. -
  8. text
    9. -
  9. graph
    10. -
  10. authored_map — newly covered
    11. -
  11. authored_vector — newly covered
    12. -
-
- -
-

AbiState struct — before and after

-

Before

-
#[app::state]
-pub struct AbiState {
-    // … existing fields covering variants 1–9 …
-}
-

After

-
#[app::state]
-pub struct AbiState {
-    // … existing fields covering variants 1–9 …
-
-    /// Covers CrdtCollectionType::authored_map
-    authored_counters: AuthoredMap<String, LwwRegister<u32>>,
-
-    /// Covers CrdtCollectionType::authored_vector
-    authored_log: AuthoredVector<LwwRegister<UserId32>>,
-}
-

Both fields follow the same nesting pattern already used by map and vector: an outer collection type wrapping an inner LwwRegister value.

-
- -
-

Golden file update — abi.expected.json

-

The expected ABI snapshot must be regenerated whenever AbiState changes. Two new entries are appended to the fields array:

-
// authored_counters entry
-{
-  "name": "authored_counters",
-  "type": {
-    "collection": "authored_map",
-    "key": { "primitive": "string" },
-    "value": {
-      "collection": "lww_register",
-      "inner": { "primitive": "u32" }
-    }
-  }
-},
-
-// authored_log entry
-{
-  "name": "authored_log",
-  "type": {
-    "collection": "authored_vector",
-    "inner": {
-      "collection": "lww_register",
-      "inner": { "primitive": "user_id32" }
-    }
-  }
-}
-

Run cargo test -p abi_conformance -- --nocapture to regenerate the snapshot and confirm it matches the golden file.

-
- -
-

How the coverage gate works

-

The gate is a CI step that compares the set of CrdtCollectionType variants declared in the schema crate against the set of variants actually emitted by every ABI file in the corpus directory.

-
    -
  • Schema source — the CrdtCollectionType enum in crates/primitives/src/crdt.rs (or equivalent).
    • -
  • Corpus — every abi.json and abi.expected.json file discovered under apps/.
    • -
  • Pass condition — the union of all collection markers found in the corpus equals the full variant set; the allowlist of exemptions must be empty.
    • -
  • Fail condition — any variant present in the schema but absent from every corpus ABI causes the gate to fail with a diff listing the missing markers.
    • -
-
- -
-

AuthoredMap and AuthoredVector semantics

-

AuthoredMap

-

An AuthoredMap<K, V> is a key-value collection where every entry carries the UserId of the peer that last wrote it. Conflict resolution is author-aware: writes from the designated author of a key take precedence over writes from other peers, falling back to LWW semantics when authorship is equal.

-

AuthoredVector

-

An AuthoredVector<V> is an append-only sequence where each element is stamped with the inserting peer's identity. Order is deterministic across replicas: elements are sorted first by logical timestamp, then by author id to break ties.

-

Both types are generic over their value type, which may itself be a CRDT (as in the conformance app where the value is LwwRegister<_>), enabling arbitrarily nested CRDT compositions.

-
- -
-

Adding a new CrdtCollectionType in the future

-

The conformance app is now the canonical place to lock every CRDT collection marker. The required steps when introducing a new variant are:

-
    -
  1. Add the variant to the CrdtCollectionType enum in the schema crate.
    2. -
  2. Implement the corresponding collection type in the CRDT runtime crate.
    3. -
  3. Add a field of that type to AbiState in apps/abi_conformance/src/lib.rs.
    4. -
  4. Regenerate abi.expected.json by running the conformance snapshot test.
    5. -
  5. Verify the coverage gate passes with an empty allowlist — do not add an exemption.
    6. -
-

Never add a new variant to the schema without a matching conformance field. A CI failure from the gate is the intended signal that step 3 was skipped.

-
- -
-

File locations

-
    -
  • apps/abi_conformance/src/lib.rs — AbiState struct definition; add new fields here.
    • -
  • apps/abi_conformance/abi.expected.json — golden ABI snapshot; regenerate after struct changes.
    • -
  • crates/primitives/src/crdt.rs — authoritative CrdtCollectionType enum.
    • -
  • scripts/check_crdt_coverage.sh (or equivalent CI step) — the coverage gate; allowlist must stay empty.
    • -
-
- -
-

Relationship to other conformance corpus entries

-

The conformance app is intentionally not a realistic application. Its sole purpose is to ensure every ABI-visible type appears at least once in a well-formed, compilable Rust struct so that:

-
    -
  • The ABI serializer is exercised for every variant on every CI run.
    • -
  • The golden snapshot acts as a regression test for ABI format changes.
    • -
  • Cross-language SDK generators can use the corpus as a test fixture without needing a real application.
    • -
-

Other apps in the repository are free to use only the CRDT types they actually need; they are not required to cover the full variant set.

-
- -
-
- - - \ No newline at end of file diff --git a/architecture/app-lifecycle.html b/architecture/app-lifecycle.html deleted file mode 100644 index 7237a60d2a..0000000000 --- a/architecture/app-lifecycle.html +++ /dev/null @@ -1,189 +0,0 @@ - - - - -App Lifecycle — Calimero Core Architecture - - - -
-
-
Home/App Lifecycle
- -

App Lifecycle

-

Signing, installation, migration

- -
-

Lifecycle at a glance

-

From first bundle to upgraded context: generate a key, build WASM, manifest, sign, package .mpk, install, create a context; for upgrades use the same signing key, install the new bundle, then context update with a migration method.

-
-/* End-to-end flow (static view) */
- 1. Generate signing key (Ed25519 → did:key)
- ↓
- 2. Build app.wasm
- ↓
- 3. Create manifest.json → 4. Sign (mero-sign)
- ↓
- 5. Package .mpk (tar.gz)
- ↓
- 6. app install on node → 7. context create
- ↓
- 8. Next version: same key, new WASM → 9. app install + context update --migrate-method -
-
- -
-
-

Key concepts

-
    -
  • SignerId — did:key:z{base58btc(0xed01 || public_key_32)}; multicodec prefix 0xed01 marks Ed25519.
    • -
  • AppKey — {package}:{signerId}; continuity of both fixes the logical app.
    • -
  • ApplicationId — for signed bundles: hash(package, signer_id) (32 bytes). Same package + same signer → same id across versions.
    • -
  • Bundle (.mpk) — tar.gz of manifest.json, app.wasm, optional abi.json.
    • -
-
-
-

Why signing?

-

Only the private key holder can ship a valid bundle; the node verifies at install. AppKey continuity blocks another signer from updating an existing application. Lose the key → no updates to that app line (no recovery in v0).

-
-
- -
-

Bundle structure & manifest

-
-app-name-version.mpk // tar.gz
-├── manifest.json // signed metadata
-├── app.wasm
-└── abi.json // optional -
-

Manifest fields (essentials)

-

version, package, appVersion, minRuntimeVersion, signerId, wasm, abi (optional), migrations (often []), signature.

-
-Example manifest shape -
-
-{
-  "version": "1.0",
-  "package": "com.example.app",
-  "appVersion": "1.0.0",
-  "minRuntimeVersion": "0.1.0",
-  "signerId": "did:key:z6Mk…",
-  "wasm": { "path": "app.wasm", "size": …, "hash": null },
-  "abi": { … },
-  "migrations": [],
-  "signature": { "algorithm": "ed25519", … }
-} -
-

Critical package must match across v1/v2; both bundles must be signed with the same key.

-
-
-
- -
-

Signing algorithm

-
    -
  1. Set signerId (from public key) and minRuntimeVersion if missing.
    2. -
  2. Strip transient fields: signature and underscore-prefixed keys (_binary, etc.).
    3. -
  3. Canonicalize JSON with JCS (RFC 8785).
    4. -
  4. bundleHash = SHA-256(canonical_bytes)
    5. -
  5. Ed25519_Sign(private_key, bundleHash) → attach signature object (ed25519, base64url keys).
    6. -
-

Install-time verification repeats canonicalization (without signature), hashes, verifies Ed25519, and checks derived signerId matches the manifest.

-
crates/node/primitives/src/bundle/signature.rs
Bundle signing / verification implementation
-
- -
-
-

Installation

-

Node extracts .mpk, verifies the manifest signature, derives ApplicationId = hash(package, signer_id), stores blob + metadata.

-
-meroctl --node <name> app install --path app.mpk -
-
-
-

Context creation

-

New isolated instance: ContextId, #[app::init] runs, creator is first member. A context is bound to a group, so create a namespace (the root group) first and pass its id as --group-id (required).

-
-meroctl --node <name> namespace create --application-id <APPLICATION_ID>
-meroctl --node <name> context create \
-  --application-id <APPLICATION_ID> --group-id <NAMESPACE_ID> -
-
-
- -
-

Migration anatomy

-

Use #[app::migrate] on a parameterless function. Read old bytes with read_raw(), deserialize an old-schema struct (field order must match Borsh layout), build and return the new state struct; the macro serializes it for the runtime.

-
-#[app::migrate]
-pub fn migrate_v1_to_v2() -> NewState {
-  let old_bytes = read_raw().expect(…);
-  let old: OldState = BorshDeserialize::deserialize(&mut &old_bytes[..]).expect(…);
-  // map fields → NewState { … }
-} -
-

Deserialization: use BorshDeserialize::deserialize(&mut &bytes[..]) — not try_from_slice. Raw bytes can include trailing storage metadata; strict slice mode fails with "Not all bytes read".

-
-When is a migration required? -
-
    -
  • Yes Adding/removing/renaming fields, or changing field types on root state.
    • -
  • No New methods only, or logic changes without schema change.
    • -
-
-
-
- -
-

Deployment flow (upgrade)

-
    -
  1. Build v2 bundle: same package, same signing key as v1.
    2. -
  2. meroctl app install --path v2.mpk (does not rewrite running contexts).
    3. -
  3. meroctl context update --context ID --application-id ID --as MEMBER_KEY --migrate-method fn_name
    4. -
-
-/* Under the hood (simplified) */
- CLI → admin POST …/contexts/{id}/application
- → verify signerId matches existing app (AppKey continuity)
- → load new WASM → run exported migrate_* (read_raw, transform, return)
- → persist new root state (atomic; failure keeps old state) -
-
- -
-

CLI cheat sheet

-
-# key + sign
-cargo run -p mero-sign -- generate-key --output key.json
-cargo run -p mero-sign -- sign manifest.json --key key.json
-
-# node
-meroctl --node <X> app install --path app.mpk
-meroctl --node <X> context update --context <ID> --application-id <ID> \
-  --as <KEY> --migrate-method <fn_name> -
-
- -
-

Troubleshooting

-
-
-

Not all bytes read

-

Use deserialize(&mut &bytes[..]) instead of try_from_slice.

-
-
-

signerId mismatch

-

Sign both versions with the same key; keep package identical.

-
-
-

Missing signature

-

Run mero-sign sign on manifest.json before tarring the .mpk.

-
-
-
- -
-
- - - diff --git a/architecture/auto-follow.html b/architecture/auto-follow.html deleted file mode 100644 index a11f8d5363..0000000000 --- a/architecture/auto-follow.html +++ /dev/null @@ -1,817 +0,0 @@ - - - - -Auto-Follow Group Membership — Calimero Core Architecture - - - -
-
-
Home/Auto-Follow
- - -

Auto-Follow Group Membership

-

Event-driven context auto-join for group members, with per-member opt-in

- -
-

Scope

-

-This page covers context auto-join — the contexts: true path. The handler -reacts to ContextRegistered / AutoFollowSet ops and emits -JoinContextRequest. It does not perform member admission and does -not act on the subgroups: true flag (see -TEE Fleet Integration below). For how a fleet node actually -gains subgroup membership, see TEE Fleet HA. -

-
- -
-
Opt-in
per-member flag
-
DAG
event-driven
-
20/min
default rate limit
-
Replay
offline catch-up
-
- - -
-

Problem

-

-Group membership in Calimero is explicit at every level. When a new context is registered in -a group, existing members do not automatically replicate it — each must call -join_context explicitly. When a subgroup is nested under a parent, the parent's -members are not automatically admitted to the child either (subgroup membership is independent). -This works for human-driven clients but breaks two concrete scenarios: -

-
    -
  • TEE fleet HA. A fleet node admitted via fleet-join joins only the contexts - that existed at admission time. Later contexts are invisible until something else triggers a - join, which — before auto-follow — was polling by the sidecar.
    • -
  • Regular members observing a growing group. A member who joined when the group had - 3 contexts sees nothing when a 4th appears, unless the client polls or the user reacts manually.
    • -
-
- - -
-

Design

-

-Each GroupMember gains a pair of opt-in flags: -

-
AutoFollowFlags {
-    contexts:  bool,   // auto-join new ContextRegistered     — default true
-    subgroups: bool,   // auto-admit into newly-nested subgroups — default false
-}
- -

-Defaults (as of #2422): contexts: true, subgroups: false. -A new member added via add_group_member auto-follows new contexts in the group by -default; subgroup auto-admit stays opt-in. ReadOnlyTee members get both flags set to -true automatically. The default lives in a single -explicit impl Default for AutoFollowFlags in -crates/store/src/key/group/mod.rs, so both the borsh-legacy decode fallback and -.unwrap_or_default() sites pick it up. -

-
-

-The subgroups: true flag is currently a no-op in the auto-follow path. The handler -implements contexts: true only. It reads the subgroups field off -AutoFollowSet and discards it (subgroup auto-follow is "implemented per-role in a -follow-up" per the module docs). So although a ReadOnlyTee member carries -subgroups: true, that flag drives no admission today. How a fleet node actually obtains -subgroup access is covered in TEE Fleet HA and summarised in -TEE Fleet Integration below. -

-
-

-

-

-Flags are toggled by the governance op GroupOp::MemberSetAutoFollow with -admin-or-self authorization: -

-
    -
  • A group admin can toggle flags for any member.
    • -
  • A member can toggle their own flags.
    • -
- -

Propagation

-

-The handler subscribes to an in-process op-apply event channel (module -context::op_events). When a relevant op is applied to local state, an -OpEvent is broadcast and the handler emits the corresponding join op. Every emitted -op is itself a DAG op, so offline catch-up comes for free via DAG replay — no separate -reconcile loop. -

- -

Authorizer-aware apply path

-

-The new apply_local_signed_group_op_at_cut function accepts an -&dyn AtCutAuthorizer alongside the op and threads it through -apply_group_op_mutations via GroupApplyCtx::new_with_apply_auth and -PermissionChecker::with_apply_auth. Namespace envelopes forward the enclosing -namespace op's causal cut and authorizer via -NamespaceGovernance::apply_group_op_inner, so group ops carried in namespace -envelopes are authorized at the correct cut. The original apply_local_signed_group_op is -now a shim that calls the new variant with the inert LIVE_FALLBACK_AUTHORIZER, -keeping all existing callers (tests, replay) unchanged. -

-

-PermissionChecker::is_admin and is_authorized_with_capability now -query the projection at the op's causal cut first via is_admin_at_cut / -is_admin_or_capability_at_cut. When the authorizer returns -Some(verdict), that verdict is used directly and the live -MembershipRepository is not consulted. The live store is queried only when the -authorizer returns None (no apply-auth context, empty parents, or an incomplete -fold). The former shadow_admin helper — which compared the projection verdict to -the live verdict and emitted unified_projection_divergence warnings on mismatch — -has been deleted along with the associated shadow block in -is_authorized_with_capability. -

-

-In apply_group_op_mutations (called from -NamespaceGovernance::apply_group_op_inner), the parents argument now -uses signed_group_op.parent_op_hashes instead of self.parents. On -the normal apply path the two are equal, but during -retry_encrypted_ops_for_group multiple buffered group candidates are re-applied -within a single KeyDelivery apply; using self.parents would judge -every replayed candidate at the KeyDelivery's cut rather than each candidate's -own cut. -

-
- - -
-

Flow

-
0. (NEW, #2422) A new member is added to the group via
-   GroupOp::MemberAdded / GroupOp::MemberJoinedViaTeeAttestation /
-   RootOp::MemberJoined (open-subgroup self-join).
-   └─ Apply path writes a fresh GroupMember row with default flags
-      (contexts: true, subgroups: false).
-   └─ build_auto_follow_set_if_enabled synthesises
-      OpEvent::AutoFollowSet { member, contexts: true, subgroups: false }
-      so the on-join backfill cascade fires without requiring an
-      explicit MemberSetAutoFollow op. This closes the Ronit/Fran
-      regression where a joiner saw no pre-existing contexts until an
-      admin manually flipped a flag.
-
-1. Admin or member publishes MemberSetAutoFollow { target, contexts, subgroups }.
-   └─ Authorized by admin-or-self check in apply_group_op_mutations.
-   └─ Store updates GroupMemberValue.auto_follow.
-   └─ op_events::notify(OpEvent::AutoFollowSet { ... }).
-
-2. Auto-follow handler (spawned once from ContextManager::started) observes
-   AutoFollowSet { member = self, contexts: true } and backfills: enumerate
-   up to BACKFILL_LIMIT contexts in the group, emit JoinContext for each.
-   Backfill is rate-limited to DEFAULT_BURST / DEFAULT_PER. Idempotent on
-   already-joined contexts, so steps 0 + 1 firing for the same member is
-   safe (e.g. TEE fleet-join: synthesised from MemberJoinedViaTeeAttestation
-   then explicit MemberSetAutoFollow from fleet_join.rs).
-
-3. Later, anyone registers a new context in the group.
-   └─ GroupOp::ContextRegistered applied.
-   └─ op_events::notify(OpEvent::ContextRegistered { group, context }).
-   └─ Handler checks: is self a member with auto_follow.contexts = true?
-   └─ If yes, emit JoinContext — same rate limit.
-
-4. Later, an admin creates a subgroup under this group.
-   └─ RootOp::GroupCreated { group_id, parent_id } applied on namespace DAG
-       (atomic create+nest — strict-tree invariant).
-   └─ op_events::notify(OpEvent::SubgroupCreated { parent, child }).
-   └─ The auto-follow handler IGNORES this event today (the SubgroupCreated /
-      AutoFollowSet { subgroups: true } variants fall through the catch-all
-      match arm). Subgroup membership is handled elsewhere — see TEE Fleet HA.
-
-4b. Admin moves an existing subgroup to a new parent.
-    └─ RootOp::GroupReparented { child, new_parent } applied.
-    └─ op_events::notify(OpEvent::SubgroupReparented
-        { old_parent, new_parent, child }).
-
- - -
-

TEE Fleet Integration

-

-A TEE fleet node calls POST /admin-api/tee/fleet-join. The handler in -server::admin::handlers::tee::fleet_join: -

-
    -
  1. Generates a TDX attestation quote bound to the node's namespace identity pubkey.
    2. -
  2. Broadcasts TeeAttestationAnnounce on the namespace topic.
    3. -
  3. Polls for admission (up to 30 s) by calling list_group_contexts. Once the - verifier's MemberJoinedViaTeeAttestation op has propagated, the list succeeds.
    4. -
  4. Joins all existing contexts in the group.
    5. -
  5. Publishes MemberSetAutoFollow { target: self, contexts: true, subgroups: true }, - signed by the node's own namespace-identity key. The admin-or-self rule is satisfied via the - self path — the admitting verifier can't do this on the member's behalf because it - usually lacks both admin authority and the member's signing key.
    6. -
-

-From this point, every new context in the group is auto-joined by the core handler (the -contexts: true path described on this page). The mero-tee sidecar's -per-group context-polling loop becomes redundant. -

-
-

-The subgroups: true flag published here is, today, a no-op. The auto-follow -handler implements contexts: true only — it does not act on subgroups: true -and does not perform member admission. Publishing subgroups: true at fleet-join -therefore does not cause the node to auto-join subgroups. The flag is set in anticipation of -the per-role subgroup follow-up, but it drives no admission in the current tree. -

-

-Subgroup access for a ReadOnlyTee fleet node comes from two other mechanisms, -not from auto-follow: -

-
    -
  • Open subgroups. Inherited membership plus the namespace key the node already holds — - no separate admission step is required.
    • -
  • Restricted subgroups. A separate tee_subgroup_admit subscriber (PR #2772) - in which an existing key-holder admits the node in response to - SubgroupCreated / TeeMemberAdmitted events. This is outside the - auto-follow path entirely.
    • -
-

-The full fleet/subgroup admission story is documented in -TEE Fleet HA. -

-
-

-Policy scope — namespace, not per-group. The canonical TeeAdmissionPolicy lives -on the namespace root only. read_tee_admission_policy in group_store::tee -resolves its argument to the namespace root before reading, and both the write handler -(handlers::set_tee_admission_policy) and the apply path in -group_store::apply_group_op_mutations refuse a TeeAdmissionPolicySet -targeting a subgroup. Subgroup policy bytes in any legacy op logs are ignored. Admission — including -how this namespace-scoped policy is enforced — is covered in depth in -TEE Fleet HA. -

-
- - -
-

Rate Limit & Backpressure

-

-The handler runs behind a token-bucket limiter. Defaults: -

-
    -
  • DEFAULT_BURST = 20 — tokens available at once.
    • -
  • DEFAULT_PER = 60 s — bucket refills fully in this window (one token every - 3 s).
    • -
  • BACKFILL_LIMIT = 1000 — per-flip cap for enumerating existing contexts. Future - contexts beyond the cap are picked up event-driven with no additional limit.
    • -
-

-Semaphore-closed and subscriber-lagged conditions are both surfaced via warn!. The -authoritative recovery mechanism is always DAG replay: if an event is missed (best-effort -broadcast), the next run of the handler walks the DAG and reconciles state. -

-
- - -
-

Role Resolution in Inherited Paths

-

-Two functions in MembershipRepository determine what role an inherited member -carries when they access a child group or context. Both were tightened as part of -#2422. -

- -

enumerate_inherited

-

-For each non-admin inherited candidate the function previously hard-coded -GroupMemberRole::Member. It now calls role_of(&anchor, &candidate) -to retrieve the candidate's actual role at the anchor group, falling back to -Member only if the row is unexpectedly absent. Admin paths continue to resolve to -Admin (unchanged). Callers therefore receive the real anchor role — for example -ReadOnlyTee — rather than a uniform Member. -

- -

New regression tests

-
    -
  • enumerate_inherited_members_preserves_read_only_tee_role — verifies the - enumeration path returns ReadOnlyTee for an inherited member carrying that - role.
    • -
-
- - -
-

Operator Notes

-
    -
  • Observability. Every auto-join emits a structured info! log line - with group_id and context_id. Failures emit warn! with - the underlying error. No new metrics — the log stream is enough for postmortems.
    • -
  • state_hash mismatch warnings. In - apply_local_signed_group_op_at_cut, a mismatch between the op's signed - state_hash and the recomputed current hash previously caused a hard error that - dropped the op. It now emits a structured tracing::warn! (including op variant, - expected/actual hashes, nonce, and signer) and proceeds to apply the op. A - local group op state_hash mismatch warning is therefore expected under concurrent - multi-node governance activity and does not indicate a rejected op; it is staleness telemetry - only. The absent-meta bypass (skip hash check when the subgroup meta row does not yet exist, - #2848) is preserved unchanged. Additionally, the per-signer nonce-window load and containment - check (load_nonce_window) now runs before the state_hash - comparison block, so a replayed op returns Ok(()) early without ever reaching the - hash logic and cannot emit a misleading mismatch warning for an op that is actually being - discarded.
    • -
  • Membership-path authorization. The member_joined_open::apply gate - now evaluates membership-path authorization against the causal-cut projection state via - NamespaceApplyCtx::projection_membership_path, which returns - Option<AtCutMembershipPath>. The live check_path call is a - fallback used only when the projection returns None (no apply-auth context, - incomplete fold, or default live-only authorizer). The former - shadow_membership_path helper — which accepted an eagerly-computed live path, - compared it against the projection, and emitted unified_projection_divergence / - plane=membership-path warnings on divergence — has been removed along with all - associated shadow-logging. Operators who were monitoring those warnings should note they will - no longer appear.
    • -FIND>>> -<<Operator Notes -
      -
    • Observability. Every auto-join emits a structured info! log line - with group_id and context_id. Failures emit warn! with - the underlying error. No new metrics — the log stream is enough for postmortems.
      • -
    • Membership-path authorization. The member_joined_open::apply gate - now evaluates membership-path authorization entirely against the unified projection via - AtCutAuthorizer::membership_path_at_cut. The former live DAG walk - (acl_view_at / prefix_walk_membership / MembershipStatus) - has been removed; those symbols no longer exist. The live check_path call is a - fallback used only when the projection returns None (no apply-auth context, - incomplete fold, or default live-only authorizer). The - shadow_membership_path divergence-logging helper has also been removed.
      • -FIND>>> -<<Membership-path authorization. The member_joined_open::apply gate - now evaluates membership-path authorization entirely against the unified projection via - AtCutAuthorizer::membership_path_at_cut. The former live DAG walk - (acl_view_at / prefix_walk_membership / MembershipStatus) - has been removed; those symbols no longer exist. The live check_path call is a - fallback used only when the projection returns None (no apply-auth context, - incomplete fold, or default live-only authorizer). The - shadow_membership_path divergence-logging helper has also been removed. -
    • Authorization source. PermissionChecker::is_admin and - is_authorized_with_capability now use the projection-at-cut verdict as - authoritative when available. Live store queries are a fallback only when the authorizer - returns None. The last-admin removal/demotion gates (ensure_not_last_admin_removal - and ensure_not_last_admin_demotion) likewise resolve against the projection at the - op's parent cut via would_orphan_admins, falling back to live membership only - when no apply-auth context is present. The shadow_last_admin divergence-logging - helper and all associated unified_projection_divergence / plane=last-admin - warnings have been removed now that the projection verdict is authoritative.
      • -
    • Shutdown. Call auto_follow::shutdown() to abort the handler task and - its refill loop. Subsequent spawn calls will start a fresh handler.
      • -
    • Migration. GroupMemberValue was extended with auto_follow - via a custom Borsh deserializer. Records written under the pre-auto-follow schema are - transparently read with default flags, and transparently upgraded on the next write. A partial - trailing byte (data corruption) surfaces as a deserialization error instead of being silently - defaulted.
      • -
    • Implementing AtCutAuthorizer. Any custom implementation of - AtCutAuthorizer must now provide is_admin_or_capability_at_cut and - the new membership_path_at_cut method. For both, returning None (as - LiveFallbackAuthorizer does) is safe and causes the gates to fall back to live - store queries. Note the distinction: Option::None means "defer to live", while - Some(AtCutMembershipPath::None) means "not a member at this cut" and is an - authoritative answer.
      • -FIND>>> -<<Implementing AtCutAuthorizer. Any custom implementation of - AtCutAuthorizer must now provide is_admin_or_capability_at_cut and - the new membership_path_at_cut method. For both, returning None (as - LiveFallbackAuthorizer does) is safe and causes the gates to fall back to live - store queries. Note the distinction: Option::None means "defer to live", while - Some(AtCutMembershipPath::None) means "not a member at this cut" and is an - authoritative answer. -
    -
- - -
-

Key Files

-
    -
  • crates/context/src/op_events.rs — op-apply event channel + OpEvent - enum.
    • -
  • crates/context/src/auto_follow.rs — handler task, rate limiter, - spawn/shutdown.
    • -
  • crates/context/src/group_store/mod.rs — apply_group_op_mutations - (now accepts parents and authorizer) handles MemberSetAutoFollow; - apply_local_signed_group_op_at_cut is the new authorizer-aware public entry point; - apply_local_signed_group_op is now a shim forwarding to it with the inert - LIVE_FALLBACK_AUTHORIZER.
    • -
  • crates/context/src/group_store/membership.rs — - set_member_auto_follow helper.
    • -
  • crates/context/primitives/src/local_governance/mod.rs — the - MemberSetAutoFollow op variant.
    • -
  • crates/store/src/key/group/mod.rs — AutoFollowFlags and the - backward-compatible BorshDeserialize for GroupMemberValue.
    • -
  • crates/context/src/handlers/admit_tee_node.rs — TEE admission publishes the - op only; flags are set by the admitted node itself.
    • -
  • crates/server/src/admin/handlers/tee/fleet_join.rs — after admission, the - member publishes MemberSetAutoFollow signed by self.
    • -
  • crates/context/src/group_store/permission.rs (or equivalent) — - PermissionChecker gains with_apply_auth; is_admin and - is_authorized_with_capability use the projection-at-cut verdict as the - authoritative decision when AtCutAuthorizer returns Some, falling - back to live store queries otherwise. The shadow_admin helper and - unified_projection_divergence logging have been removed. MembershipPolicy - now optionally carries the op's parent cut hashes and an AtCutAuthorizer reference, - set via the new with_apply_auth builder method. - ensure_not_last_admin_removal and ensure_not_last_admin_demotion - now delegate to the new would_orphan_admins helper, which calls - AtCutAuthorizer::is_last_admin_at_cut to read the pre-mutation admin set from - the projection as of the op's parent cut. The live membership computation is used only as a - fallback when no apply-auth context is present (local pre-checks, cascades, tests) or the fold - is incomplete. The shadow_last_admin helper and all - unified_projection_divergence / plane=last-admin warnings have been - deleted. GroupApplyCtx::new calls .with_apply_auth(parents, authorizer) - on the MembershipPolicy it constructs, activating the projection-at-cut last-admin - gate for all group-op apply paths.
    • -
  • crates/context/src/authorizer.rs (or equivalent) — AtCutAuthorizer - trait gains the required is_admin_or_capability_at_cut method, the - is_last_admin_at_cut method, and the new required - membership_path_at_cut method. All methods must return None when - parents is empty (the empty-cut contract), so that genesis ops are never - falsely rejected. EphemeralProjectionAuthorizer delegates - is_last_admin_at_cut to ScopeProjections::is_last_admin_at_cut - and implements membership_path_at_cut by delegating to - ScopeProjections::membership_path_at_cut, mapping the result to the new - AtCutMembershipPath enum (projecting away role/anchor detail). An empty - parents slice returns None (defer to live), consistent with the - existing is_admin_at_cut pattern. LiveFallbackAuthorizer implements - all three methods as always-None, keeping all non-apply-auth constructions on - the live-fallback path. -
    Note: the live governance-cut resolver (acl_view_at, - MembershipStatus, prefix_walk_membership, - MembershipTransition, resolve_membership_from_transitions, - heads_equal, MAX_PREFIX_WALK_NODES) has been deleted from - governance-store/src/membership/status.rs. Membership authorization is now - handled entirely by the unified projection; do not reference these removed symbols.
    • -FIND>>> -<<crates/context/src/authorizer.rs (or equivalent) — AtCutAuthorizer - trait gains the required is_admin_or_capability_at_cut method, the - is_last_admin_at_cut method, and the new required - membership_path_at_cut method. All methods must return None when - parents is empty (the empty-cut contract), so that genesis ops are never - falsely rejected. EphemeralProjectionAuthorizer delegates - is_last_admin_at_cut to ScopeProjections::is_last_admin_at_cut - and implements membership_path_at_cut by delegating to - ScopeProjections::membership_path_at_cut, mapping the result to the new - AtCutMembershipPath enum (projecting away role/anchor detail). An empty - parents slice returns None (defer to live), consistent with the - existing is_admin_at_cut pattern. LiveFallbackAuthorizer implements - all three methods as always-None, keeping all non-apply-auth constructions on - the live-fallback path. -
-
- -

Re-drive Buffered Ops on GroupCreated & Born-Open Subgroups (#2848 / #2771)

- -

-This section documents four related changes that close a permanent-stall window where buffered -encrypted ops for a subgroup could never be applied, and extends RootOp::GroupCreated -to carry subgroup visibility at creation time. -

- -

The Stall Window (Motivation)

-

-The only previous trigger for retrying buffered encrypted ops was KeyDelivery. If -KeyDelivery arrived before GroupCreated, the retry failed -immediately because the subgroup's GroupMeta row did not yet exist, and no -subsequent trigger ever re-fired. The buffered ops remained stranded indefinitely. This affected -re-admitted members and any topology where DAG op ordering placed GroupCreated after -the key delivery. -

- -

Part A — Re-drive on GroupCreated Apply

-

-After RootOp::GroupCreated is applied, the governance layer now checks whether the -node holds any key for the new subgroup (or the namespace key, for Open subgroups). If a key is -held, it immediately calls the buffered-op retry path for that group — the same path that -KeyDelivery triggers. This closes the stall window regardless of which op arrives -first. -

-

-The retry is triggered on both apply entry points: -

-
    -
  • apply_local_signed_group_op_at_cut — local group-op path.
    • -
  • NamespaceGovernance::apply_group_op_inner — namespace envelope path.
    • -
- -

Part B — Bypass state_hash Staleness Check When Subgroup Meta Is Absent

-

-Both apply paths now skip the state_hash staleness check entirely when the -subgroup's GroupMeta row does not yet exist, rather than calling -compute_state_hash and failing with GroupNotFoundForHash. This mirrors -the existing zero-hash fast-path and makes the meta-absent case consistent with it: -

-
// Before: compute_state_hash → GroupNotFoundForHash
-// After:
-if meta_absent || state_hash == ZERO_HASH {
-    // skip staleness check
-}
- -

Part C — Curative Startup Sweep

-

-A one-shot redrive_stranded_ops_sweep runs at node startup (inside -self_purge::run, after subscribing to op_events and before the event -loop). It: -

-
    -
  1. Iterates every namespace the node holds an identity for.
    2. -
  2. Finds groups with buffered ops whose key is already held (the inverse of - groups_awaiting_key).
    3. -
  3. Re-drives each one via redrive_encrypted_ops_for_group_counted.
    4. -
  4. Loops until a full pass applies nothing new (cross-signer convergence), bounded by - MAX_PASSES = 64, with pass-narrowing so only groups that made progress in the - previous pass are revisited.
    5. -
-

-Errors are logged and swallowed; the sweep never blocks startup. Nodes that were stranded before -this release will self-heal on the next restart without operator intervention. -

- -

TEE-Replica Bootstrap — Seed Root Default Capabilities

-

-seed_bootstrap_admin_if_absent (the KeyDelivery-seed path used by TEE -replicas) seeds the namespace root's default capabilities to -CAN_JOIN_OPEN_SUBGROUPS when that capability key is absent. The seed is gated on -absence so it is idempotent and never clobbers a later admin-authored -DefaultCapabilitiesSet op. Without this, a TEE replica could hold the group key -but still be unable to join Open subgroups because the capability row was never written. -

-

-Founder is no longer pinned here (#2474). This path no longer pins the -KeyDelivery signer as admin/owner. It now writes a non-authoritative -placeholder (zero-key admin_identity) plus a Member row for the -deliverer, alongside the default-capabilities bootstrap above. The real founding admin is -derived authoritatively from the synced DAG genesis op -(RootOp::NamespaceCreated { founder }), which overwrites the placeholder on apply — -replacing the previous trust-on-first-KeyDelivery behavior. -

-

-Pre-existing stranded replicas need either a restart (triggering the startup -sweep in Part C) or a gossiped DefaultCapabilitiesSet op from an admin to acquire -the capability row. -

- -

Born-Open Atomic Subgroup Create (RootOp::GroupCreated wire change)

-

-RootOp::GroupCreated gains a boolean restricted field -(Borsh wire break — nodes must reset). The apply handler writes the subgroup's -visibility key via CapabilitiesRepository::set_subgroup_visibility during apply, -before OpEvent::SubgroupCreated is queued, and only on first apply (gated on -has_subgroup_visibility absence to avoid clobbering a later -SubgroupVisibilitySet flip): -

-
    -
  • restricted: false — subgroup is born Open; tee_subgroup_admit - skips it and no transient direct ReadOnlyTee row is written.
    • -
  • restricted: true — legacy behavior; subgroup is born Restricted. This is - the default and preserves wire compatibility for any tooling that constructs the op directly.
    • -
-

-The op-adapter projection now reads restricted from the live op instead of -hardcoding true. -

-

-API surface changes: -

-
    -
  • CreateGroupRequest gains a restricted field (default - true).
    • -
  • The create-group-in-namespace REST endpoint accepts an optional - visibility field ("open" or "restricted", default - "restricted").
    • -
- -

Operator Checklist

-
    -
  • Wire break. The RootOp::GroupCreated Borsh shape has - changed. All nodes in a namespace must reset before the new binary is deployed; mixed-version - clusters will fail to deserialize the op.
    • -
  • Stranded replicas. TEE replicas admitted before this release may lack the - root default-capability row. A restart or a gossiped DefaultCapabilitiesSet from - an admin is required for those nodes to join Open subgroups.
    • -
  • Startup sweep. The sweep is logged at info! level per group - re-driven and at warn! on error. A clean node with no buffered ops completes the - sweep in a single pass with no log output beyond the start/end lines.
    • -
  • Born-Open subgroups. Creating a subgroup with - visibility: "open" is now atomic — no separate SubgroupVisibilitySet - op is needed. Any tooling that issued a two-step create+flip can be simplified.
    • -
-
-

Namespace Genesis — RootOp::NamespaceCreated

- -

-(#2474 / #2932) A new NamespaceCreated { founder: PublicKey } -variant is appended to the RootOp enum. It is the first op in every -namespace DAG: no parents, nonce 1, signed with the namespace key. It carries the -namespace_created op-kind label and is classified as a cheap-timeout op in -governance_broadcast. -

- -
-

-Wire break. NamespaceCreated is appended at the end of the -RootOp enum so existing Borsh discriminants do not renumber. It is nevertheless a -Borsh-breaking addition for any consumer that pins calimero-governance-types. All -nodes in a namespace must coordinate a reset before deploying the new binary. Consumers such as -mero-tee must bump their core-rev dependency accordingly. -

-
- -

Apply Rules

-

-The handler in ops/namespace/namespace_created.rs is self-authorizing -— it does not call require_namespace_admin. Established-ness is keyed solely on -admin_identity != PLACEHOLDER_ADMIN_IDENTITY. The handler branches as follows: -

-
    -
  • Not established + has parents: returns - Err(NamespaceCreatedRejected(NotGenesis { parent_count })). This is - load-bearing: an Err propagates before advance_dag_head, so the - head remains empty and a subsequent parentless genesis can still apply.
    • -
  • Not established + no parents + signer ≠ founder: returns - Err(NamespaceCreatedRejected(SignerNotFounder)).
    • -
  • Not established + no parents + signer == founder: writes root meta with - admin == owner == founder (preserving other fields from any existing partial - meta), unconditionally upserts the founder's Admin member row, and seeds default - CAN_JOIN_OPEN_SUBGROUPS capabilities if absent.
    • -
  • Established + same founder + no parents (genesis-shaped re-arrival): - ensures the Admin member row (upgrade-only, never downgrade), repairs a diverged - owner_identity back to the founder, seeds default caps if absent. Returns - Ok.
    • -
  • Established + same founder + has parents: pure no-op Ok - (not genesis-shaped; no state mutation).
    • -
  • Established + different founder: pure no-op Ok - (anti-hijack guard).
    • -
- -

Placeholder Admin Identity

-

-A module-level constant PLACEHOLDER_ADMIN_IDENTITY: [u8; 32] = [0u8; 32] and -companion function placeholder_admin_identity() -> PublicKey are defined in -governance-store/src/lib.rs. The all-zeros encoding is cryptographically safe: it -corresponds to the Ed25519 torsion/identity point, outside the prime-order subgroup, so no real -keypair can produce it. -

- -

Seed Behavior Change — Authority No Longer Conferred by seed_bootstrap_admin_if_absent

-

-seed_bootstrap_admin_if_absent now writes the PLACEHOLDER_ADMIN_IDENTITY -sentinel as both admin_identity and owner_identity. The -KeyDelivery signer receives a non-authoritative GroupMemberRole::Member -row instead of Admin. The real founding admin is established only when the -NamespaceCreated genesis op is applied: -

-
    -
  • Genesis arrives after seed: the handler detects the placeholder and - overwrites admin_identity / owner_identity with the founder, then - upgrades the founder's member row to Admin.
    • -
  • Genesis arrives before seed: the namespace is already established; the - seed writes into an already-established store and the anti-hijack no-op path applies on any - re-arrival of the genesis op.
    • -
-

-Both orderings converge on the genesis-supplied founder as the authoritative admin. The former -trust-on-first-KeyDelivery behavior — where the deliverer's identity was pinned as -admin — has been removed. Any operator documentation or threat-model notes describing -the seed as the source of admin authority are now obsolete. -

- -

Trust Residual on Bare Replicas (#2932)

-

-A TEE replica that has received a KeyDelivery but not yet the -NamespaceCreated genesis op is in a partially-established state: it holds -the placeholder admin identity and a Member row for the deliverer. It cannot -authorize admin-level ops until the genesis op arrives and overwrites the placeholder. Operators -should ensure the genesis op is replicated promptly; the startup sweep described in the -Re-drive Buffered Ops section will retry any buffered ops once the meta row is -established. -

- -

New Error Types

-
    -
  • NamespaceCreatedRejection::SignerNotFounder — signer key does not match the - declared founder field.
    • -
  • NamespaceCreatedRejection::NotGenesis { parent_count } — the op has parents - but the namespace is not yet established.
    • -
  • ApplyError::NamespaceCreatedRejected(NamespaceCreatedRejection) — wraps the - above and is returned from the apply handler without advancing the DAG head.
    • -
- -

GroupKeyring::delete_key_by_id

-

-A new method GroupKeyring::delete_key_by_id(key_id: &[u8; 32]) deletes a -single stored group key by its id. Unlike delete_all_for_group it does not require -the membership-removed purge precondition. Its sole intended use is the namespace-root genesis -rollback path, where a key installed speculatively must be removed if the genesis op is -subsequently rejected. -

- -

Key Files

-
    -
  • calimero-governance-types/src/root_op.rs — RootOp::NamespaceCreated - variant (appended).
    • -
  • governance-store/src/ops/namespace/namespace_created.rs — apply handler.
    • -
  • governance-store/src/lib.rs — PLACEHOLDER_ADMIN_IDENTITY constant - and placeholder_admin_identity() helper.
    • -
  • governance-store/src/errors.rs — NamespaceCreatedRejection enum - and ApplyError::NamespaceCreatedRejected variant.
    • -
  • governance-store/src/keyring.rs — GroupKeyring::delete_key_by_id.
    • -
  • governance-store/src/namespace/tests.rs, - governance-store/src/tests.rs — regression and unit tests (13 new cases including - the #2474 regression guard).
    • -
-
-

State-Hash Staleness Checks Removed

- -

-As of this change, the state_hash field has been removed from the governance op -wire format entirely. The warn-and-apply staleness check that previously existed in -apply_local_signed_group_op_at_cut and apply_group_op_inner has been -deleted along with all supporting infrastructure. The per-signer nonce window is now the -sole anti-replay and deduplication gate on the apply path. -

- -

What Was Removed

-
    -
  • Staleness check blocks in apply_local_signed_group_op_at_cut - and NamespaceGovernance::apply_group_op_inner — the code that recomputed the - group or namespace state hash and emitted a tracing::warn! on mismatch is gone. - No warning of the form local group op state_hash mismatch will ever appear; any - monitoring or alerting on that string can be retired.
    • -
  • NamespaceGovernance::state_hash_for_op() — computed the appropriate hash - domain for a namespace op at sign time.
    • -
  • root_op_commits_to_namespace_state() — a module-level whitelist of - RootOp variants whose correctness previously depended on namespace state.
    • -
  • pre_apply_state_hash capture in GroupGovernancePublisher and - the state_hash parameter threading through publish_post_gate, - sign_and_publish_post_gate, and sign_and_publish_without_apply.
    • -
  • GroupHandle::compute_state_hash() — a public thin wrapper over - MetaRepository::compute_state_hash that had no remaining callers.
    • -
- -

Wire and Serialization Impact

-

-The synthesized SignedGroupOp inside decrypt_and_apply_group_op no -longer copies ns_op.state_hash (the field is gone from the struct). The DAG delta -helpers signed_op_to_delta and signed_namespace_op_to_delta now pass -[0u8; 32] as expected_root_hash because governance ops no longer carry -a meaningful root claim. -

- -

Deleted Tests

-

-The following unit tests and their shared setup_state_hash_test_fixture helper were -deleted because they pinned behaviour that no longer exists: -

-
    -
  • namespace_group_op_zero_state_hash_bypasses_check
    • -
  • namespace_group_op_with_current_state_hash_applies
    • -
  • namespace_group_op_with_stale_state_hash_applies_with_warning
    • -
  • group_op_with_stale_hash_and_absent_meta_applies
    • -
  • namespace_root_op_with_current_state_hash_applies
    • -
  • namespace_root_op_with_stale_state_hash_applies_with_warning
    • -
  • state_hash_allows_sequential_ops
    • -
  • state_hash_zero_skips_validation
    • -
- -

Operator Notes

-
    -
  • The absent-meta bypass documented in the Re-drive Buffered Ops - section (skip the staleness check when the subgroup GroupMeta row does not yet - exist) is also gone — there is no check left to bypass. The absent-meta case is now - unremarkable.
    • -
  • Any tooling or client library that populated state_hash - when constructing a SignedGroupOp or SignedNamespaceOp must be - updated to omit the field. Passing a non-zero value is a wire error on the new schema.
    • -
  • The nonce-window dedup path is unchanged. A replayed op is still detected and returns - Ok(()) early, as before.
    • -
-
-
-
- - - diff --git a/architecture/concepts.html b/architecture/concepts.html deleted file mode 100644 index e9eb472fc7..0000000000 --- a/architecture/concepts.html +++ /dev/null @@ -1,171 +0,0 @@ - - - - -Core Concepts — Calimero Core Architecture - - - -
-
-
Home/Core Concepts
-

Core Concepts

-

Groups, contexts, roles, capabilities, and CRDTs for app builders

- -
-

Namespaces

-

A namespace is a root group (a group with no parent). It is the application instance boundary and the identity scope for a node.

-
    -
  • Each namespace gets its own Ed25519 keypair, auto-generated on first use and persisted in the datastore.
    • -
  • All subgroups and contexts within the namespace share the same node identity.
    • -
  • Different namespaces have different keys — identity isolation across application instances.
    • -
  • Subgroups inherit the namespace’s target_application_id; upgrading the root upgrades the entire tree.
    • -
  • The admin API exposes GET /admin-api/namespaces to list, query, and inspect namespace identity.
    • -
  • All groups within a namespace share a single governance DAG. Operations are either cleartext (RootOps: group creation, member join, key delivery) or encrypted (GroupOps: membership changes, capabilities).
    • -
  • New members receive group encryption keys via ECDH-wrapped KeyDelivery on the namespace DAG.
    • -
-
- -
-

Groups

-

A group is a governance boundary within a namespace. It has members, an application (inherited from the namespace root), and one or more contexts. All membership management, access control, and upgrades happen at the group level via signed governance operations that propagate via P2P gossip.

-
    -
  • Think of the group as where you vote, invite people, and upgrade the app.
    • -
  • Contexts live inside a group; they do not replace group governance.
    • -
  • Every group has at least one Admin who can always perform administrative actions.
    • -
-
- -
-

Contexts

-

A context is a running instance of a WASM application with its own isolated state. State stays in sync across context members automatically, using CRDT-based replication.

-
    -
  • Each context belongs to exactly one group.
    • -
  • Multiple contexts can run the same application (or different services from the same bundle) with independent state.
    • -
  • Joining a context requires group membership. By default (auto_join: true), joining a group auto-joins all its contexts.
    • -
  • For multi-service applications, each context specifies which service it runs via an optional service_name.
    • -
-
- -
-

Services (multi-WASM bundles)

-

An application bundle (.mpk) can contain multiple named services, each with its own WASM binary and optional ABI.

-
    -
  • Single-service bundles (one wasm field in the manifest) work as before — no service name needed.
    • -
  • Multi-service bundles declare a services array in the manifest, each with a name and WASM path.
    • -
  • When creating a context for a multi-service app, specify service_name to choose which WASM to run.
    • -
  • All services in a bundle share the same ApplicationId and upgrade atomically.
    • -
  • Example: a chat application with services user-mgmt, chat, and storage — each running in its own context with independent state.
    • -
-
- -
-

Member roles

-

Roles define what a member can do inside a group. The platform enforces them consistently on every node.

-
-

Admin

Full control: all governance operations, member management, and state writes.

-

Member

Standard participant: read and write app state. Some operations still require explicit capabilities.

-

ReadOnly

Observer: can join and read state. Mutations are rejected locally and on remote peers.

-
-
- -
-

Subgroups

-

Groups form a tree. A child group links to a parent.

-

Membership and admin inheritance

-
    -
  • Membership inherits downward: a parent member is also a member of descendant groups.
    • -
  • Admin authority inherits for structural operations on the tree (add/remove members, manage subgroups).
    • -
-

Restricting access via subgroups

-

To restrict access to specific contexts, create a subgroup and add only the intended members. Contexts within that subgroup are only accessible to its direct members (and inherited parent members). This replaces the former per-context visibility/allowlist model with a simpler group-membership-based approach.

-
- -
-

Capabilities

-

Beyond the base role, fine-grained capabilities gate specific actions.

-
    -
  • Admins always pass capability checks.
    • -
  • Default capabilities can be configured per group and applied to new members automatically.
    • -
-

Flags

-

Each capability corresponds to a bit in the member’s capability set (shown conceptually below).

-
// conceptual bit layout (crates/context/config/src/lib.rs)
-bit 0 — CAN_CREATE_CONTEXT
-bit 1 — CAN_INVITE_MEMBERS
-bit 2 — CAN_JOIN_OPEN_SUBGROUPS
-bit 3 — MANAGE_MEMBERS
-bit 4 — MANAGE_APPLICATION
-bit 5 — CAN_CREATE_SUBGROUP
-bit 6 — CAN_DELETE_SUBGROUP
-bit 7 — CAN_MANAGE_VISIBILITY
-bit 8 — CAN_MANAGE_METADATA
-
-

CAN_CREATE_CONTEXT

Bit 0. Register new contexts in this group.

-

CAN_INVITE_MEMBERS

Bit 1. Issue invitations for new members.

-

CAN_JOIN_OPEN_SUBGROUPS

Bit 2. Be inherited as a member of Open subgroups beneath this group (granted to non-admins by default; admins revoke it per-member as a deny-list).

-

MANAGE_MEMBERS

Bit 3. Add or remove non-admin members and set their roles.

-

MANAGE_APPLICATION

Bit 4. Set the target application and handle migration.

-

CAN_CREATE_SUBGROUP

Bit 5. Create a subgroup directly under the namespace root (the creator becomes its owner/admin). Honored only at root level — the apply-time check must be verifiable by every namespace member.

-

CAN_DELETE_SUBGROUP

Bit 6. Cascade-delete a subgroup and its subtree. (A subgroup’s owner and namespace admins can also delete it.)

-

CAN_MANAGE_VISIBILITY

Bit 7. Flip a subgroup’s visibility (Open ↔ Restricted) without full admin on it.

-

CAN_MANAGE_METADATA

Bit 8. Set the name/data metadata record of this group, its members, or its registered contexts. (A member may always set their own member metadata without this bit.)

-
-
- -
-

Context access

-

Context access is determined entirely by group membership. Any member of a group can join any context within that group. To restrict access to a subset of contexts, create a subgroup and register those contexts there — only subgroup members (direct or inherited) can join them.

-

By default, auto_join: true causes joining a group to automatically subscribe to all its contexts. Context membership is implicit from group membership — no separate governance op is required.

-
- -
-

CRDT state

-

All application state uses CRDTs for conflict-free synchronization. State changes produce deltas that are broadcast via gossip. Every node converges to the same logical state even when messages arrive in different orders.

-

Built-in types

-

Pick the structure that fits your app; the runtime handles merging.

-
-

UnorderedMap

Key-value map with last-write-wins per key.

-

Vector

Ordered, append-only list of elements.

-

Counter

Distributed counter (G-Counter style, per-node slots).

-

LwwRegister

Single value with last-write-wins by timestamp.

-

ReplicatedGrowableArray (RGA)

Character-level text CRDT for collaborative editing.

-
-
-What “converge” means for your app -
-

Peers may receive updates in different orders. CRDT rules guarantee that once all deltas are applied, every replica shows the same logical state. You design against the data types above instead of inventing your own merge logic.

-
-
-
- -
-

Cross-context communication

-

xcall is a fire-and-forget call from one context’s WASM into another context on the same node.

-
    -
  • Calls are queued while your handler runs and execute after the current commit completes.
    • -
  • The caller gets no return value, but the outcome — success, denial, or target error — is broadcast as an XCall event for observers.
    • -
  • The target context processes the call as its own member.
    • -
-

Authorization

-

An xcall must clear three checks before it runs; a failure is denied (and reported on the XCall event) without aborting the caller:

-
    -
  • Namespace boundary (L1) — the source and target contexts must resolve to the same namespace. Cross-namespace xcalls are refused by the node. Always enforced, fail-closed.
    • -
  • Entry-point gate (L3) — opt-in. If the target app marks any method #[app::xcall], only those methods are reachable via xcall; an app that marks none stays ungated.
    • -
  • Provenance (L2a) — the target can read the verified source context via env::xcall_origin(). The node sets this; a guest cannot forge it.
    • -
-

When to use it

-

Good fits: secondary indexes, notifications, audit logs, or any work that should not block the caller’s transaction on a synchronous response.

-
-Same-node scope -
-

xcall is intentionally local to one node. For replication or network-wide effects, use normal context state and sync; use xcall for coordination between contexts co-located on that peer.

-
-
-
- -
-
- - - diff --git a/architecture/config-reference.html b/architecture/config-reference.html deleted file mode 100644 index 19dc049227..0000000000 --- a/architecture/config-reference.html +++ /dev/null @@ -1,693 +0,0 @@ - - - - -Config Reference — Calimero Core Architecture - - - - -
-
-
Home/Config Reference
- -

Config Reference

-

Complete config.toml schema with all fields, types, and defaults

- - -
-

merod init defaults

-

Running merod init generates a config.toml with these CLI-controllable defaults:

- -
-# Generate default config -merod --home ~/.calimero --node my-node init - -# All flags and their defaults: - --home ~/.calimero # base directory for data + config - --node (required) # human-readable node name - --swarm-port 2428 # libp2p swarm listen port - --server-port 2528 # HTTP API listen port - --server-host 127.0.0.1 # HTTP API listen host - --mdns true # enable mDNS discovery - --protocol /calimero/devnet/global # rendezvous namespace - --boot-nodes [] # bootstrap multiaddrs - --mock-tee false # enable mock TEE/attestation (dev/test only — refused when real KMS attestation is configured) -
-
- - -
-

Top-Level Structure

-
# config.toml — Top-level sections -[identity] # libp2p Ed25519 keypair (namespace identities are per root group in the datastore) -[swarm] # libp2p listen addresses -[bootstrap] # bootstrap node list -[discovery] # mDNS, rendezvous, relay, autonat -[server] # HTTP server config (listen, admin, jsonrpc, ws, sse, auth) -[sync] # sync timeouts and intervals -[datastore] # RocksDB data path -[blobstore] # blob storage path -[context] # context client config -[tee.kms.phala] # optional TEE/KMS config (nested under [tee.kms]) -[specialized_node] # specialized node settings
-
- - - - -
-[identity] — Node Identity -
-

Ed25519 keypair for the node. Generated automatically on merod init.

-
Rust type: IdentityConfig
- -
FieldTypeDefaultDescription
-
-
mode
-
String
-
"Standard"
-
Node operation mode. Standard for full participation, ReadOnly for read-only sync.
-
-
-
secret_key
-
String (hex)
-
(generated)
-
Ed25519 secret key in hex. Auto-generated on init, never commit to source control.
-
-
-
(removed)
-
—
-
—
-
Group/namespace identities are now auto-generated per namespace and stored in the datastore (not in config). See Namespaces.
-
- -
[identity] -mode = "Standard" -secret_key = "a1b2c3...hex..."
-
-
- - -
-[swarm] — libp2p Swarm -
-

Multiaddrs the libp2p swarm listens on for peer connections.

-
Rust type: SwarmConfig
- -
FieldTypeDefaultDescription
-
-
listen
-
Vec<Multiaddr>
-
(see below)
-
Array of multiaddrs to listen on. Supports TCP and QUIC transports.
-
- -
[swarm] -listen = [ - "/ip4/0.0.0.0/tcp/2428", - "/ip4/0.0.0.0/udp/2428/quic-v1" -]
-
-
- - -
-[bootstrap] — Bootstrap Nodes -
-

List of bootstrap peers to connect to on startup for peer discovery.

-
Rust type: BootstrapConfig
- -
FieldTypeDefaultDescription
-
-
nodes
-
Vec<Multiaddr>
-
[]
-
Multiaddrs of bootstrap peers (e.g., /ip4/1.2.3.4/tcp/2428/p2p/12D3...).
-
- -
[bootstrap] -nodes = [ - "/ip4/35.123.45.67/tcp/2428/p2p/12D3KooW..." -]
-
-
- - -
-[discovery] — Peer Discovery -
-

Configures mDNS, rendezvous, relay circuit, autonat, and address advertisement.

-
Rust type: DiscoveryConfig
- -
FieldTypeDefaultDescription
-
-
mdns
-
bool
-
true
-
Enable mDNS for local network peer discovery.
-
-
-
advertise_address
-
bool
-
false
-
Whether to advertise external addresses to the network.
-
- -

[discovery.rendezvous]

-
FieldTypeDefaultDescription
-
-
namespace
-
String
-
"/calimero/devnet/global"
-
Rendezvous namespace for peer discovery grouping.
-
-
-
registrations_limit
-
usize
-
3
-
Max concurrent rendezvous registrations.
-
- -

[discovery.relay]

-
FieldTypeDefaultDescription
-
-
registrations_limit
-
usize
-
3
-
Max concurrent relay circuit registrations.
-
- -

[discovery.autonat]

-
FieldTypeDefaultDescription
-
-
probe_interval
-
Duration
-
10s
-
Interval between autonat probes for NAT detection.
-
-
-
max_candidates
-
usize
-
5
-
Maximum number of autonat probe candidates.
-
- -
[discovery] -mdns = true -advertise_address = false - -[discovery.rendezvous] -namespace = "/calimero/devnet/global" -registrations_limit = 3 - -[discovery.relay] -registrations_limit = 3 - -[discovery.autonat] -probe_interval = "10s" -max_candidates = 5
-
-
- - -
-[server] — HTTP Server -
-

HTTP/WebSocket/SSE API server for meroctl and external clients.

-
Rust type: ServerConfig
- -
FieldTypeDefaultDescription
-
-
listen
-
Vec<Multiaddr>
-
["/ip4/127.0.0.1/tcp/2528"]
-
Multiaddrs for the HTTP API server to listen on.
-
-
-
auth_mode
-
String
-
"Proxy"
-
Auth mode: Proxy (trust upstream headers) or Embedded (built-in JWT auth).
-
-
-
admin
-
AdminConfig
-
(enabled)
-
Admin API endpoint configuration.
-
-
-
jsonrpc
-
JsonRpcConfig
-
(enabled)
-
JSON-RPC endpoint configuration for application calls.
-
-
-
websocket
-
WsConfig
-
(enabled)
-
WebSocket endpoint for real-time event subscriptions.
-
-
-
sse
-
SseConfig
-
(enabled)
-
Server-Sent Events endpoint for event streaming.
-
- -

[server.embedded_auth]

-

Embedded authentication settings (used when auth_mode = "Embedded").

-
FieldTypeDefaultDescription
-
-
jwt_secret
-
Option<String>
-
(generated)
-
HMAC secret for signing JWT tokens. Auto-generated if not set.
-
-
-
jwt_expiry_secs
-
u64
-
86400
-
JWT token expiry in seconds (default: 24 hours).
-
-
-
refresh_expiry_secs
-
u64
-
604800
-
Refresh token expiry in seconds (default: 7 days).
-
-
-
storage
-
String
-
"rocksdb"
-
Token storage backend: rocksdb or memory.
-
-
-
cors_origins
-
Vec<String>
-
["*"]
-
Allowed CORS origins for the auth endpoints.
-
-
-
secure_cookies
-
bool
-
false
-
Send cookies with the Secure flag (requires HTTPS).
-
- -
[server] -listen = ["/ip4/127.0.0.1/tcp/2528"] -auth_mode = "Proxy" - -# Embedded auth (when auth_mode = "Embedded") -[server.embedded_auth] -jwt_expiry_secs = 86400 -refresh_expiry_secs = 604800 -storage = "rocksdb" -cors_origins = ["*"] -secure_cookies = false
-
-
- - -
-[sync] — Sync Engine -
-

Timeouts and intervals for the state synchronization engine.

-
Rust type: SyncConfig
- -
FieldTypeDefaultDescription
-
-
timeout_ms
-
u64
-
30000
-
Maximum time in ms to wait for a sync response before timing out.
-
-
-
interval_ms
-
u64
-
5000
-
Base interval in ms between sync rounds for a context.
-
-
-
frequency_ms
-
u64
-
10000
-
Minimum ms between consecutive sync attempts for the same context.
-
- -
[sync] -timeout_ms = 30000 -interval_ms = 5000 -frequency_ms = 10000
-
-
- - -
-[datastore] — RocksDB Storage -
-

Path for the RocksDB persistent storage engine.

-
Rust type: DataStoreConfig
- -
FieldTypeDefaultDescription
-
-
path
-
PathBuf
-
"data"
-
Relative or absolute path to the RocksDB data directory.
-
- -
[datastore] -path = "data"
-
-
- - -
-[blobstore] — Blob Storage -
-

Path for binary blob (WASM applications, large files) storage.

-
Rust type: BlobStoreConfig
- -
FieldTypeDefaultDescription
-
-
path
-
PathBuf
-
"blobs"
-
Relative or absolute path to the blob storage directory.
-
- -
[blobstore] -path = "blobs"
-
-
- - -
-[context] — Context Client -
-

Configuration for the context management subsystem.

-
Rust type: ContextConfig
- -
FieldTypeDefaultDescription
-
-
client
-
ContextClientConfig
-
(default)
-
Context client connection and retry settings.
-
- -
[context] -# Uses defaults — typically no manual configuration needed
-
-
- - -
-[tee.kms.phala] — TEE / KMS -
-

Optional Trusted Execution Environment and Key Management Service configuration. Only nodes with a [tee] section perform the KMS key-fetch and attestation flow at startup; other nodes use libp2p as usual without KMS. The section is nested: [tee] contains [tee.kms], which contains the provider block [tee.kms.phala] (Phala Cloud KMS / mero-kms-phala is currently the only provider). There is no flat enabled / kms_url key — presence of the [tee] block enables TEE mode, and the endpoint is tee.kms.phala.url. See TEE Mode for the full startup flow.

-
Rust type: TeeConfig / KmsConfig / PhalaKmsConfig (defined in crates/config/src/lib.rs; consumed by crates/merod/src/kms/mod.rs)
- -

merod run --mock-tee

-

Dev/test only. Pass --mock-tee (or set MEROD_MOCK_TEE=1) to merod run to make the fleet-join and attest handlers produce and accept mock attestation quotes without real TDX hardware. merod will refuse to start (bail!) if the node's [tee.kms.phala.attestation] has real attestation configured (TeeConfig::has_real_attestation returns true). A loud warning is emitted on startup whenever --mock-tee is active; an additional warning fires if a Phala KMS provider is configured but real attestation is disabled (likely misconfiguration). The flag is threaded through NodeConfig::mock_tee → AdminState::mock_tee at runtime and is never persisted to config.toml. Never use in production.

- -

[tee.kms.phala]

-
FieldTypeDefaultDescription
-
-
url
-
Url
-
(required)
-
URL of the mero-kms-phala service (the KMS endpoint). Required when the [tee] block is present.
-
- -

[tee.kms.phala.attestation]

-

KMS self-attestation verification policy (verified via POST /attest before key requests).

-
FieldTypeDefaultDescription
-
-
enabled
-
bool
-
false
-
Enable KMS attestation verification before requesting keys.
-
-
-
accept_mock
-
bool
-
false
-
Accept mock attestation quotes. Development/testing only — bypasses real attestation guarantees. When accept_mock = false (or enabled = true with real measurement values), TeeConfig::has_real_attestation returns true and merod run --mock-tee will be refused at startup.
-
-
-
allowed_tcb_statuses
-
Vec<String>
-
["UpToDate"]
-
Allowed TCB statuses for KMS quote verification.
-
-
-
allowed_mrtd
-
Vec<String>
-
[]
-
Allowed KMS MRTD values (hex, with or without 0x prefix). Required when enabled=true and accept_mock=false.
-
-
-
allowed_rtmr0
-
Vec<String>
-
[]
-
Allowed KMS RTMR0 values (hex). Required when enabled=true and accept_mock=false.
-
-
-
allowed_rtmr1
-
Vec<String>
-
[]
-
Allowed KMS RTMR1 values (hex). Required when enabled=true and accept_mock=false.
-
-
-
allowed_rtmr2
-
Vec<String>
-
[]
-
Allowed KMS RTMR2 values (hex). Required when enabled=true and accept_mock=false.
-
-
-
allowed_rtmr3
-
Vec<String>
-
[]
-
Allowed KMS RTMR3 values (hex). Required when enabled=true and accept_mock=false.
-
-
-
binding_b64
-
Option<String>
-
None
-
Optional base64-encoded 32-byte binding value for /attest. If unset, merod uses the default domain-separator binding.
-
-
-
policy_json_path
-
Option<Utf8PathBuf>
-
None
-
Optional absolute path to an externally-generated attestation policy JSON (e.g. a signed mero-tee release policy). Must reside under /etc/calimero or /run/calimero. Its allowlists merge over the inline values above.
-
- -

[tee.kms.phala.tls]

-

Optional TLS hardening for KMS transport. All paths must be absolute and point to existing PEM files; setting any TLS path requires tee.kms.phala.url to use https://.

-
FieldTypeDefaultDescription
-
-
ca_cert_path
-
Option<Utf8PathBuf>
-
None
-
PEM-encoded CA certificate path for private trust roots. Added to merod's trust store for KMS TLS.
-
-
-
client_cert_path
-
Option<Utf8PathBuf>
-
None
-
PEM-encoded client certificate path for mTLS. Must be set together with client_key_path.
-
-
-
client_key_path
-
Option<Utf8PathBuf>
-
None
-
PEM-encoded client private key path for mTLS. Must be set together with client_cert_path.
-
- -
[tee.kms.phala] -url = "https://kms-host:8443/" - -[tee.kms.phala.attestation] -enabled = true -accept_mock = false -allowed_tcb_statuses = ["UpToDate"] -allowed_mrtd = ["<trusted_kms_mrtd_hex>"] -allowed_rtmr0 = ["…"] # allowed_rtmr1..3 similarly in production - -[tee.kms.phala.tls] -# ca_cert_path = "/etc/calimero/kms-ca.pem" -# client_cert_path = "/etc/calimero/kms-client.pem" -# client_key_path = "/etc/calimero/kms-client.key"
-
-
- - -
-[specialized_node] — Specialized Node -
-

Settings for specialized node roles (e.g., TEE nodes that handle key shares).

-
Rust type: SpecializedNodeConfig
- -
FieldTypeDefaultDescription
-
-
invite_topic
-
String
-
"mero_specialized_node_invites"
-
Gossipsub topic for receiving specialized node invitations.
-
-
-
accept_mock_tee
-
bool
-
false
-
Accept mock TEE attestations (for development/testing only).
-
- -
[specialized_node] -invite_topic = "mero_specialized_node_invites" -accept_mock_tee = false
-
-
- - -
-

Complete Example

-

A typical production config.toml with commonly customized fields:

-
[identity] -mode = "Standard" - -[swarm] -listen = [ - "/ip4/0.0.0.0/tcp/2428", - "/ip4/0.0.0.0/udp/2428/quic-v1" -] - -[bootstrap] -nodes = [] - -[discovery] -mdns = true -advertise_address = false - -[discovery.rendezvous] -namespace = "/calimero/devnet/global" -registrations_limit = 3 - -[discovery.relay] -registrations_limit = 3 - -[discovery.autonat] -probe_interval = "10s" -max_candidates = 5 - -[server] -listen = ["/ip4/127.0.0.1/tcp/2528"] -auth_mode = "Proxy" - -[sync] -timeout_ms = 30000 -interval_ms = 5000 -frequency_ms = 10000 - -[datastore] -path = "data" - -[blobstore] -path = "blobs" - -[specialized_node] -invite_topic = "mero_specialized_node_invites" -accept_mock_tee = false
-
- -

Governance Migration

-

Guide for migrating between group governance modes.

- -
-

Default Configuration

-
-merod init --group-governance local -
-

Local governance is the default (and only) governance mode. Group operations are signed locally and propagated via gossip.

-
- -
-

Backup

-

Back up the node data directory (RocksDB store path in config.toml) regularly. The group_store contains all governance state and can be rebuilt from the persistent op log, but a backup provides faster recovery.

-
- -
-
- - - - - diff --git a/architecture/crates/auth.html b/architecture/crates/auth.html deleted file mode 100644 index f39e35caca..0000000000 --- a/architecture/crates/auth.html +++ /dev/null @@ -1,347 +0,0 @@ - - - - -Auth Service — Calimero Core Architecture - - - -
-
-
Home/System Overview/Auth Service
- -

Auth Service

-

mero-auth

- - -

Purpose

-

Forward authentication service for the Calimero node. Handles JWT issuing and verification, pluggable authentication providers (NEAR, Ethereum, custom), a challenge/nonce flow for wallet-based auth, and key management. Can run embedded in the Axum server or behind an external reverse proxy.

- -
-
JWT
token system
-
2
auth modes
-
N−1
pluggable providers
-
Ed25519
key management
-
- - -
-

Architecture

-

The AuthService coordinates four subsystems: token management, provider routing, key storage, and secret management. Incoming token requests are routed by auth_method to the appropriate provider.

- - - - - - - - - - - TokenRequest - auth_method + credentials - - - - - - AuthService - - - - TokenManager - encode / decode with jsonwebtoken - JwtConfig · Access + Refresh tokens - - - - AuthProviders - pluggable · route by auth_method - - - Ethereum - - - Custom - - - - KeyManager - signing keys - Ed25519 · rotation - - - - SecretManager - JWT secrets - HMAC · rotation - - - - - - - - - RocksDB / In-Memory Store - - - - - - - TokenResponse - access_token + refresh_token - - - - - - Challenge Flow - 1. Request challenge nonce - 2. Sign with wallet key - 3. Submit signed challenge - - - - - Token requests routed by auth_method to pluggable providers — challenge/nonce for wallet-based authentication - - -
-
AuthService boundary
-
Token management
-
Provider routing
-
Key management
-
Secret management
-
-
- - -
-

JWT Details

-

Token structure, claims schema, and the challenge-based authentication flow.

- -
-
-

TokenManager

-

Encodes and decodes JWTs using the jsonwebtoken crate. Configured via JwtConfig which specifies algorithm (HS256/HS384), secret rotation policy, and expiration windows for access and refresh tokens.

-
-
encode()
-
Claims → signed JWT string
-
-
-
decode()
-
JWT string → validated Claims
-
-
-
refresh()
-
Refresh token → new access token
-
-
-
-

TokenType

-

Two token variants with different lifetimes and permissions scope.

-
-pub enum TokenType {
-    Access,  // short-lived, carries permissions
-    Refresh, // long-lived, exchange for new access
-} -
-
-
- -

Claims Schema

-

Breaking change: The wallet field has been removed from RootKey and ClientKey. Existing stored or transmitted JSON containing a wallet field will no longer deserialize into these structs. The WalletType and NearNetworkId enums, and the wallet_type constructor parameter, are also deleted.

-
-pub struct Claims {
-    sub: String,          // key id (public key hash)
-    iss: String,          // issuer (node identity)
-    aud: String,          // audience
-    exp: u64,             // expiration timestamp
-    iat: u64,             // issued at
-    jti: String,          // unique token id (UUID)
-    permissions: Vec<String>, // granted permissions
-    node_url: Option<String>, // optional node URL
-} -
- -

ChallengeClaims

-
-pub struct ChallengeClaims {
-    challenge: String, // random nonce string
-    exp: u64,          // challenge expiration (short TTL)
-    iat: u64,          // issued at
-} -
- -
- Challenge / Nonce Authentication Flow -
-
1
-

Request Challenge

-

Client calls the challenge endpoint. The AuthService generates a random nonce, wraps it in ChallengeClaims, and returns a short-lived challenge JWT (e.g. 5-minute TTL).

-
-
2
-

Sign Challenge

-

Client signs the challenge nonce with their wallet key (Ethereum secp256k1, or a custom provider key). The signature proves ownership of the corresponding public key without revealing the private key. Note: NEAR Ed25519 wallet signing is no longer supported as the NEAR Wallet provider has been removed.

-
-
3
-

Submit Signed Challenge

-

Client sends the signed challenge back along with their public key and auth_method. The appropriate AuthProvider verifies the signature against the public key and challenge nonce.

-
-
4
-

Issue Tokens

-

On successful verification, TokenManager generates an access token and refresh token with the authenticated key ID as sub claim. Tokens are returned to the client for subsequent API calls.

-
-
-
-
- - -
-

Auth Modes

-

Two operational modes that determine how authentication is enforced on incoming requests.

-
The NEAR Wallet authentication provider (NearWalletProvider) has been removed. The near-crypto, near-jsonrpc-client, and near-primitives Cargo dependencies are no longer present. Existing integrations that relied on NEAR wallet-based login must migrate to a Custom provider.
- -
-
-

Proxy Mode

- external -

No in-process JWT verification. Authentication is handled by an external reverse proxy (nginx, Envoy, etc.) that sits in front of the Calimero node. The proxy validates tokens and forwards authenticated requests with identity headers.

-

The node trusts the proxy's identity injection and does not perform any token validation itself. Suitable for enterprise deployments with existing auth infrastructure.

-
-// Proxy mode config
-struct ProxyConfig {
-    trusted_header: String, // e.g. "X-Auth-Key"
-} -
-
-
-

Embedded Mode

- recommended -

Full JWT lifecycle managed in-process. The AuthGuardLayer middleware intercepts every request, extracts the token from the Authorization: Bearer header or ?token= query parameter, and verifies it against the local secret.

-

On successful verification, an AuthenticatedKey is injected into the Axum request extensions, making the caller's identity available to all downstream handlers.

-
-// Embedded mode middleware
-struct AuthGuardLayer {
-    token_manager: Arc<TokenManager>,
-}

-// Injected into request extensions
-struct AuthenticatedKey {
-    key_id: String,
-    permissions: Vec<String>,
-} -
-
-
-
- - -

Storage

-

Auth state can be persisted in RocksDB (production) or held in-memory (development/testing). The storage backend is selected at initialization and abstracted behind a trait interface.

- -
-
-

RocksDB Backend

-

Persistent storage for signing keys, refresh token metadata, and revocation lists. Uses a dedicated column family within the node's RocksDB instance. Survives node restarts.

-
-
-

In-Memory Backend

-

Ephemeral storage for development and testing. All auth state is lost on restart. Useful for local development where persistent tokens are unnecessary.

-
-
- -crates/mero-auth - -

API Reference

- -
-

Public Endpoints

-
-GET /auth/login — React SPA for authentication -GET /auth/challenge — Get signing challenge -POST /auth/token — Exchange challenge for root key JWT -POST /auth/refresh — Refresh expired tokens -GET /auth/providers — List available providers -GET /auth/identity — Service information -GET /auth/validate — Forward auth validation endpoint -
- -

Protected Endpoints

-

Note: The wallet field has been removed from RootKey and ClientKey response objects. Clients that read or write the wallet field will break and must be updated.

-
-— Root Key Management — -GET /admin/keys — List root keys -POST /admin/keys — Create root key -DELETE /admin/keys/{key_id} — Delete root key - -— Client Key Management — -GET /admin/keys/clients — List client keys -POST /admin/client-key — Generate client key -DELETE /admin/keys/{key_id}/clients/{client_id} — Delete client key - -— Permissions — -GET /admin/keys/{key_id}/permissions — Get key permissions -PUT /admin/keys/{key_id}/permissions — Update permissions - -— Token & System — -POST /admin/revoke — Revoke current token -GET /admin/metrics — Service metrics -
-
- -
-

Deployment

-
-
-

Nginx (Forward Auth)

-

Configure auth_request to proxy to /auth/validate. Protected routes get validated transparently. Auth routes (/auth/*, /admin/*) proxy directly to the auth service.

-
-
-

Traefik

-

Use forwardauth.address middleware pointing to http://auth:3001/auth/validate. Label auth service routes with PathPrefix(/auth) and node routes with the forward-auth middleware.

-
-
-
- -
-

Configuration

-

All settings can be overridden via AUTH_-prefixed environment variables (e.g. AUTH_JWT__ISSUER, AUTH_STORAGE__TYPE).

-
-config.toml example -
-
-listen_addr = "0.0.0.0:3001" - -[jwt] -issuer = "calimero-auth" -access_token_expiry = 3600 -refresh_token_expiry = 2592000 - -[storage] -type = "rocksdb" -path = "/data/auth_db" - -[providers] -# near_wallet provider has been removed -
-
-
-
- -
-
- - - diff --git a/architecture/crates/context.html b/architecture/crates/context.html deleted file mode 100644 index 747d73f785..0000000000 --- a/architecture/crates/context.html +++ /dev/null @@ -1,691 +0,0 @@ - - - - -Context & Groups — Calimero Core Architecture - - - -
-
-
Home/System Overview/Context & Groups
- -

Context & Groups

-

calimero-context + calimero-context-primitives + calimero-context-config

- - -
-

Purpose

-

ContextManager is an Actix actor that manages contexts (application instances), groups (governance units), and namespace governance DAGs. Every context belongs to a group within a namespace. The manager holds per-namespace DagStores (via HashMap<namespace_id, Arc<Mutex<DagStore>>>), handles 30+ message types, coordinates with a modular GroupStore for persistence, and runs lifecycle recovery and Prometheus metrics collection.

- - - - - - - - - - ContextManager - - actor - - contexts cache · namespace_dags HashMap - GroupStore (modular) · namespace governance - 30+ Handler<ContextMessage> variants - lifecycle recovery · metrics · 30s heartbeat - crates/context/src/lib.rs - - - - Server / RPC - ContextClient - - - - - NodeManager - delta routing - - - - - GroupStore - persistence - - - - - NetworkManager - gossip publish - - -
- - -
-

Module Structure

- -

calimero-context

-
lib.rs
ContextManager actor — startup, fields, Actix lifecycle, namespace DAG management
-
lifecycle.rs
Startup recovery: recover_in_progress_upgrades(), start_namespace_heartbeat() (30s periodic). ContextManager::started calls auto_follow::spawn before self_purge::spawn — this ordering is load-bearing: it ensures the auto-follow broadcast receiver exists before the curative startup sweep can emit OpEvent::ContextRegistered events, preventing those events from being silently dropped.
-
metrics.rs
Prometheus metrics: execution count/duration, namespace retry/decode events, membership policy rejections
-
group_store/mod.rs
Authoritative persistence layer (refactored into modules) — apply_local_signed_group_op
-
group_store/namespace.rs
Namespace identity resolution: resolve_namespace(), get_or_create_namespace_identity_bundle()
-
group_store/namespace_governance.rs
Namespace governance DAG application, key unwrapping, skeleton storage
-
self_purge.rs
Startup curative sweep: redrive_stranded_ops_sweep runs once after subscribing to op_events and before the event loop. Iterates every namespace the node has an identity for, finds groups with buffered ops whose key is already held (inverse of groups_awaiting_key), and re-drives each via redrive_encrypted_ops_for_group_counted. Loops until a full pass applies nothing new (cross-signer convergence), bounded by MAX_PASSES=64, with pass-narrowing so only groups that made progress are revisited. Errors are logged and swallowed; the sweep never blocks startup.
-
auto_follow.rs
auto_follow::spawn now subscribes to op_events synchronously on the caller thread before spawning the async task, passing the receiver into run(). Must be spawned before self_purge::spawn (see lifecycle.rs).
-
group_store/namespace_membership.rs
Namespace membership tracking and policy enforcement
-
group_store/membership.rs
Group membership management with parent chain walking (max depth 16)
-
group_store/group_keys.rs
Group encryption key management: generate, store, wrap/unwrap via ECDH. CapabilitiesRepository::set_subgroup_visibility is called during group_created::apply to write the subgroup's visibility key atomically on first apply (gated on has_subgroup_visibility absence, so it never clobbers later SubgroupVisibilitySet flips).
-
group_store/capabilities.rs
Per-member capability bitmask management
-
group_store/context_tree.rs
Context-group binding: register, detach, list contexts per group
-
handlers/create_group.rs
Namespace root creation (parent_group_id == None) now signs, applies, and publishes a RootOp::NamespaceCreated { founder: admin_identity } genesis op via sign_apply_and_publish_namespace_op. A NoPeersSubscribed publish error is treated as success (apply succeeded locally; publish is best-effort). Any other error triggers a full rollback of all locally-written root rows (meta, founder Admin member row, default caps, group key, signing key, optional name metadata) via individual idempotent deletes before returning Err. The namespace identity row is deliberately not rolled back so a retry gets the same identity and founder. The DAG head does not need rollback because apply_signed_op is head-atomic (the head advances only after a successful apply).
-
group_store/tests.rs
Comprehensive group store tests including namespace identity, governance, key delivery. Also contains dag_releases_out_of_order_ops_and_the_projection_converges: a #[tokio::test] that builds a four-op causal chain spanning the admin, membership, ACL, and data planes, feeds it through DagStore<Op> + UnifiedApplier in four delivery orders (in-order, full reverse, and two interleaved), and asserts that every op is applied (none stuck pending) and that scope_root matches the result of direct in-order folding in all cases. Also contains persisted_op_log_reloads_and_replays_to_the_same_projection: a #[tokio::test] that persists a three-op causal chain of mixed-plane ops (AdminChanged → MemberAdded → SetWriters) via persist_op, reloads them from a fresh store handle via load_scope_ops, replays the loaded ops through DagStore<Op> + UnifiedApplier, and asserts the reconstructed scope_root matches an independent in-memory fold of the same ops; also asserts that loading an empty scope returns zero rows.
-
unified_op_store.rs
Public module exposing two functions for durable unified-op persistence. persist_op borsh-serializes an Op and writes it under ScopeUnifiedOpKey in Column::UnifiedOp; the write is idempotent because the key is content-addressed by op id. load_scope_ops performs a seek-then-entries range scan over the scope prefix and deserializes all rows back into Vec<Op>. Important: rows are returned in key order, not causal order; callers must replay the loaded ops through DagStore<Op> + UnifiedApplier to recover causal ordering and projections. DAG heads are deliberately not persisted here. No production apply or sync path calls these functions yet. Exported as pub mod unified_op_store from crates/context/src/lib.rs.
-
unified_applier.rs
UnifiedApplier — implements DeltaApplier<Op> backed by an Arc<std::sync::RwLock<ScopeProjections>>. Its apply method acquires a write lock (held only across the synchronous ingest_op call, never across an .await), calls ScopeProjections::ingest_op with the delta's payload, and returns Ok(()). Lock poison is recovered rather than propagated, matching the stance already documented in NodeState::read_scope_projections. Two constructors: new() for a private/standalone projection, and with_projection(Arc<RwLock<ScopeProjections>>) as the C2.1 dual-write seam that attaches to the node's live shared projection. A projection() accessor exposes the Arc for inspection. Exported as pub mod unified_applier from crates/context/src/lib.rs, making UnifiedApplier part of the crate's public API.
-
projection_membership_equivalence.rs
Equivalence tests asserting member_at_cut_authoritative agrees with the live resolver: (1) projection_matches_live_across_inherited_join_and_root_removal — after an inherited join, member_at_cut_authoritative returns Some(true); after root removal that live rejects, it must NOT return Some(true) (over-grant guard); (2) projection_matches_live_across_leave_and_rejoin_inheritance — after a leave-and-rejoin cycle, member_at_cut_authoritative restores access
-
governance_dag.rs
DAG bridge — GroupGovernanceApplier + NamespaceGovernanceApplier. The standalone GroupGovernanceApplier::apply continues to call the plain apply_local_signed_group_op (live-fallback gates); a code comment documents why: a SignedGroupOp's parent_op_hashes belong to the per-group op log, not the namespace governance log the projection is keyed by, so passing them to EphemeralProjectionAuthorizer would resolve against the wrong id-space and silently no-op. Defines the AtCutAuthorizer trait with three required methods: is_admin_at_cut, is_admin_or_capability_at_cut, and the newly added is_last_admin_at_cut. All three must return None when parents is empty — this is the empty-cut contract enforced across every method: violating it would falsely reject genesis ops by evaluating against an empty fold. Both LiveFallbackAuthorizer and EphemeralProjectionAuthorizer implement the trait. LiveFallbackAuthorizer implements all three returning None. EphemeralProjectionAuthorizer now holds a Mutex<Option<(ContextGroupId, Arc<FoldedProjection>)>> cache; its private folded method checks the cache before calling ScopeProjections::ephemeral_projection, reusing the folded DAG for the same group on a cache hit. All three trait methods go through folded. EphemeralProjectionAuthorizer::is_last_admin_at_cut delegates to ScopeProjections::is_last_admin_at_cut, guarding against empty parents before folding. The real group-auth shadow only fires on the namespace-envelope group-op path, not the standalone DAG applier path.
-
scope_projection.rs
Shadow unified-op projection — op_from_signed_namespace_op() derives projection ops from namespace governance deltas; folded into scope projections on DAG apply (additive, not yet authoritative). ScopeProjections::namespace_current_heads resolves a ContextGroupId to its owning namespace, reads that namespace's DAG head record, and returns the parent hashes — the "now" frontier used as the cut argument for current-state membership reads via the projection. Returns None when the group cannot be resolved or the head record is unreadable; callers should fall back to live membership in that case. - -Per-query projection helpers (ephemeral path): ScopeProjections::member_now_ephemeral builds a fresh ScopeProjections from the store on each call — resolves the namespace, folds the governance DAG via ops_for_namespace (the stable indirection layer that currently delegates to collect_namespace_ops), then reads member_at_cut at the heads recorded after the fold (so a concurrent governance-op advance cannot name ops the fold hasn't processed). Returns None when the namespace is unresolvable, when ops_for_namespace reports a store fault (surfaced with a warn), or when the cut-completeness guard finds the heads don't cover the full fold. Intended for context-layer handlers that lack a cached projection. - -ScopeProjections::member_now_checked is the single decision point for all membership gates. It calls member_now_ephemeral first; when the projection can decide it acts on that verdict and cross-checks MembershipRepository::is_member best-effort (a transient live error does not override a projection verdict). When projection and live definitely disagree it emits a tracing::warn with marker=unified_projection_divergence and plane=membership-query. When the projection returns None it falls back to live as authoritative and propagates live's error normally. - -ScopeProjections::is_last_admin_at_cut answers, at the op's parent cut (pre-mutation), whether removing or demoting the given member would orphan the group's admin set. It mirrors live logic exactly: a member counts as admin if they hold a direct Admin-role row or are the genesis admin; another admin requires a distinct Admin-role row (the genesis admin alone does not satisfy it). Returns None when ancestry is incomplete, deferring to live. The causal-position semantics are intentional: because the read is at the parent cut rather than current state, the result can legitimately differ from a live current-state read under concurrency — this is the causal-honor property, not a bug. - -ScopeProjections::membership_path_at_cut calls the projection's member_path_at_cut walk using the auth-cut context, returning Option<calimero_authz::MemberPathAtCut>. Returns None when the parents slice is empty (empty-cut contract), when the namespace is unresolvable, or when the fold is incomplete. - -ScopeProjections::role_at_cut_for_group resolves the effective role of a member in a group at the governance cut. It uses member_path_at_cut (the same derivation already validated by the membership-role plane) to handle three cases: direct membership (folded role row), inherited-via-admin (always Admin), and inherited-via-anchor (anchor's role row). It gates on cut_ancestry_complete via auth_cut_context and returns None rather than guessing when the anchor row is missing, preventing spurious divergence reports. Returns None when ancestry is incomplete, the member is absent from the projection, or the namespace is unresolvable — callers should fall back to live role reads in those cases. - -ScopeProjections::scope_root_for computes a governance-folded convergence root for a given scope as SHA-256(entities_root ‖ acl_hash ‖ governance_hash). Returns None when the scope has not been folded yet; callers must treat a None return as "cannot compare" rather than a divergence — never as evidence of mismatch. The ACL component is currently empty because SetWriters ops are not yet routed through the governance DAG; this will change at C2. - -ScopeProjections::group_scope_root_ephemeral is a companion free function that builds an ephemeral ScopeProjections from the raw store (no NodeState required) and adds an ancestry-completeness guard: if the governance DAG cut is not fully folded it returns None to avoid false divergences from mid-backfill nodes. - -ScopeProjections::ops_for_namespace is the stable indirection point for all projection-rebuild call sites (backfill_namespace, member_now_ephemeral/ephemeral_fold, and the scope_root reconstruction). It currently delegates to collect_namespace_ops (the governance-DAG walk). The C2.2b flip — switching to load_scope_ops (the unified op-store) — is explicitly deferred: an encrypted MemberAdded op can be applied to the governance DAG before its group key arrives; the apply-time dual-write freezes it as Noop in the op-store, while collect_namespace_ops re-decrypts at read time once the key lands. Reading from the op-store directly would therefore silently drop those members and break joiner sync. The C2.2c fix for this is ScopeProjections::repersist_namespace_ops (see below). - -ScopeProjections::repersist_namespace_ops repairs the op-store after a group key is delivered. It re-walks a namespace through collect_namespace_ops (the canonical read-time decryption path) and calls persist_op for every op found. Because ops are content-addressed by the same op id, a now-decryptable op (e.g. MemberAdded) overwrites the stale Noop entry that was frozen at apply time when the key was absent. The method is best-effort (errors are logged, not propagated), idempotent, and bounded by the same MAX_BACKFILL_OPS walk cap used elsewhere. It should be called after any group key delivery to ensure the op-store remains consistent with the governance-DAG fold. - -ScopeProjections::persist_namespace_head_ops is a new public method that mirrors locally-authored governance ops into the unified op-store immediately after they are published. Rather than walking only the DAG heads (which would be racy — a concurrent peer op applied during the author's publish round-trip can advance the head, orphaning the just-authored op from a heads-only walk), the method delegates to repersist_namespace_ops, which performs a full gov-DAG fold bounded by MAX_BACKFILL_OPS. The call is idempotent and best-effort; failures are logged and never affect authoring success. It is wired into four handlers: create_group (after GroupCreated is successfully reported), delete_group (after GroupDeleted is applied), join_context (after MemberJoinedOpen is published), and join_subgroup_inheritance (after MemberJoinedOpen is published). In each case the call is a no-op if the preceding apply or publish step errored. - -ScopeProjections::scope_root_from_op_store provides the op-store's view of a scope's governance root. It loads all persisted ops for a given ScopeId via load_scope_ops, folds them into a fresh ScopeProjections instance, and returns the governance scope_root using a fixed zero entities_root. Returns Ok(None) when no ops exist for the scope (empty scope is not an error), and Err on store faults. Intended for completeness verification before C2.2b — it answers "does the op-store reconstruct to the same governance root as the maintained DAG projection?" - -ScopeProjections::check_op_store_completeness is a diagnostic-only observe gate that compares an already-computed gov-DAG fold (dag_ops) against the unified op-store. It extracts the op-ids from dag_ops, loads the persisted op-ids for the same scope via load_scope_ops, and emits a warn-level tracing event with marker op_store_incomplete (carrying missing_count, dag_count, op_store_count, and a hex-encoded sample_missing id) for every op present in the gov-DAG but absent from the store. If the op-store cannot be loaded at all it emits a distinct op_store_gate_unavailable marker and returns None — a store fault cannot masquerade as a verified-clean result. Return semantics: None = completeness unverifiable (store unreadable), Some(vec![]) = verified complete, Some(ids) = those op-ids are missing from the store. This gate is observe-only and retires at C3 Stage 4 when the projection read flips onto the op-store. - -

Note: ScopeProjections::shadow_compare_op_store and scope_root_from_governance_dag have been removed. The observe-only cross-check in refresh_projection_for_cut that triggered a full governance-DAG fold on every backfill has also been deleted. That comparison was flawed: the maintained projection holds live ingest_op-fed ops that may not be in the op-store, so a gap in the op-store could be masked by the live apply path and go undetected. The unified_op_store_divergence log marker is no longer emitted.

- -

Note: The live governance-cut membership resolver (acl_view_at), the MembershipStatus enum (Member/Removed/NeverMember/Unknown), prefix_walk_membership, the MembershipTransition state machine, heads_equal, MAX_PREFIX_WALK_NODES, and resolve_membership_from_transitions have all been deleted from governance-store/src/membership/status.rs. Membership authorization is now handled entirely by the unified projection (ScopeProjections::member_now_checked). Re-exports of these symbols have been removed from governance-store/src/membership/mod.rs, governance-store/src/lib.rs, and context/src/lib.rs.

-
handlers/
30+ handler files, one per ContextMessage variant (create, join, delete, execute, namespace ops, etc.). Membership gates in get_cascade_status, get_group_upgrade_status, list_group_contexts, and list_all_groups now call ScopeProjections::member_now_checked instead of MembershipRepository::is_member directly — the direct MembershipRepository import has been removed from those four modules. list_all_groups silently skips groups whose membership cannot be determined (a tracing::warn is emitted) rather than aborting the entire listing.
- -

calimero-context-primitives

-
client/mod.rs
ContextClient typedef — thin async façade wrapping LazyRecipient<ContextMessage>
-
client/sync.rs
Sync-related client helpers (group delta request/response coordination)
-
client/context_api.rs
Context-level API methods (create, join, delete, execute, config queries)
-
client/crypto.rs
Cryptographic helpers (signing, key derivation, namespace identity)
-
local_governance/mod.rs
Wire types: SignedGroupOp, GroupOp, SignedNamespaceOp, NamespaceOp — Ed25519, borsh, schema v3. ⚠ WIRE BREAK: RootOp::GroupCreated gains a boolean restricted field; nodes must reset. restricted: true is the default and preserves legacy behavior. The op-adapter projection reads restricted from the live op instead of hardcoding true. A born-Open subgroup (restricted: false) is already Open when tee_subgroup_admit reacts, so no transient direct ReadOnlyTee row is written. CreateGroupRequest gains a restricted field (default true); the REST endpoint accepts an optional visibility field ("open"|"restricted", default "restricted").
-
local_governance/tests.rs
Unit tests for governance wire types and signing
-
group.rs
Actix messages — group-level and namespace-level message types for actor communication
-
messages.rs
ContextMessage enum — top-level message envelope with 30+ variants (including namespace ops)
- -

calimero-context-config

-
client_config.rs
Configuration for context client connections and transport settings
-
types.rs
Shared configuration types used across context crates
-
- - -
-

ContextManager Actor

-

The central actor for all context and group operations. Initialized on node start, it rebuilds in-memory state from persistent storage and starts background coordination tasks.

- -
-
-

Key Fields

-
-pub struct ContextManager {
-    store: Arc<Store>,
-    node_client: NodeClient,
-    context_client: ContextClient,
-    group_dags: HashMap<GroupId, DagStore>,
-    contexts: HashMap<ContextId, ContextState>,
-    upgrade_propagators: HashMap<GroupId, UpgradeProp>,
-    // ... network, runtime refs
-} -
-
-
-

Startup Sequence

-
1
-

recover_in_progress_upgrades

-

Resume any application upgrades that were interrupted by a previous shutdown.

-
-
!
-

auto_follow::spawn (before self_purge)

-

Subscribes to op_events on the caller thread and spawns the auto-follow task. Must run before self_purge::spawn so the receiver is live before the curative sweep emits OpEvent::ContextRegistered.

-
-
!
-

redrive_stranded_ops_sweep (self_purge)

-

One-shot curative sweep: for every namespace the node has an identity for, finds groups with buffered ops whose key is already held and re-drives them via redrive_encrypted_ops_for_group_counted. Loops up to MAX_PASSES=64 until nothing new is applied.

-
-
2
-

reload_group_dags

-

For each group in the store, read the full OpLog and rebuild the in-memory DagStore via restore_applied_delta.

-
-
3
-

start_group_heartbeat (30s)

-

Periodic timer publishes GroupStateHeartbeat with current dag_heads for every group, triggering catch-up on peers.

-
-
-
- - - - - - - - Actor::started() - recover upgrades - reload DAGs - start heartbeat - - - - - OpLog (storage) - per-group sequence - borsh(SignedGroupOp) - - - - - DagStore (memory) - restore_applied_delta - rebuild heads + pending - - - - - Heartbeat (30s) - GroupStateHeartbeat - broadcast dag_heads - -
- - -
-

Handler Map

-

All 20+ ContextMessage variants, grouped by function. Each variant is handled by a dedicated file in handlers/.

- -
-
-

Context Lifecycle

-
CreateContext
Create a new context, bind to group, emit ContextRegistered ops
-
JoinGroupContext
Join a context within a group (requires group membership)
-
DeleteContext
Remove context and clean up all associated state
-
Execute
Run WASM method, produce state delta, broadcast
-
UpdateApplication
Upgrade WASM binary for a context
-
-
-

Group Governance

-
AddGroupMembers
Add one or more members to a group
-
RemoveGroupMembers
Remove members with cascade to all contexts
-
ApplySignedGroupOp
Ingest a signed governance operation into the DAG
-
SetMemberCapabilities
Grant or revoke per-member capability flags
-
UpdateMemberRole
Change member role (admin, member, etc.)
-
UpgradeGroup
Propagate application upgrade across group contexts
-
-
- -
-
-

Membership & Access

-
JoinGroup
Join a group via invitation claim flow
-
JoinGroupContext
Join a specific context within a group
-
CreateGroupInvitation
Generate a SignedGroupOpenInvitation
-
SetDefaultCapabilities
Default capability flags for new group members
-
-
-

Configuration & Sync

-
SyncGroup
Trigger group state synchronization with peers
-
SetGroupAlias
Set human-readable alias for a group
-
SetMemberAlias
Set human-readable alias for a member
-
DeleteGroup
Delete group and cascade to all owned contexts
-
DetachContextFromGroup
Unbind a context from its group
-
ListGroupMembers
Enumerate effective group members. Reads from ScopeProjections::member_entries_with when a folded projection context is available; falls back to the live MembershipRepository::list ∪ enumerate_inherited union when the projection returns None. Both paths are sorted by PublicKey before skip(offset).take(limit) to guarantee stable, path-independent pagination order across a mid-backfill transition.
-
-
- - - - - - - - Caller - Server, NodeMgr, - SyncManager - - - - - ContextMessage - enum dispatch - 20+ variants - - - - - - - - Context Lifecycle (5) - - - Group Governance (6) - - - Membership & Access (7) - - - Config & Sync (5) - - - handlers/ - one file per variant - async fn handle(&mut self, ...) - - - -
- - -
-

Projection-Backed Member Count & Migration Cohort

-

Two operations now derive their member sets from the governance projection fold rather than the live store, with a live fallback when the projection is unavailable.

- -

GetGroupInfo — member_count

-

Handler<GetGroupInfoRequest> computes member_count as the length of the identity set returned by member_identities_with on the already-built projection fold, instead of calling MembershipRepository::count + MembershipRepository::enumerate_inherited directly. The shadow divergence log that previously compared both paths is removed. If member_identities_with returns None (unfed namespace or incomplete ancestry fold), the handler falls back to the old live count + enumerate_inherited path.

- -

collect_migration_cohort

-

collect_migration_cohort calls member_identities_subtree_ephemeral at the top and returns its result immediately on success, bypassing the live list ∪ enumerate_inherited loop entirely. The previous shadow comparison block is removed. The live path is retained only as a fallback when the projection returns None.

- -

Governance Preflight & Signing Key Resolution

-

Every governance mutation handler calls governance_preflight() before signing and publishing. The common pattern is encapsulated in sign_and_publish_group_op() which handles the boilerplate.

- -

governance_preflight Flow

-
1
Resolve requester identity — from explicit parameter, or fall back to node_namespace_identity() which walks the parent chain to the namespace root
-
2
Verify group exists — load_group_meta() must return Some
-
3
Admin check (optional) — require_group_admin() if require_admin=true
-
4
Resolve signing key via ancestor walk — resolve_group_signing_key() checks self, then walks parent chain up to MAX_NAMESPACE_DEPTH (16) levels. Returns the first signing key found for the requester identity.
- -

Signing Key Hierarchy

-

When a child group is created via create_group_in_namespace, the signing key is stored only at the namespace root. Child groups resolve signing authority by walking up the parent chain:

-
-// resolve_group_signing_key (signing_keys.rs)
-for _ in 0..MAX_NAMESPACE_DEPTH {
-    if let Some(sk) = get_group_signing_key(store, &current, public_key)? {
-        return Ok(Some(sk));
-    }
-    current = get_parent_group(store, &current)?; // walk up
-}
-get_group_signing_key(store, &current, public_key) // final check at root -
-

Key revocation at the namespace level automatically propagates — child groups lose access because the ancestor walk finds nothing. No stale copies to clean up.

- -

sign_and_publish_group_op Helper

-

Encapsulates the governance handler boilerplate: preflight → clone datastore/node_client → build signer → sign, apply, publish via async actor response. Used by 7 handlers (set_group_alias, set_default_capabilities, set_default_visibility, update_group_settings, set_member_alias, set_member_capabilities, update_member_role).

- -

GroupStore Deep-Dive

-

The authoritative persistence layer for all group and governance state. Handles signature verification, state-hash optimistic locking, nonce-based idempotency, DAG maintenance, and cascading side effects.

- -

apply_local_signed_group_op Flow

-
1
-

Cap parent_op_hashes

-

Ensure parent hashes reference valid DAG heads (capped at 64 concurrent heads).

-
-
2
-

verify_signature

-

Ed25519 signature verification on the signable_bytes of the operation.

-
-
3
-

check state_hash

-

Causal-ancestry guard — the op's state_hash is checked for consistency with the current computed group state hash. Concurrent ops from different admins that were each signed against the same genesis state are both accepted; the nonce window deduplicates replays of already-seen ops without changing member count or state hash. The integration test state_hash_mismatch_does_not_reject_concurrent_ops verifies convergence: two admins independently add different members against the same genesis state, the ops are applied in opposite order on two nodes, and both nodes converge to an identical four-member set with equal compute_state_hash values.

-
-
4
-

check nonce (idempotent on duplicate)

-

Per-signer monotonic nonce. If the nonce was already seen, the op is silently treated as idempotent (no error, no re-apply).

-
-
5
-

dispatch GroupOp

-

Match on the GroupOp variant and apply the state mutation (add member, set visibility, cascade removal, etc.).

-
-
6
-

append OpLog

-

Serialize the SignedGroupOp via borsh and append to the persistent op log with an incrementing sequence number.

-
-
7
-

recompute dag_heads

-

Update the set of DAG head hashes — the new op becomes a head, its parents are removed from the head set.

-
-
8
-

set nonce

-

Record the signer's nonce for replay protection.

-
- -

Key Storage Prefixes

-

All stored in Column::Group with typed key prefixes:

-
-
-
GroupMeta
Group metadata (name, created_at)
-
GroupMember
Member records per group
-
GroupOpLog
Persisted signed operations
-
GroupOpHead
Current DAG head hashes
-
-
-
GroupLocalGovNonce
Per-signer monotonic nonce
-
GroupContextIndex
Context → group mapping
-
ContextGroupRef
Reverse group → context ref
-
GroupMemberContext
Per-member context tracking
-
-
-
GroupCapability
Member capability grants
-
GroupAlias
Human-readable group alias
-
MemberAlias
Human-readable member alias
-
-
- -
- sign_apply_and_publish helper -
-

Convenience function used by most handlers to emit governance ops. Combines three steps into one call:

-
-async fn sign_apply_and_publish(
-    &self,
-    group_id: GroupId,
-    op: GroupOp,
-) -> Result<()> {
-    // 1. Sign op with node's Ed25519 key
-    // 2. apply_local_signed_group_op (persist + DAG)
-    // 3. Publish via gossipsub on group topic
-} -
-
-
- - - - - - - - GroupOp - incoming op - - - - - verify + check - sig · state_hash - nonce · parents - - - - - dispatch - match GroupOp variant - - - - - persist - OpLog + heads + nonce - - - - - DAG updated - new heads computed - -
- - -
-

Membership Gate: Projection-First Decision Flow

-

Four handlers (GetCascadeStatus, GetGroupUpgradeStatus, ListGroupContexts, ListAllGroups) previously called MembershipRepository::is_member directly. They now route through ScopeProjections::member_now_checked, which implements a projection-first, live-fallback decision with divergence telemetry:

- -
1
member_now_ephemeral — builds a fresh ScopeProjections from the store, folds the namespace governance DAG, and reads member_at_cut at post-fold heads. Returns None on store faults, unresolvable namespace, or incomplete fold coverage.
-
2
Projection verdict — when member_now_ephemeral returns Some(verdict), that verdict is used as the gate decision. Live membership (MembershipRepository::is_member) is still consulted best-effort for cross-checking; a transient live error does not override the projection verdict.
-
3
Divergence logging — when the projection and live results definitely disagree, a tracing::warn is emitted with marker=unified_projection_divergence and plane=membership-query.
-
4
Live fallback — when the projection returns None (undecidable), MembershipRepository::is_member is used as authoritative and its errors propagate normally — except in list_all_groups, where errors cause the group to be skipped with a tracing::warn rather than aborting the whole listing.
- -

ContextClient

-

Thin async façade wrapping LazyRecipient<ContextMessage>. Used by Server (for RPC execution) and NodeManager (for delta routing) to invoke context-level operations without importing Actix directly.

- -
-pub struct ContextClient {
-    recipient: LazyRecipient<ContextMessage>,
-}

-impl ContextClient {
-    pub async fn send(&self, msg: ContextMessage) -> Result<ContextResponse>;
-} -
- -
-
-

Context Operations

-
create_context
Create new context bound to group
-
delete_context
Remove context and clean up state
-
execute
Run WASM method, produce delta (auto-resolves executor identity)
-
get_context
Query context metadata
- -

Group Operations

-
add_group_members
Add members to a group
-
remove_group_members
Remove members with cascade
-
apply_signed_group_op
Ingest signed op into group DAG
-
apply_signed_namespace_op
Ingest signed op into namespace DAG
-
set_member_capabilities
Set capability flags for member
-
-
-

Membership

-
join_group
Join group/namespace via invitation (generates namespace identity, unwraps group key)
-
create_group_invitation
Generate invitation token
-
list_namespaces
List namespaces with identity info
-
get_namespace_identity
Get this node's namespace identity public key
- -

Configuration

-
sync_group
Trigger group sync with peers
- -

Query

-
context_config
Get context configuration
-
get_context_application
Get WASM app metadata
-
get_context_member_page
Paginated member listing
-
-
-
- - -
-

Operator Notes

- -

Startup curative sweep

-

On startup the node automatically re-drives any buffered encrypted ops that are already decryptable but were never applied due to a missing GroupCreated event (e.g., arrived out of order before the key was held). The sweep runs inside self_purge::run, iterating all known namespace identities (NamespaceRepository::iter_identities) and finding groups with buffered ops whose key is already held via NamespaceRetryService::groups_with_held_key_buffered_ops. It is bounded to MAX_PASSES=64 and never blocks startup on error.

-

Spawn ordering: auto_follow::spawn is called before self_purge::spawn in ContextManager::started. This is load-bearing — if reversed, OpEvent::ContextRegistered events emitted by the sweep may be dropped by the broadcast channel before the auto-follow receiver exists.

- -

Namespace root creation — NamespaceCreated genesis op

-

When a namespace root group is created (parent_group_id == None), the handler now signs, applies, and publishes a RootOp::NamespaceCreated { founder: admin_identity } genesis op via sign_apply_and_publish_namespace_op. A NoPeersSubscribed error from publish is treated as success — the apply succeeded locally and publish is best-effort. Any other error from the genesis apply is fatal: all locally-written root rows (meta, founder Admin member row, default caps, group key, signing key, optional name metadata) are rolled back via individual idempotent deletes (MetaRepository::delete, MembershipRepository::remove_member, CapabilitiesRepository::delete_default, GroupKeyring::delete_key_by_id, SigningKeysRepository::delete_key, MetadataRepository::delete_group) before returning Err. The namespace identity row is deliberately not rolled back so a retry gets the same identity and founder. The DAG head requires no rollback because apply_signed_op is head-atomic — the head advances only after a successful apply.

- -

RootOp::GroupCreated wire change

-

⚠ WIRE BREAK: RootOp::GroupCreated now carries a boolean restricted field. Existing nodes must reset their op logs. restricted: true is the default and preserves all legacy behavior. A group created with restricted: false is "born-Open": group_created::apply writes the subgroup's visibility key via CapabilitiesRepository::set_subgroup_visibility during apply (before OpEvent::SubgroupCreated is queued, and only on first apply), so tee_subgroup_admit sees the group as already Open and skips writing a transient direct ReadOnlyTee row.

- -

Two log signals are emitted by the list_group_members projection shadow and are relevant for monitoring:

-
    -
  • unified_projection_divergence / plane=membership-role — a live member's role differs from the projected role. These are pre-flip validation failures: once the divergence rate reaches zero, the handler can be flipped to authoritative projection. Persistent warnings indicate a bug in role derivation or ACL fold logic.
    • -
  • When member_roles_for abstains (group unfolded or absent from view.groups) the role shadow is skipped entirely for that group, consistent with the identity shadow's materialized-fallback parity behavior.
    • -
- -

UnifiedApplier

-

UnifiedApplier is the C2.0 projection-backing applier. Use UnifiedApplier::new() for standalone replay or tests (creates a private ScopeProjections), and UnifiedApplier::with_projection(arc) to attach to the node's live shared ScopeProjections during dual-write (C2.1 seam). The std::sync::RwLock is intentional — the lock is never held across an .await. Poison is recovered, not panicked, matching the node's existing projection-reader stance. The projection() accessor returns the underlying Arc for inspection by callers (e.g., tests asserting convergence via scope_root).

- -

ops_for_namespace / collect_namespace_ops / repersist_namespace_ops

-

ScopeProjections::ops_for_namespace is the single stable dispatch point for loading ops at every projection-rebuild site. All internal callers — backfill_namespace, member_now_ephemeral/ephemeral_fold, and the scope_root reconstruction — call ops_for_namespace rather than collect_namespace_ops directly. Today ops_for_namespace simply delegates to collect_namespace_ops (the governance-DAG walk). The flip to load_scope_ops (unified op-store) is deferred; see the C2.2b deferral note in scope_projection.rs above for the root cause.

-

ScopeProjections::repersist_namespace_ops is the C2.2c bridge that keeps the op-store consistent after a group key is delivered. After key delivery, collect_namespace_ops can now decrypt ops that were previously frozen as Noop in the op-store. Calling repersist_namespace_ops re-walks the namespace via collect_namespace_ops and calls persist_op for each op, atomically overwriting stale Noop entries with their fully-decrypted forms. The operation is idempotent (content-addressed keys), best-effort (errors logged, not propagated), and bounded by MAX_BACKFILL_OPS. It should be invoked after every successful group key delivery to ensure the op-store remains reconstruction-accurate.

- -

ScopeProjections::persist_namespace_head_ops is the local-author companion to repersist_namespace_ops. It is called immediately after a locally-authored governance op is published to mirror that op into the unified op-store without racing against concurrent peer advances. Because walking only the current DAG heads would be racy (a peer op can advance the head during the author's publish round-trip, orphaning the just-authored op), the method delegates to repersist_namespace_ops for a full gov-DAG fold bounded by MAX_BACKFILL_OPS. The call is idempotent and best-effort — failures are logged and never block the authoring response. Callers: create_group (after GroupCreated is successfully reported), delete_group (after GroupDeleted is applied), join_context (after MemberJoinedOpen is published), and join_subgroup_inheritance (after MemberJoinedOpen is published). The new test persist_namespace_head_ops_lands_locally_authored_op_in_the_op_store simulates the local-author gap (an encrypted MemberAdded present in the gov-DAG but absent from the op-store), calls persist_namespace_head_ops, and asserts the op appears in the op-store and that folding the op-store projection yields member_at_cut returning Some(true) — proving the encrypted payload was decoded correctly, not just that the op ID was stored.

- -

unified_op_store (persist_op / load_scope_ops)

-

unified_op_store::persist_op and unified_op_store::load_scope_ops are the durable persistence surface for unified ops. persist_op is idempotent: writing the same op twice is safe because the key is content-addressed by op id — this is the property that makes repersist_namespace_ops safe to call multiple times. load_scope_ops returns rows in key order, not causal order — callers must replay the returned Vec<Op> through DagStore<Op> + UnifiedApplier to recover causal ordering and a valid projection. DAG heads are deliberately not persisted by this module. No production apply or sync path uses these functions yet; they are available for future backfill and crash-recovery paths.

- -

scope_root_for / group_scope_root_ephemeral / scope_root_from_op_store

-

ScopeProjections::scope_root_for and its companion group_scope_root_ephemeral are public API on ScopeProjections. Important caller contract: a None return must always be treated as "skip comparison", never as divergence. The ancestry-completeness guard in group_scope_root_ephemeral means mid-backfill nodes will return None rather than a spurious mismatch. The unit test scope_root_for_moves_on_hash_neutral_membership_change_and_is_none_when_unfolded holds entities_root fixed, ingests an AdminChanged op then a causally-chained MemberAdded op, and asserts the scope root differs before and after; it also asserts None is returned for an unfolded scope.

- -

ScopeProjections::scope_root_from_op_store reconstructs a governance scope root purely from persisted ops (via unified_op_store::load_scope_ops) without relying on any maintained in-memory DAG. Returns Ok(None) for scopes with no persisted ops. Two assertions have been added to the existing persisted_op_log_reloads_and_replays_to_the_same_projection test: (1) scope_root_from_op_store equals the expected in-memory fold result after ops are persisted; (2) scope_root_from_op_store returns None for a scope that has no persisted ops.

- -

Note: ScopeProjections::shadow_compare_op_store has been removed from the hot path. The C2.2c regression gate is the enabled test op_store_reconstruction_recovers_late_decrypted_membership_after_key_delivery in tests/op_store_reconstruction.rs (formerly op_store_reconstruction_matches_governance_dag_for_late_decrypted_membership, previously #[ignore]d). The test is structured as an explicit before/after: it first asserts the op-store holds a frozen Noop (Some(false)) before re-persist, then calls repersist_namespace_ops, then asserts the op-store reconstruction matches the governance-DAG fold (Some(true)). The #[ignore] attribute and its explanatory message have been removed; the test now runs in CI as a live regression gate for the C2.2c fix.

- -

A separate completeness-gate test completeness_gate_flags_governance_ops_missing_from_the_op_store in tests/op_store_reconstruction.rs verifies check_op_store_completeness end-to-end: it builds three governance ops, persists only two to the op-store, calls check_op_store_completeness with all three as the dag-fold, and asserts the return is Some(vec![op3.id]) (one gap). It then persists the third op and asserts the return becomes Some(vec![]) (verified complete), confirming that the None vs empty-vec distinction works correctly and that the gate accurately tracks store coverage.

- -

AtCutAuthorizer trait

-

Implementors of AtCutAuthorizer must provide three required methods. The empty-cut contract applies to all of them: every method must return None when parents is empty — violating this would falsely reject genesis ops by evaluating against an empty fold.

-
    -
  • is_admin_at_cut — returns whether a member holds admin authority at the given DAG cut.
    • -
  • is_admin_or_capability_at_cut — returns whether a member holds admin authority or a specific capability flag at the given cut. LiveFallbackAuthorizer returns None. EphemeralProjectionAuthorizer delegates to the cached fold via the private folded helper.
    • -
  • is_last_admin_at_cut — newly required; returns whether the given member is the last admin at the given DAG cut, i.e., whether removing or demoting them would orphan the group's admin set. A member counts as admin if they hold a direct Admin-role row or are the genesis admin; another admin requires a distinct Admin-role row. LiveFallbackAuthorizer returns None. EphemeralProjectionAuthorizer delegates to ScopeProjections::is_last_admin_at_cut, guarding against empty parents before folding. Returns None when ancestry is incomplete, deferring to live.
    • -
  • membership_path_at_cut — newly required; returns the membership path for a given member at the DAG cut, as an AtCutMembershipPath. An empty parents slice returns None (defer to live), consistent with the empty-cut contract. LiveFallbackAuthorizer returns None unconditionally. EphemeralProjectionAuthorizer delegates to ScopeProjections::membership_path_at_cut, which calls the projection's member_path_at_cut walk using the auth-cut context and returns Option<calimero_authz::MemberPathAtCut>; the result is then mapped to AtCutMembershipPath, projecting away the role/anchor detail.
    • -
  • ScopeProjections::role_at_cut_for_group — node-side accessor that derives the effective role of a member in a group at the governance cut. Uses member_path_at_cut to handle three cases: direct membership (returns the folded role row), inherited-via-admin (always returns Admin), and inherited-via-anchor (returns the anchor's role row). Gates on cut_ancestry_complete via auth_cut_context. Returns None when ancestry is incomplete, the member is absent from the projection, the anchor row is missing (to prevent spurious divergence reports), or the namespace is unresolvable — callers must fall back to live role reads in those cases. (Note: acl_view_at and MembershipStatus have been removed; this projection-based accessor is the sole membership-authorization path.)
    • -
-

Note: the group-auth shadow that uses EphemeralProjectionAuthorizer only fires on the namespace-envelope group-op path. The standalone GroupGovernanceApplier::apply retains plain live-fallback gates (see governance_dag.rs above).

- -

Dependencies

-

How the three context crates relate to other Calimero crates. Arrows show compile-time dependencies.

- - - - - - - - - - calimero-context - ContextManager actor + handlers - - - - calimero-context-primitives - ContextClient + messages + wire types - - - - calimero-context-config - client config + types - - - - - - - - - store - - - - dag - - - - node-primitives - - - - network-primitives - - - - primitives - - - - authz - authorization - - - governance-types - shared gov types - - - op - op types - - - op-adapter - op adaptation - - - projection - scope projection - - - - - - - - - - - - - - - - - - - - - actix - - - tokio - - - borsh - - - ed25519-dalek - - - serde - - -
-
Context crates
-
Storage
-
Node
-
Network
-
Primitives
-
New dependencies (authz, governance-types, op, op-adapter, projection)
-
External crates
-
-
- -
-
- - - diff --git a/architecture/crates/dag.html b/architecture/crates/dag.html deleted file mode 100644 index 6e5528e1b9..0000000000 --- a/architecture/crates/dag.html +++ /dev/null @@ -1,173 +0,0 @@ - - - - -Causal DAG — Calimero Core Architecture - - - - -
-
-
Home/System Overview/Causal DAG
- -

Causal DAG

-

calimero-dag

- -
-
31
tests
-
~200 B
+ payload / delta
-
O(1)
add_delta when applied
-
Pure
no network / storage in crate
-
- -
-

Purpose

-

Pure directed acyclic graph for causal delta tracking with automatic dependency resolution. The crate manages causal relationships between state changes so they are applied in the right order even when messages arrive out of order from the network.

-

crates/dag · See also Storage layer and Node for how deltas persist and execute.

-
- -
-

Core types

-

Three central abstractions: the delta record, the in-memory store, and the async applier injected by callers.

- -
-pub struct CausalDelta<T> {
-    pub id: [u8; 32],
-    pub parents: Vec<[u8; 32]>,
-    pub payload: T,
-    pub timestamp: u64,
-} -
- -
-pub struct DagStore<T> {
-    deltas: HashMap<[u8; 32], CausalDelta<T>>,
-    applied: HashSet<[u8; 32]>,
-    pending: HashMap<[u8; 32], PendingDelta<T>>,
-    heads: HashSet<[u8; 32]>,
-} -
- -
-#[async_trait]
-pub trait DeltaApplier<T>: Send + Sync {
-    async fn apply(&self, delta: &CausalDelta<T>) -> Result<(), ApplyError>;
-} -
-
- -
-

Key features

-
-
-

Out-of-order delivery

-

Deltas whose parents are not yet applied go into pending. When a parent arrives and is applied, dependents are reconsidered and may apply in a cascade.

-
-
-

Concurrent updates

-

Multiple heads mean a fork. A merge delta whose parents list includes those heads resolves the fork back to a single tip.

-
-
-

Automatic cascade

-

Applying one delta can unlock several pending children; the store chains applications until no more ready work remains.

-
-
-
- -
-

API

-

Primary query and mutation surface on DagStore<T>:

-
    -
  • add_delta — ingest a delta; returns whether it applied immediately (vs buffered)
    • -
  • get_heads — current DAG tips
    • -
  • get_missing_parents — parent IDs still needed from the network
    • -
  • get_delta — lookup by id
    • -
  • cleanup_stale — evict old pending entries by age
    • -
  • pending_stats — counts and missing-parent totals for observability
    • -
-
- -
-

Integration

-

The node wraps the DAG with WASM execution: DeltaStore holds an Arc<RwLock<DagStore<Vec<Action>>>> and an applier that runs storage actions in the context runtime.

-
-// Conceptual: node wires DAG to WASM
-pub struct DeltaStore {
-    dag: Arc<RwLock<DagStore<Vec<Action>>>>,
-    applier: Arc<ContextStorageApplier>,
-} -
-

ContextStorageApplier implements DeltaApplier<Vec<Action>>: serialize actions, call the context client’s sync entry (e.g. __calimero_sync_next), and complete when WASM finishes.

-
- -
-

Design

-
-
-
    -
  • Pure logic — no network, storage, or WASM inside calimero-dag
    • -
  • Generic payload — any T for delta contents
    • -
  • Dependency injection — behavior via DeltaApplier for tests and integration
    • -
  • Memory-only — DAG state lives in RAM; persistence is the wrapper’s job
    • -
-
-
-

Performance

-
    -
  • Memory: ~200 bytes plus payload size per stored delta
    • -
  • add_delta: O(1) when applied immediately; O(P) when pending (P = work to find unlocked children)
    • -
  • get_heads: O(H) for H heads (typically small)
    • -
  • cleanup_stale: O(P) over pending
    • -
  • Cascade: O(N) where N is the number of pending deltas that become ready in one wave
    • -
-
-
-
- -
-

Test coverage

-

31 tests across basic flows, out-of-order ingestion, concurrent branches and merges, error handling, pending management, query helpers, and stress cases (on the order of 500–1000 deltas, random order, wide merges).

- -
-Test categories (expand) -
-
Basic
Creation, linear sequences, duplicate handling
-
Out-of-order
Buffering, cascade, deep pending chains
-
Concurrent
Forks, two- and three-way merges, complex topology
-
Errors
Apply failures and recovery paths
-
Pending
Stats, cleanup, missing-parent tracking
-
Query
has_delta, get_delta, get_deltas_since
-
Stress
Large chains, many branches, shuffled delivery (500–1000 deltas)
-

Run: cargo test -p calimero-dag

-
-
-
- -
-

Comparison vs vector clocks

-

Calimero’s DAG model trades implicit vector-clock causality for explicit structure and mergeable payloads.

- - - - - - - - - - -
FeatureDAG (Calimero)Vector clocks
CausalityExplicit parent referencesImplicit per-replica counters
StructureConcrete deltasAbstract clock tuples
MergingCRDT-style payload merging at merge deltasTypically external merge logic
Partial stateSupported (sync subsets of history)Often needs fuller history for correctness
-
- -
-
- - - diff --git a/architecture/crates/network.html b/architecture/crates/network.html deleted file mode 100644 index 825986639c..0000000000 --- a/architecture/crates/network.html +++ /dev/null @@ -1,500 +0,0 @@ - - - - -Network & P2P — Calimero Core Architecture - - - -
-
-
Home/System Overview/Network & P2P
- -

Network & P2P

-

calimero-network + calimero-network-primitives

- - -
-

Purpose

-

NetworkManager is an Actix actor that owns and drives the libp2p Swarm with 10+ composed behaviours. It handles gossipsub pub/sub, Kademlia DHT, mDNS discovery, relay circuits, rendezvous registration, hole punching, and multiplexed streams.

-

The network layer only moves opaque bytes — message decoding happens in the node layer. This clean boundary means calimero-network never imports application-level types; it simply ferries Vec<u8> payloads between peers and hands them up through NetworkEvent variants.

- -
-
10+
libp2p behaviours
-
2
stream protocols
-
14
NetworkClient methods
-
-
- - -
-

libp2p Behaviour Stack

-

All behaviours are composed into a single #[derive(NetworkBehaviour)] struct. Each behaviour manages one protocol concern, and the swarm dispatches events through pattern-matched handlers.

- - - - - - - - - - - - Gossipsub - pub/sub mesh relay - topic-based broadcast - - - Kademlia - DHT peer routing - /calimero/kad/1.0.0 - - - mDNS - LAN discovery - optional toggle - - - Relay - circuit relay v2 - NAT traversal fallback - - - - Rendezvous - namespace registration - peer discovery bootstrap - - - DCUtR - direct connection upgrade - hole punching - - - AutoNAT v2 - reachability detection - public / private probe - - - Identify - version & protocol push - agent string exchange - - - - Ping - liveness heartbeat - RTT measurement - - - Streams (multiplexed) - /calimero/stream/0.0.2 · /calimero/blob/0.0.2 - general sync, delta exchange, blob transfer - - - Request-Response - specialized node invite - verification & response - - - - #[derive(NetworkBehaviour)] struct Behaviour { … } - all behaviours composed into a single libp2p Swarm driven by NetworkManager actor - - - - - - - - - - Core protocol - - NAT / relay - - Connectivity - - Diagnostics - - Discovery - - Specialized - -
- - -
-

Topic Management

-

Gossipsub uses IdentTopic for two distinct topic namespaces. The network crate only sees opaque topic strings — the semantic meaning is defined at the node layer.

- -

Context Topics

-
-let topic = IdentTopic::new(context_id); -
// Carries: StateDelta, HashHeartbeat -
// One topic per context — peers subscribe on context join -
- -

Group Topics

-
-let topic = IdentTopic::new(format!("group/{}", hex::encode(group_id))); -
// Carries: SignedGroupOpV1, GroupGovernanceDelta, -
// GroupStateHeartbeat, GroupMutationNotification -
// One topic per group — cross-context governance traffic -
- -
-

Important boundary

-

BroadcastMessage is not defined in the network crate — it lives in calimero-node-primitives. The network layer only sees raw bytes; deserialization into BroadcastMessage variants happens in NodeManager.

-
-
- - -
-

Gossipsub Peer Scoring

-

Calimero uses membership-biased gossipsub scoring (#2513) to prefer mesh peers that are members of the same contexts. Scores influence which peers are selected into the gossipsub mesh and how long non-mesh peers remain eligible.

- -

How it works

-
-
-

Membership derivation (PR 1/2)

-

The node layer tracks which peers share context membership with the local node. When a peer's membership status changes (join, leave, eviction), the node emits a score update to NetworkManager with a pre-computed membership-derived score for that peer.

-
    -
  • Peers sharing ≥1 context membership receive a positive score boost.
    • -
  • Peers with no shared membership receive a neutral (zero) boost.
    • -
  • Evicted peers receive a negative penalty that decays over time.
    • -
-
-
-

Score push (PR 2/2)

-

When context membership changes (subscribe, unsubscribe, member add/remove), the node recomputes the score for all affected peers and calls NetworkClient::set_peer_score. The NetworkManager applies the score to libp2p's gossipsub scoring substrate, which then uses it as an application-layer score component during mesh management.

-
-// Conceptual flow:
-NodeManager detects membership change
- → computes membership-derived score per peer
- → NetworkClient::set_peer_score(peer_id, score)
- → gossipsub uses score in mesh peer selection -
-
-
- -
-

Design intent

-

Membership-biased scoring ensures that context-relevant peers are mesh-preferred, reducing relay hops for context traffic. It does not affect security — gossipsub message signing and topic access control are independent of scores. Scores are advisory for mesh selection; non-member peers still receive gossiped messages if they are subscribers.

-
-
- - -
-

Stream Protocols

-

Two multiplexed stream protocols ride on the same libp2p::Stream infrastructure. Both use length-delimited framing for reliable message boundaries.

- -
-
-

/calimero/stream/0.0.2

-

General-purpose sync stream for delta exchange, key sharing, and state negotiation.

-
    -
  • Length-delimited frames
    • -
  • 8 MiB max frame size
    • -
  • StreamMessage → InitPayload dispatch
    • -
  • Used by sync engine for catch-up & real-time replication
    • -
-
-
-

/calimero/blob/0.0.2

-

Blob transfer protocol for large binary payloads (WASM, application data).

-
    -
  • JSON BlobRequest / BlobResponse handshake
    • -
  • Borsh-encoded BlobChunk stream
    • -
  • Chunked transfer for large payloads
    • -
  • Announce / query / request lifecycle
    • -
-
-
- - - - - Peer A - open_stream() - - - - Yamux Multiplexed Connection - - /calimero/stream/0.0.2 - - /calimero/blob/0.0.2 - - - - Peer B - accept_stream() - - - - - - - - - - Frame: - - len (4B) - - payload (≤ 8 MiB) - - - -
- - -
-

NetworkClient

-

NetworkClient is the async handle held by other actors to communicate with NetworkManager. Each method sends a typed message and awaits the response.

- -
-
-

Connection

-
-fn dial(addr) → Result
-fn listen_on(addr) → Result
-fn bootstrap() → Result -
-
-
-

Pub/Sub

-
-fn subscribe(topic) → Result
-fn unsubscribe(topic) → Result
-fn publish(topic, data) → Result -
-
-
-

Streams

-
-fn open_stream(peer) → Stream -
-
-
- -
-
-

Mesh Info

-
-fn peer_count() → usize
-fn mesh_peer_count(topic) → usize
-fn mesh_peers(topic) → Vec -
-
-
-

Blob Transfer

-
-fn announce_blob(hash) → Result
-fn query_blob(hash) → Providers
-fn request_blob(peer, hash) → Blob -
-
-
-

Specialized Node

-
-fn send_specialized_node_
-  verification_request(..) → Result
-fn send_specialized_node_
-  invitation_response(..) → Result -
-
-
-
- - -
-

Discovery

-

The discovery subsystem combines multiple libp2p mechanisms into a state machine that progressively establishes connectivity with the mesh.

- - - - - Boot - dial bootstrap nodes - - - - Register - rendezvous namespace - - - - Discover - find context peers - - - - Connect - dial discovered peers - - - - - - - - - - - - Relay Reservation - request circuit relay slot - from configured relay nodes - - - AutoNAT Probe - test reachability from - external peers - - - DCUtR Upgrade - attempt hole punch - upgrade relayed → direct - - - Public IP - optional advertise - external address - - - - - - - - - - - - - - - mDNS (LAN) - concurrent local discovery (if enabled) - - - - - - Kademlia DHT - routing table & provider records - - - -
- - -
-

Transport

-

The transport layer supports multiple stacks for different network conditions. All connections are authenticated and encrypted.

- -
-
-

TCP Stack

-

Primary transport for reliable connections.

-
-// Transport chain:
-TCP → TLS (libp2p-tls) → Noise (XX handshake)
-   → Yamux (stream multiplexing) -
-
-
-

QUIC Stack

-

UDP-based transport with built-in encryption and muxing.

-
-// Transport chain:
-QUIC (UDP + TLS 1.3 built-in)
-   → native stream multiplexing
-// No separate Noise/Yamux needed -
-
-
- - - - Yamux Frame Codec: - - - version (1B) - - - type (1B) - - - flags (2B) - - - stream ID (4B) - - - length (4B) - - - payload (variable) - - - Noise XX Handshake: - → e - ← e, ee, s, es - → s, se - ✓ mutual authentication complete - -
- - -
-

Dependencies

-

Key crate dependencies and their roles in the network layer.

- -
-
-

calimero-network

-
libp2p
Core networking stack — swarm, transports, all behaviour crates
-
tokio
Async runtime for swarm event loop and stream I/O
-
futures
Stream/Sink adapters for length-delimited framing
-
serde / serde_json
Blob protocol JSON handshake serialization
-
borsh
Blob chunk binary serialization
-
tracing
Structured logging for network events
-
calimero-network-primitives
Shared types: NetworkConfig, stream protocol IDs
-
-
-

calimero-network-primitives

-
libp2p-identity
PeerId, Keypair re-exports
-
multiaddr
Multi-address types for listen/dial configuration
-
serde
Config serialization (NetworkConfig, BootstrapConfig, etc.)
-
strum
Enum display/from_str for protocol IDs
-
-
-
- -
-
- - - diff --git a/architecture/crates/node.html b/architecture/crates/node.html deleted file mode 100644 index 6005790b70..0000000000 --- a/architecture/crates/node.html +++ /dev/null @@ -1,561 +0,0 @@ - - - - -Node — Calimero Core Architecture - - - -
-
-
Home/System Overview/Node
- -

Node Crate

-

calimero-node + calimero-node-primitives

- - -
-

Purpose

-

NodeManager is an Actix actor that orchestrates the entire node. It receives network events via a dedicated channel bridge, manages blob caching, periodic heartbeats, delta handling, and routes streams to the sync manager.

-

It does not run libp2p directly — NetworkManager does that. NodeManager communicates with the network through NodeClient, which wraps a LazyRecipient<NodeMessage> and a NetworkClient handle. The crate also contains the SyncManager (a long-lived async task, not an Actix actor) and a GarbageCollector actor for storage tombstone cleanup.

- -
-
2
crates
-
30+
source files
-
3
actors / tasks
-
4
sync protocols
-
-
- - -
-

Module Structure

-

File layout across the two crates. Each handler lives in its own focused file following the Single Responsibility Principle.

- -

calimero-node

-
src/lib.rs
NodeManager actor definition, Actix lifecycle hooks
-
src/manager.rs
NodeManager struct with SRP decomposition: NodeClients, NodeManagers, NodeState
-
manager/startup.rs
Startup hooks: setup_startup_subscriptions(), setup_maintenance_intervals(), setup_hash_heartbeat_interval()
-
src/state.rs
NodeState: blob cache (multi-phase LRU eviction), delta stores (DashMap), sync sessions, pending invites
-
src/run.rs
start() entry point, NodeConfig, wires Store / BlobManager / actors / server / GC
-
src/handlers.rs
Handler<NodeMessage> dispatch — routes GetBlobBytes, specialized node invites
-
handlers/network_event/mod.rs
Handler<NetworkEvent> — gossip, stream, subscription, broadcast message routing
-
handlers/network_event/namespace.rs
Namespace event handlers: handle_namespace_governance_delta(), handle_namespace_state_heartbeat(), backfill protocol
-
handlers/network_event/specialized.rs
Specialized node: discovery, TEE attestation, join confirmation
-
handlers/state_delta/mod.rs
handle_state_delta — DeltaStore buffering, context routing, sync session awareness, bounded KeyDelivery wait; gossip write-auth now delegated entirely to authorize_delta_at_edge_projected (projection-only, no live resolver called); refresh_projection_for_cut (DAG-walk/backfill for projection reads; calls ops_for_namespace — the stable indirection layer that currently delegates to collect_namespace_ops (governance-DAG walk) and will be flipped to load_scope_ops once late-decrypted membership is resolved; after ops_for_namespace returns the gov-DAG fold, check_op_store_completeness is called with those ops as a purely diagnostic gate — one op-store read on cold/refresh cuts (warm cuts skip via namespace_to_refresh returning None) — whose result is ignored by the caller and never affects whether or how apply_backfill proceeds; the observe-only shadow_compare_op_store cross-check and its companion scope_root_from_governance_dag have been removed — the comparison was flawed because the maintained projection holds live ingest_op-fed ops that may not be in the op-store); resolve_cut_membership (pub(crate) — single shared projection verdict renderer: performs a scoped projection refresh, then reads both member_at_cut and role_at_cut_for_group under a single read guard for epoch consistency, returns a CutMembership value; used by the gossip apply path, parent-fetch in request_missing_deltas, snapshot-replay in replay_buffered_delta, and DAG-catchup in SyncManager); projection_member_at_cut (pub(crate) — delegates to resolve_cut_membership and discards the role, ensuring the two paths cannot drift; used by SyncManager::peer_is_group_member); drain_governance_pending — re-applies, drops, or re-buffers buffered deltas based on three projection outcomes (Some(true)→apply, Some(false)→drop, None→re-buffer); the live authorize_delta_at_edge function is deleted — no acl_view_at calls remain on any delta authorization path
-
src/state_delta_bridge.rs
StateDeltaActor on dedicated Arbiter, bounded mailbox, drop-on-overflow with rebroadcast recovery
-
src/sync_session_bridge.rs
SyncSessionActor on dedicated Arbiter — both initiator and responder sync sessions; bounded mailbox + semaphore (max_concurrent), per-session timeout
-
sync/manager/namespace_sync.rs
Group-key delivery (pull-based): on a join request the responder validates the invitation and serves the ECDH-wrapped key (handle_namespace_join_request → build_group_key_delivery); joiners recover missing keys via recover_missing_group_keys. After a successful direct key delivery and the existing drain/reconcile steps, ScopeProjections::repersist_namespace_ops is called for the affected namespace_id so that any previously frozen Noop ops (encrypted MemberAdded entries that could not be decrypted at ingest time) are re-decrypted and written into the op-store. Because recover_missing_group_keys is the single runtime key-delivery chokepoint, all late-arriving keys trigger this op-store refresh automatically. RootOp::KeyDelivery is published only by admin-initiated adds (add-members / TEE), not on join. After sign_and_publish_without_apply successfully publishes the MemberJoinedAt op during await_namespace_ready (Stage 2 of namespace join), the op is written into the unified op-store via ScopeProjections::repersist_namespace_ops; previously this locally-authored publish wrote to the gov-DAG but skipped the op-store because the receive-handler dual-write does not cover locally-authored publishes. Note: the thin alias persist_namespace_head_ops has been deleted and all post-author mirror calls at create_group, delete_group, join_context, join_subgroup_inheritance, remove_group_members, reparent_group, create_group_in_namespace, and join_namespace have been removed; repersist_namespace_ops is the canonical method and is called only at the key-delivery and join-completion chokepoints described above.
-
sync/manager/mod.rs (peer_is_group_member)
SyncManager::peer_is_group_member — encapsulates inbound-peer membership authorization for all sync call sites. Checks MembershipRepository::is_member (live) first, then resolves current governance heads via ScopeProjections::namespace_current_heads and calls projection_member_at_cut (now pub(crate)) for the projection verdict. The projection's answer is authoritative; if it returns None (cold or partially folded ancestry) the method falls back to the live result via projected.unwrap_or(live) so no peer is spuriously rejected — the existing catch-up retry will re-resolve once governance advances. No divergence warning is emitted when projection and live disagree; the live read is retained solely as the None fallback. SyncManager::verify_inbound_member and the materialization-wait verification now both route through peer_is_group_member instead of calling MembershipRepository::is_member directly.
-
handlers/blob_protocol.rs
Blob transfer handler — serves blob data to requesting peers
-
src/delta_store.rs
Per-context delta buffering for out-of-order delta arrival
-
src/gc.rs
GarbageCollector actor — periodic tombstone cleanup across all contexts
- -

calimero-node / sync

-
sync/mod.rs
SyncManager, SyncConfig, protocol re-exports, metrics collector traits
-
sync/manager/mod.rs
Periodic sync loop, protocol selection, stream handling, peer tracking. pull_namespace_governance(context_id, peer) is now an inherent async method on SyncManager — resolves the namespace for the context via namespace_sync::resolve_namespace_id and calls sync_namespace_from_peer targeting the specific diverging peer, reusing the existing NamespaceBackfillRequest machinery edge-triggered on the verdict; best-effort (no-op if the context has no resolvable namespace). The ProtocolDispatch trait method pull_namespace_governance is now a one-line delegate to SyncManager::pull_namespace_governance. Pre-sync governance divergence check (P6.S3): when select_protocol returns SyncProtocol::None (entity roots agree, no entity walk needed) and the peer advertises a non-None scope_root, the manager now computes a scope_verdict. If the verdict is ScopeVerdict::GovDiverged (local and peer scope roots differ despite identical entity roots), it calls pull_namespace_governance and returns early, bypassing the entity walk entirely. This fills the gap where a governance rotation or ACL change — invisible to the entity Merkle — would otherwise persist uncorrected because the post-sync P6.S2 hook only fires after HC/LevelWise actually runs. The datastore read is gated on peer_scope_root being Some; when a peer sends scope_root: None (cold projection or non-group context) the store is never touched and scope_verdict is never computed. After a successful pre-sync governance pull the manager emits a debug-level gov_divergence_pull_complete marker, pairing with the existing gov_divergence_pull_triggered info log. After a successful HashComparison or LevelWise session, if gov_divergence_detected is true, the ProtocolSelector likewise calls pull_namespace_governance and logs gov_divergence_pull_triggered; previously only a passive warning was emitted with no corrective action. The pull is self-limiting: once governance converges the verdict is no longer GovDiverged and no further pull is issued. Implementors of ProtocolDispatch must implement pull_namespace_governance (delegating to the inherent method).
-
sync/hash_comparison_protocol.rs
Merkle DFS traversal — cheapest sync protocol; run_initiator_impl calls query_peer_current_root (pub(crate) — also reused by the LevelWise initiator; returns (root_hash, Option<scope_root>)); query_peer_dag_state return type extended from Option<(Hash, Vec<[u8; DIGEST_SIZE]>)> to Option<(Hash, Vec<[u8; DIGEST_SIZE]>, Option<Hash>)> — the scope_root field from DagHeadsResponse (previously discarded as scope_root: _) is now captured and threaded back to all callers; on the re-query convergence path, stats.root_hash_verified is now set via helpers::scope_verdict(...).converged() — the inline match (local_scope_root, peer_scope_root) block is replaced by a call to the shared scope_verdict helper; the scope_root_governance_divergence WARN is emitted by pattern-matching ScopeVerdict::GovDiverged(local, peer) directly, eliminating the prior if let (true, Some(...), Some(...)) guard; HashComparisonStats gains a gov_divergence_detected: bool field set to true when the verdict is ScopeVerdict::GovDiverged, exposing the signal to the ProtocolSelector; the WARN text changes from "awaiting governance sync to propagate the rotation" to "pulling governance from the peer to propagate the rotation"; the deterministic sync simulator sends scope_root: None
-
sync/level_sync.rs
BFS level-wise sync for wide trees. The initiator calls query_peer_current_root (pub(crate)) at end-of-session for a fresh (root_hash, Option<scope_root>) rather than comparing against the stale handshake root; convergence is now decided via helpers::scope_verdict(...).converged() — the inline match (local_scope_root, peer_scope_root) block is replaced by the shared helper, mirroring HashComparison. Transport faults fall back to the handshake root; an older peer returning Ok(None) falls back silently. The responder loop has an explicit InitPayload::DagHeadsRequest arm: after all leaf/tombstone merges it re-reads the local entity root, folds the governance projection to obtain scope_root (None for non-group or cold projection), and replies with a DagHeadsResponse carrying both fields. LevelWiseStats gains a gov_divergence_detected: bool field set to true when the verdict is ScopeVerdict::GovDiverged, mirroring HashComparisonStats. When root_hash_verified is false, the scope_root_governance_divergence WARN is emitted by pattern-matching ScopeVerdict::GovDiverged(local, peer); the WARN text changes from "awaiting governance sync to propagate the rotation" to "pulling governance from the peer to propagate the rotation"; all other mismatches emit the general non-convergence warning using the freshly re-queried peer root.
-
sync/snapshot.rs
Full snapshot transfer — heaviest fallback protocol
-
sync/blobs.rs
Blob sharing protocol
-
sync/helpers.rs
Sync helper utilities; includes local_scope_root — looks up the group owning a context, calls ScopeProjections::group_scope_root_ephemeral, returns None for non-group contexts or store faults (TODO(perf, C1+): replace full DAG walk with maintained projection reads). Also defines ScopeVerdict { Converged, GovDiverged([u8;32],[u8;32]), DataDiverged } and the free function scope_verdict(local_scope_root, peer_scope_root, local_entity_root, peer_entity_root) -> ScopeVerdict — the single convergence authority used by both HashComparison and LevelWise protocols, and by the pre-sync SyncProtocol::None divergence check in SyncManager. GovDiverged carries both resolved scope roots directly so callers can pattern-match and log them without re-destructuring Options. When either scope root is None (cold projection or non-group context), scope_verdict falls back to bare entity-root comparison (treating asymmetric None as None/None would cause false GovDiverged alarms on partially-warmed nodes). The pre-sync path gates the store read on peer_scope_root being Some, so the common already-in-sync path for cold/non-group peers incurs no I/O.
-
sync/tracking.rs
Per-peer sync history tracking
-
sync/metrics.rs
SyncMetricsCollector trait, PhaseTimer, NoOpMetrics
-
sync/prometheus_metrics.rs
Production Prometheus metric implementation
- -

calimero-node-primitives

-
primitives/src/client.rs
NodeClient façade — subscribe, broadcast, heartbeat, group ops, sync trigger
-
client/application.rs
Application module root — get/has app, module declarations
-
client/application/install.rs
All installation paths (path, URL, blob, bundle), uninstall, artifact extraction
-
client/application/bundle.rs
Bundle manifest verification, signature validation, path safety checks
-
client/application/query.rs
List applications, packages, versions; update compiled app
-
client/blob.rs
Blob add / get / delete / list / auth operations
-
client/alias.rs
Generic alias create / delete / lookup / resolve / list
-
primitives/src/messages.rs
NodeMessage enum — GetBlobBytes, RegisterPendingSpecializedNodeInvite, etc.
-
primitives/src/sync.rs
BroadcastMessage, sync wire types, protocol traits, state machine; MessagePayload::DagHeadsResponse now carries an additive scope_root: Option<Hash> field — None means the responder could not resolve/fold the scope; initiators must skip the shadow comparison in that case
-
primitives/src/delta_buffer.rs
DeltaBuffer — bounded buffer for deltas received during snapshot sync
-
primitives/src/topic_manager.rs
TopicManager — deduplication-aware gossipsub subscription management with RwLock
-
primitives/src/join_bundle.rs
JoinBundle — data package for namespace join (group key envelope, governance snapshot, context IDs)
-
- - -
-

Key Types

-

Central structs in the node crate. NodeManager is the actor; its fields are split into three injected concerns: clients, managers, and mutable state.

- -

NodeManager

-
-pub struct NodeManager {
-    clients: NodeClients,     // context + node façades
-    managers: NodeManagers,   // blobstore + sync
-    state: NodeState,       // mutable runtime state
-} -
- -
-
-

NodeClients

-
-struct NodeClients {
-    context: ContextClient,
-    node: NodeClient,
-} -
- -

NodeManagers

-
-struct NodeManagers {
-    blobstore: BlobManager,
-    sync: SyncManager,
-} -
-
-
-

NodeState

-
-struct NodeState {
-    blob_cache: Arc<DashMap<BlobId, CachedBlob>>,
-    delta_stores: Arc<DashMap<ContextId, DeltaStore>>,
-    pending_specialized_node_invites: PendingSpecializedNodeInvites,
-    accept_mock_tee: bool,
-    node_mode: NodeMode,
-    sync_sessions: Arc<DashMap<ContextId, SyncSession>>,
-} -
-
-
- -

NodeConfig

-
-pub struct NodeConfig {
-    home: Utf8PathBuf,
-    identity: Keypair,
-    // namespace identity is auto-generated per root group in the datastore
-    network: NetworkConfig,
-    sync: SyncConfig,
-    datastore: StoreConfig,
-    blobstore: BlobStoreConfig,
-    context: ContextConfig,
-    server: ServerConfig,
-    gc_interval_secs: Option<u64>,
-    mode: NodeMode,
-    specialized_node: SpecializedNodeConfig,
-    mock_tee: bool,          // dev/test only — refused if TeeConfig::has_real_attestation
-} -
- -
- ⚠ --mock-tee (dev/test only) -

The --mock-tee flag (or MEROD_MOCK_TEE env var) on merod run makes fleet-join and attest handlers produce and accept mock TDX attestation quotes instead of requiring real hardware. When active, a loud tracing::warn is emitted at startup; a second warn fires if a Phala KMS provider is configured but real attestation is disabled (likely misconfiguration). The flag is refused (bail!) when TeeConfig::has_real_attestation() returns true — i.e. when real KMS attestation is configured. NodeConfig::mock_tee threads the value into AdminState (via AdminState::new); it is never persisted to the config file. Must not be used in production.

-
-
-
- - -
-

NodeClient primitives

-

Thin async façade in calimero-node-primitives. Used by Server and ContextManager to interact with node-level concerns. Wraps a Store, BlobManager, NetworkClient, and a LazyRecipient<NodeMessage>.

- -
-pub struct NodeClient {
-    datastore: Store,
-    blobstore: BlobManager,
-    network_client: NetworkClient,
-    node_manager: LazyRecipient<NodeMessage>,
-    event_sender: broadcast::Sender<NodeEvent>,
-    ctx_sync_tx: mpsc::Sender<(Option<ContextId>, Option<PeerId>)>,
-    specialized_node_invite_topic: String,
-} -
- -

Context & Group Subscriptions

-
-
-
fn
-

subscribe(context_id)

-

Subscribe to a context's gossipsub topic via NetworkClient

-
-
fn
-

unsubscribe(context_id)

-

Unsubscribe from a context topic

-
-
fn
-

subscribe_group(group_id)

-

Subscribe to group/{hex} topic

-
-
fn
-

unsubscribe_group(group_id)

-

Unsubscribe from group topic

-
-
-
-
fn
-

broadcast(context, sender, artifact, …)

-

Publish a StateDelta on the context's gossipsub topic

-
-
fn
-

publish_signed_group_op(group_id, op)

-

Publish a SignedGroupOpV1 on the group topic

-
-
fn
-

publish_group_heartbeat(group_id, …)

-

Broadcast GroupStateHeartbeat for catch-up detection

-
-
fn
-

get_peers_count(context)

-

Global peer count or per-topic mesh peers

-
-
-
- -

Application & Blob Management

-

NodeClient exposes application lifecycle through four focused sub-modules (application/):

-
-
-application.rs (module root) → install.rs (all install paths + uninstall) → bundle.rs (manifest verification + path safety) → query.rs (list/search) -
-
-

Bundle signature verification: verify_and_extract_manifest (strict, production) vs extract_manifest_allow_unsigned (dev installs — verifies if present, allows unsigned).

-
-
-

Applications

-
    -
  • install_application()
    • -
  • install_from_path()
    • -
  • install_from_url()
    • -
  • install_from_bundle_blob()
    • -
  • uninstall_application()
    • -
  • list_applications()
    • -
  • list_packages()
    • -
-
-
-

Blobs

-
    -
  • add_blob()
    • -
  • get_blob() / get_blob_bytes()
    • -
  • delete_blob()
    • -
  • list_blobs()
    • -
  • find_blob_providers()
    • -
  • announce_blob_to_network()
    • -
  • create_blob_auth()
    • -
-
-
-

Aliases

-
    -
  • create_alias()
    • -
  • delete_alias()
    • -
  • lookup_alias()
    • -
  • resolve_alias()
    • -
  • list_aliases()
    • -
-
-
-
- - -
-

Startup Flow

-

The start() function in run.rs bootstraps the entire node. Each component is initialized in dependency order, with LazyRecipients breaking circular initialization.

- - - - - - - - - - merod run - CLI entry point - - - - - - start() - run.rs entry - - - - - - Store open - RocksDB + optional encryption - - - - - - BlobManager - filesystem backend - - - - - - - - NetworkManager - actor · libp2p swarm - - - - - - NetworkClient + EventChannel - mpsc(1000) · 80% warning - - - - - - NodeClient - store + blob + network + events - - - - - - ContextManager - actor · context_recipient - - - - - - NodeManager - actor · node_recipient - - - - - - NetworkEventBridge - tokio::spawn · channel → actor - - - - - - - SyncManager.start() - async task · periodic loop - - - - - - - Server - tokio::spawn · Axum - - - - GC Actor - 12h default interval - - - 1 - 2 - 3 - 4 - 5 - 6 - - - - tokio::select! { sync, server, bridge } - - - - - - - Node - - Network - - Context - - Storage - - Sync - - Server - - GC - -
- - -
-

Event Handling

-

⚠ Migration note: Convergence decisions in both HashComparison and LevelWise protocols are now delegated to helpers::scope_verdict(local_scope_root, peer_scope_root, local_entity_root, peer_entity_root) -> ScopeVerdict. Callers set stats.root_hash_verified via verdict.converged() and emit the scope_root_governance_divergence WARN by pattern-matching ScopeVerdict::GovDiverged(local, peer) — the prior inline match (local_scope_root, peer_scope_root) blocks and the if let (true, Some(...), Some(...)) guard are removed from both protocols. HashComparisonStats and LevelWiseStats each gain a gov_divergence_detected: bool field set to true on GovDiverged; the ProtocolSelector reads this flag after a session completes and, if set, calls ProtocolDispatch::pull_namespace_governance(context_id, peer) — an active governance pull replacing the previous passive warning. Additionally, governance divergence is now detected and repaired on the pre-sync SyncProtocol::None path (entity roots agree, no entity walk needed): when a peer advertises a non-None scope_root and scope_verdict returns GovDiverged, SyncManager::pull_namespace_governance is called immediately, bypassing the entity walk. query_peer_dag_state return type is extended to Option<(Hash, Vec<[u8; DIGEST_SIZE]>, Option<Hash>)> so the peer's scope_root is available to this pre-sync check. The datastore read for local_scope_root is skipped entirely when peer_scope_root is None, preserving zero I/O overhead on cold/non-group peers. After a successful pre-sync pull, a debug-level gov_divergence_pull_complete marker is emitted; operators and e2e suites should monitor the gov_divergence_pull_triggered / gov_divergence_pull_complete pair on both pre-sync and post-sync paths to verify governance convergence. pull_namespace_governance is now an inherent method on SyncManager; the ProtocolDispatch trait impl delegates to it in one line. The gov_divergence_pull_triggered info marker is logged when the pull is issued; operators can monitor it to detect per-tick storms. Implementors of ProtocolDispatch must add a pull_namespace_governance implementation. ScopeProjections::persist_namespace_head_ops is removed; all post-author mirror calls across governance op sites are deleted. MessagePayload::DagHeadsResponse carries a scope_root: Option<Hash> field. None means the responder could not fold the scope. The HC initiator's convergence verdict (stats.root_hash_verified) now requires scope_root agreement when both peers resolve one; if either peer returns None it falls back to bare entity-root comparison. The previous C0 observe-only scope_root_shadow_divergence WARN (which logged divergence without acting on it) is removed. A new scope_root_governance_divergence WARN is emitted when both peers resolve a scope_root, bare entity roots agree, but scope roots differ — indicating a pure ACL/governance-plane divergence where the entity tree-walk has nothing to reconcile; operators and log aggregators should monitor this marker as it signals a governance divergence awaiting sync propagation. The governance_drain_outcome metric no longer emits the "lookup_error", "removed", or "never_member" labels; all non-member drops are now reported under the single "not_member" label. Dashboards or alerts filtering on "removed" or "never_member" must be updated to "not_member". The unified_projection_divergence warns previously fired on the grant arm of drain_governance_pending, on the gossip Authorized role-shadow check (plane=data-write-role), and on divergences in peer_is_group_member have all been removed — dashboards or alerts keyed to those events will stop firing. The data_write_decision_shadow debug logs have likewise been removed. Gossip deltas whose governance ancestry is not yet fully folded are now buffered rather than dropped; operators should not expect silent delta loss on incomplete-fold membership cases. Code matching on DeltaAuthOutcome::MembershipReject must no longer expect a group field — that field has been removed from the variant. The live authorize_delta_at_edge function (backed by acl_view_at) is deleted entirely — all delta authorization paths (gossip apply, parent-fetch, snapshot-replay, DAG-catchup) now go through authorize_delta_at_edge_projected with resolve_cut_membership as the resolver closure. The CutMembership::Incomplete variant carries no embedded needed set; the Buffer.needed is populated from the governance position's heads. request_missing_deltas and verify_fetched_parent gain a node_state: &NodeState parameter to carry the projection reference.

-

NodeManager implements Handler<NetworkEvent> (via the bridge) and Handler<NodeMessage>. Network events arrive through a dedicated mpsc channel to avoid cross-arbiter message loss. BroadcastMessage::StateDelta dispatch is forwarded to a dedicated StateDeltaActor running on its own Arbiter so delta processing does not compete with sync, heartbeat, blob, or namespace handlers on the NodeManager mailbox. Inbound and outbound HashComparison/LevelWise sync sessions are likewise routed through a dedicated SyncSessionActor on its own Arbiter (responder via StreamOpened, initiator via SyncManager::start), so a slow session can’t starve the swarm-poll task and trigger gossipsub mesh starvation.

- -
- BroadcastMessage Dispatch -
-

When a gossipsub message arrives, NodeManager deserializes it as a BroadcastMessage variant and routes accordingly:

-
-
-
1
-

StateDelta

-

Routed via StateDeltaSender::try_send to StateDeltaActor on a dedicated Arbiter. The actor's Handler<StateDeltaJob> calls handle_state_delta which checks sync session state — if a snapshot sync is active the delta is buffered, otherwise it's authorized and applied. Mailbox bounded at STATE_DELTA_CHANNEL_CAPACITY (2048); on overflow the dispatch site logs a warn! and drops, relying on heartbeat-driven rebroadcast.

-

Projection-only authorization across all delta paths (authorize_delta_at_edge_projected + resolve_cut_membership): The live authorize_delta_at_edge function is deleted. authorize_delta_at_edge_projected is now the sole authorizer for all delta authorization call sites: gossip apply, parent-fetch (request_missing_deltas / verify_fetched_parent), snapshot-replay (replay_buffered_delta), and DAG-catchup head-pull in SyncManager. Each call site passes resolve_cut_membership as the resolver closure. resolve_cut_membership performs a scoped projection refresh under write lock, then reads both member_at_cut and role_at_cut_for_group under a single read guard for epoch consistency. The projection refresh calls ops_for_namespace — the stable indirection layer that currently wraps collect_namespace_ops (governance-DAG walk). The flip to load_scope_ops (unified op-store) is deferred because an encrypted MemberAdded op can be applied to the governance DAG before its group key arrives, freezing as Noop in the op-store; collect_namespace_ops re-decrypts at read time once the key lands, so using the op-store as the projection source would silently drop members and break joiner sync. The closure returns a CutMembership value: Member(role) → Authorized, NotMember → MembershipReject, Incomplete → Buffer (governance ancestry not yet fully folded — delta is held for retry rather than dropped; no embedded needed set on the variant). projection_member_at_cut is refactored to delegate to resolve_cut_membership and discard the role. All previous shadow/co-authorizer logic, divergence warning logs (membership-cut-grant, data-write-role, data-write-decision planes), and the grant-override on MembershipReject are removed. projection_member_at_cut_authoritative has been deleted. DeltaAuthOutcome::MembershipReject no longer carries a group field. The observe-only shadow_compare_op_store method and its companion scope_root_from_governance_dag have been removed from ScopeProjections; the call site in refresh_projection_for_cut that triggered a full governance-DAG fold on every backfill is deleted.

-

Governance-drain projection authority: drain_governance_pending uses member_at_cut (projection) as the sole decision authority — no live acl_view_at DAG walk. The three projection outcomes map directly to drain outcomes: Some(true) → re-apply the buffered delta; Some(false) → drop it (records a governance_drain_outcome metric with the unconditional label "not_member"; the previous live call that distinguished "removed" / "never_member" / "not_member" has been removed entirely); None → re-buffer (ancestry still being folded). The "lookup_error", "removed", and "never_member" label values no longer exist. The re-buffer debug log no longer includes needed_count.

-

Inbound-sync authorization (peer_is_group_member): projection_member_at_cut is pub(crate) and is reused by SyncManager::peer_is_group_member for all inbound-sync peer authorization. That helper calls the live MembershipRepository::is_member first, then resolves governance heads and calls projection_member_at_cut. The projection's answer is authoritative; a None result (cold or partially folded) falls back to the live result via projected.unwrap_or(live) so no peer is spuriously rejected. No divergence warning is emitted. Both the materialization-wait verification and verify_inbound_member route through peer_is_group_member instead of calling MembershipRepository::is_member directly.

-
-
2
-

HashHeartbeat / Scope-root authoritative verdict (scope_verdict) + active governance pull

-

On the re-query convergence path in run_initiator_impl (HashComparison) and at end-of-session in LevelWise, stats.root_hash_verified is now set via helpers::scope_verdict(local_scope_root, peer_scope_root, local_entity_root, peer_entity_root).converged(). The scope_verdict helper returns a ScopeVerdict enum: Converged (roots agree on both planes), GovDiverged([u8;32],[u8;32]) (entity roots agree but scope roots differ — pure ACL/governance-plane divergence), or DataDiverged (entity roots disagree). Both protocols pattern-match ScopeVerdict::GovDiverged(local, peer) to emit the scope_root_governance_divergence WARN (text: "pulling governance from the peer to propagate the rotation"), and both stats structs set gov_divergence_detected = true. After the session returns, the ProtocolSelector checks gov_divergence_detected and, if set, calls ProtocolDispatch::pull_namespace_governance(context_id, peer) (which delegates to the inherent SyncManager::pull_namespace_governance); a gov_divergence_pull_triggered info marker is logged and a debug-level gov_divergence_pull_complete marker is emitted on completion. Previously only a warning was emitted with no corrective action. Pre-sync path (P6.S3): when select_protocol returns SyncProtocol::None and the peer's scope_root (surfaced from the extended query_peer_dag_state return value) is non-None, the manager computes scope_verdict and, on GovDiverged, calls SyncManager::pull_namespace_governance directly without entering the entity walk. When peer_scope_root is None the store read for local_scope_root is skipped entirely. When either scope root is None, scope_verdict falls back to bare entity-root comparison. Both protocols call the shared query_peer_current_root (pub(crate)) helper to obtain a fresh (root_hash, Option<scope_root>). The query_peer_dag_state return type is extended to Option<(Hash, Vec<[u8; DIGEST_SIZE]>, Option<Hash>)> to surface the peer's scope_root. The previous observe-only scope_root_shadow_divergence WARN is removed. Operators and e2e suites should monitor both the scope_root_governance_divergence WARN and the gov_divergence_pull_triggered / gov_divergence_pull_complete pair on all sync paths (pre-sync, HashComparison, and LevelWise).

-
-
2
-

HashHeartbeat

-

Periodic root hash announcement. Compared with local state; mismatches trigger sync via ctx_sync_tx channel.

-
-
3
-

SignedGroupOpV1

-

Forwarded to ContextManager's group operation handler. Signature verified, then applied to the local GroupStore DAG.

-
-
-
-
4
-

GroupStateHeartbeat

-

Group-level hash heartbeat for governance DAG catch-up detection.

-
-
5
-

GroupGovernanceDelta

-

Governance state delta. Routed to ContextManager for DAG ingestion.

-
-
6
-

GroupMutationNotification

-

Notification of group membership changes. Triggers re-evaluation of subscriptions and capabilities.

-
-
-
-
-
- -
- Stream Routing -
-

When a peer opens a stream, stream_opened.rs inspects the protocol prefix to route:

-
-
-

/calimero/stream/…

-

Sync protocol streams — routed via SyncSessionSender::try_send to the dedicated SyncSessionActor Arbiter (issue #2316). The actor calls SyncManager::handle_opened_stream for hash comparison, level-wise sync, snapshot transfer, or key sharing. Bounded mailbox + a semaphore sized to sync_config.max_concurrent caps concurrent in-flight sessions; on overflow the inbound stream is dropped and peers retry.

-
-
-

/calimero/blob/…

-

Blob transfer protocol — handled by blob_protocol.rs. Serves blob data from BlobManager to requesting peers.

-
-
-
-
- -
- Subscription Lifecycle -
-

NodeManager manages gossipsub subscriptions for both context topics (one per context) and group topics (group/{hex32}).

-
ctx
-

Context Topics

-

Subscribed when a context is joined/created via NodeClient. Carries StateDelta and HashHeartbeat messages. Unsubscribed on context leave.

-
-
grp
-

Group Topics

-

Subscribed for each group the node participates in. Carries SignedGroupOpV1, GroupStateHeartbeat, and GroupGovernanceDelta. On peer subscription events, the node auto-triggers group sync and alias re-broadcast.

-
-
-
- -
- Periodic Tasks -
-
-
-
⏱
-

Heartbeats

-

SyncManager periodically broadcasts HashHeartbeat per context and GroupStateHeartbeat per group. Interval is configurable via SyncConfig.

-
-
⏱
-

Tombstone GC

-

GarbageCollector actor runs every gc_interval_secs (default 12 hours). Scans all contexts for expired CRDT tombstones past TOMBSTONE_RETENTION_NANOS and deletes them.

-
-
-
-
⏱
-

Blob Eviction

-

Cached blobs in NodeState::blob_cache are evicted based on last_accessed timestamps. The CachedBlob::touch() method refreshes access time on read.

-
-
⏱
-

Sync Loop

-

SyncManager runs a continuous async loop: listen for sync triggers from ctx_sync_rx, select the cheapest applicable protocol (hash comparison → level-wise → snapshot), execute, and track results per peer.

-
-
-
-
-
-
- - -
-

Dependencies

-

What the node crate depends on and what depends on it.

- -

calimero-node depends on

-
calimero-context
ContextManager actor — started by node during bootstrap
-
calimero-context-primitives
ContextClient façade for sending messages to ContextManager
-
calimero-context-config
ContextGroupId, group configuration types
-
calimero-network
NetworkManager actor — started by node, runs libp2p swarm
-
calimero-network-primitives
NetworkClient, NetworkEvent, NetworkConfig, specialized invite types
-
calimero-node-primitives
NodeClient, NodeMessage, BroadcastMessage, sync wire types, DeltaBuffer
-
calimero-store
Store handle, typed keys, column families
-
calimero-store-rocksdb
RocksDB backend for Store
-
calimero-store-encryption
Optional AES encryption layer for the datastore
-
calimero-storage
CRDT storage engine, tombstone constants, EntityIndex
-
calimero-blobstore
BlobManager for binary object storage
-
calimero-server
Axum HTTP server, started by node as tokio task
-
calimero-primitives
Core types: ContextId, BlobId, PublicKey, events
-
calimero-crypto
SharedKey for encrypted sync handshakes
-
calimero-dag
DAG operations used by sync protocols
-
calimero-tee-attestation
TEE attestation verification for specialized node invites; TeeConfig::has_real_attestation() is the predicate used to deny --mock-tee when real KMS attestation is configured
-
calimero-utils-actix
LazyRecipient, ActorExt, Actix utilities
- -

Depended on by

-
merod
Binary crate — calls calimero_node::start() from merod run command
-
calimero-server
Uses NodeClient (from primitives) for blob and application operations
-
calimero-context
Uses NodeClient (from primitives) for broadcast, subscribe, and sync triggers
- -
-
Node
-
Context
-
Network
-
Storage
-
Server
-
Shared / Primitives
-
TEE
-
-
- -
- - - - diff --git a/architecture/crates/runtime.html b/architecture/crates/runtime.html deleted file mode 100644 index 6f42d4e698..0000000000 --- a/architecture/crates/runtime.html +++ /dev/null @@ -1,296 +0,0 @@ - - - - -WASM Runtime — Calimero Core Architecture - - - -
-
-
Home/System Overview/WASM Runtime
- -

WASM Runtime

-

calimero-runtime

- -
-
50+
host functions
-
Wasmer
engine
-
Cranelift
compiler
-
16
VM limits
-
- - -

Purpose

-

Wasmer-based WASM execution engine. The Engine compiles application modules using Cranelift. Module::run builds a VMLogic instance with shared and private Storage, an optional NodeClient, and a ContextHost. It calls the exported method, catches panics, and returns an Outcome containing the return value, logs, events, and storage delta.

-crates/runtime - - -
-

Execution Flow

-

From SDK-generated WASM to executed outcome in a single synchronous call path.

- - - - - - - - - - SDK - Rust / TS / JS - - - - - .wasm - compiled module - - - - - Engine::compile - Cranelift JIT → Module - - - - - - - - Module::run(method, VMContext, Storage) - - - - Storage (shared) - - - Storage (private) - - - NodeClient? - - - ContextHost - - - Wasmer Store + Imports - - - - - - - - instantiate + call - execute exported fn, catch panics - - - - - VMLogic.finish() - collect results + delta - - - - - Outcome - return_value, logs, events, delta - - - → caller - -
- - -
-

VMContext & VMLimits

-

The VMContext carries all input data and identity information needed for a single execution. VMLimits enforces resource boundaries to prevent runaway modules.

- -
-
-

VMContext

-
-pub struct VMContext {
-    input: Vec<u8>, // serialized args
-    context_id: ContextId,
-    executor_public_key: PublicKey,
-    governance_epoch: u64,
-    // group DAG heads for consistency
-} -
-
-
-

VMLimits

-
-pub struct VMLimits {
-    max_memory_pages: u32,
-    max_stack_size: u32,
-    max_registers: u64,
-    max_register_size: u64,
-    max_logs: u64,
-    max_log_size: u64,
-    max_events: u64,
-    max_event_size: u64,
-    max_xcalls: u64,
-    max_storage_key_size: u64,
-    max_storage_value_size: u64,
-    max_method_name_len: u32,
-    max_module_size: u64,
-} -
-
-
- -

Limits are enforced at the host function boundary — each call checks remaining budget before proceeding. Exceeding any limit causes the method to abort with a LimitExceeded error, and all mutations are rolled back.

-
- - -
-

Host Functions

-

The runtime imports 50+ host functions into the WASM module. Each function is registered as a Wasmer import under the "env" namespace. Organized by domain:

- -
- Identity & I/O -
-
context_id
Write the current context ID into a register
-
executor_id
Write the executor's public key into a register
-
input
Read the method's serialized input arguments into a register
-
value_return
Set the return value for the method call
-
log_utf8
Emit a UTF-8 log message (counted against max_logs)
-
panic
Abort execution with an error code
-
panic_utf8
Abort execution with a UTF-8 error message
-
-
- -
- Registers -
-
register_len
Get the byte length of data in a register
-
read_register
Copy register contents into WASM linear memory
-

Registers act as a transfer buffer between host and guest. Host functions write results into numbered registers; the guest reads them out. This avoids complex multi-return calling conventions.

-
-
- -
- Events & Cross-Context -
-
emit
Emit a typed event (broadcast to subscribed clients)
-
emit_with_handler
Emit an event with a named handler method for reactive processing
-
xcall
Cross-context call — invoke a method on another context in the same namespace (async, enqueued)
-
xcall_origin
Read the verified source context of the current xcall — node-set, absent for direct calls
-

Events are accumulated in the Outcome and broadcast to WebSocket/SSE subscribers after execution completes. Cross-context calls are queued and executed asynchronously; each one’s outcome (success, denial, or target error) is broadcast as an XCall event.

-
-
- -
- KV Storage (shared) -
-
storage_read
Read a value by key from shared context state
-
storage_write
Write a key-value pair to shared context state
-
storage_remove
Delete a key from shared context state
-
storage_has
Check if a key exists in shared context state
-
commit
Flush pending mutations to a changeset
-
persist_root_state
Compute and persist the Merkle root hash of current state
-
read_root_state
Read the last computed root state hash
-
apply_storage_delta
Apply a remote delta to local storage (used during sync)
-
flush_delta
Flush accumulated storage changes as a CausalDelta
-
-
- -
- Private Storage -
-
private_storage_read
Read from node-local private state (never synced to peers)
-
private_storage_write
Write to node-local private state
-
private_storage_remove
Delete from node-local private state
-

Private storage is scoped to the local node. Use cases include caching, local preferences, and derived data that doesn't need to be replicated.

-
-
- -
- CRDT / JS Bridge Collections -
-

These host functions back the SDK's CRDT collection types. The js_std_d_* prefix indicates they implement the JavaScript SDK's distributed data structures.

- -

Map operations

-
js_std_d_map_new
Create a new distributed map
-
js_std_d_map_get / set / delete
Read, write, and remove entries
-
js_std_d_map_entries / keys / values
Iterate over map contents
- -

Set operations

-
js_std_d_set_new
Create a new distributed set
-
js_std_d_set_add / delete / has
Mutate and query set membership
- -

Vector & RGA operations

-
js_std_d_vector_new / push / get
Distributed vector (append-mostly)
-
js_std_d_rga_*
Replicated Growable Array for ordered sequences
- -

User & Frozen storage

-
user_storage_*
Higher-level user-facing storage API
-
frozen_storage_*
Read-only snapshot of storage at a specific point
-
-
- -
- Blobs -
-
blob_bytes
Read blob content by ID (returns data into register)
-
blob_exists
Check if a blob exists in the local store
-
blob_create
Store new blob content (content-addressed, returns BlobId)
-
blob_available
Check if a blob is locally available (may need to fetch from peers)
-
-
- -
- Merge Registration -
-
__calimero_register_merge
Register a WASM-exported merge function for custom CRDT conflict resolution
-

Applications can export a custom merge function. When the runtime applies a remote delta and detects a conflict, it invokes the registered merge function instead of using the default strategy. This enables application-specific conflict resolution beyond LWW.

-
-
-
- - -

Storage Interface

-

The runtime uses a simple KV Storage trait that abstracts over the actual storage backend. This allows test isolation via InMemoryStorage.

- -
-
-
-pub trait Storage {
-    fn get(&self, key: &[u8]) -> Option<Vec<u8>>;
-    fn set(&mut self, key: Vec<u8>, value: Vec<u8>);
-    fn remove(&mut self, key: &[u8]) -> Option<Vec<u8>>;
-    fn has(&self, key: &[u8]) -> bool;
-} -
-
-
-
-

InMemoryStorage

-

A HashMap<Vec<u8>, Vec<u8>> implementation used in unit tests and the SDK's local test harness. Provides identical semantics to the RocksDB-backed storage without any external dependencies.

-
-
-

RocksDB-backed Storage

-

In production, the Storage trait is implemented by a thin wrapper over the calimero-store Database, scoped to the executing context's State / PrivateState columns.

-
-
-
- -
-
- - - diff --git a/architecture/crates/sdk.html b/architecture/crates/sdk.html deleted file mode 100644 index fef12d6cd6..0000000000 --- a/architecture/crates/sdk.html +++ /dev/null @@ -1,703 +0,0 @@ - - - - -SDK & Apps — Calimero Core Architecture - - - -
-
-
Home/System Overview/SDK & Apps
- -

SDK & Apps

-

calimero-sdk · calimero-sdk-macros · calimero-sys · calimero-wasm-abi

- - -

Purpose

-

Proc-macro SDK for building Calimero applications. Developers define state with #[app::state], implement methods with #[app::logic], and emit events with #[app::event]. The SDK compiles to WASM and exposes methods as JSON-RPC-callable endpoints. All persistent collections use CRDT types for conflict-free replicated state across peers.

- - -
-

Proc Macro Attributes

-

Each attribute transforms user-written Rust into the glue code needed for WASM execution, storage integration, and ABI generation.

- -
-#[app::state] — persistent state struct -
-

Marks the persistent state struct. Generates automatic storage keys, versioning info, Borsh derives, and CRDT merge registration. Each field with a CRDT type gets its own storage subtree.

-

The optional version = N attribute sets the schema target version for identity-gated migration (see migrate_my_entries in the migrations guide §5). When the state contains an AuthoredMap / AuthoredVector field and version = N is declared, the macro auto-generates a migrate_my_entries() WASM export that sweeps the caller's stale entries up to that target. Defaults to 0 (inert).

-
-use calimero_sdk::app; -use calimero_sdk::borsh::{BorshSerialize, BorshDeserialize}; -use calimero_storage::collections::{UnorderedMap, LwwRegister}; - -// Basic usage — no events, no version target -#[app::state] -pub struct MyApp { items: UnorderedMap<String, LwwRegister<String>> } - -// With events -#[app::state(emits = for<'a> Event<'a>)] -pub struct MyApp { items: UnorderedMap<String, LwwRegister<String>> } - -// With schema-version target (auto-generates migrate_my_entries for AuthoredMap/AuthoredVector fields) -#[app::state(version = 2, emits = for<'a> Event<'a>)] -pub struct NotesV2 { - notes: AuthoredMap<String, LwwRegister<String>>, // migrate_my_entries auto-generated -} -
-
-
- -
-#[app::logic] — WASM-exported methods -
-

Transforms an impl block's public methods into WASM exports. Each method gets input deserialization, output serialization, and error propagation. Private methods are not exported.

-
-#[app::logic] -impl MyApp { - pub fn set(&mut self, key: String, value: String) -> app::Result<()> { - self.items.insert(key, value.into())?; - self.count.increment()?; - Ok(()) - } - - pub fn get(&self, key: &str) -> app::Result<Option<String>> { - Ok(self.items.get(key)?.map(|v| v.get().clone())) - } -} -
-
-
- -
-#[app::event] — typed event declarations -
-

Declares event types that can be emitted via app::emit!. Events are Borsh-serialized and delivered to subscribers via SSE/WebSocket.

-
-#[app::event] -pub enum Event<'a> { - ItemSet { key: &'a str, value: &'a str }, - ItemRemoved { key: &'a str }, -} - -// Emit without handler (fire-and-forget): -app::emit!(Event::ItemSet { key: &k, value: &v }); - -// Emit with handler (triggers on receiving nodes): -app::emit!((Event::ItemSet { key: &k, value: &v }, "on_item_set")); -
-
-
- -
-#[app::init] — initialization hook -
-

Called on first deployment. Sets up initial state and default values. Runs before any logic methods are available.

-
-#[app::init] -pub fn init() -> MyApp { - MyApp { - items: UnorderedMap::new(), - count: Counter::new(), - } -} -
-
-
- -
-#[app::migrate] — state migration between versions -
-

Reads raw bytes from the previous schema, transforms into the new layout, and returns the new state. Use BorshDeserialize::deserialize (not try_from_slice) to handle trailing storage metadata.

-
-#[derive(BorshDeserialize)] -struct OldState { - items: UnorderedMap<String, LwwRegister<String>>, -} - -#[app::migrate] -pub fn migrate_v1_to_v2() -> MyAppV2 { - let old_bytes = read_raw().expect("no state"); - let old: OldState = BorshDeserialize::deserialize(&mut &old_bytes[..]).unwrap(); - MyAppV2 { - items: old.items, - count: Counter::new(), // new field - } -} -
-
-
- -
-#[app::view] — declare a read-only method -
-

Marks a public method as read-only. The declaration is stored in the compiled ABI so the node runtime can take a shared read lock instead of an exclusive write lock, enabling parallel execution of concurrent view calls. Mutually exclusive with #[app::init] (initializers always write state).

-
-#[app::logic] -impl MyApp { - // Exclusive write lock — state may change - pub fn set(&mut self, key: String, val: String) -> app::Result<()> { .. } - - // Shared read lock — concurrent calls run in parallel - #[app::view] - pub fn get(&self, key: &str) -> app::Result<Option<String>> { .. } -} -
-

A method marked #[app::view] must take &self (not &mut self); the macro enforces this at compile time. No state writes may occur inside a view method — the intent is compiler-documented and runtime-exploited for parallelism. As of 0.11.0-rc.6, #[app::view] is registered as a no-op marker and resolves correctly when used through a dependency crate. Design note: see docs/design/context-method-read-write-intent.md for the parallel-read accelerator design.

-
-
- -
-#[app::migration_check] — pre-commit migration guard -
-

Optional companion to #[app::migrate]. Runs against a staging buffer — before the migrated state is committed — and returns bool. false triggers a logical abort: the staging buffer is discarded and the context stays on v1 with zero residue (no snapshot/restore needed). A failed check is retryable; no migration marker is recorded, so the context re-runs migrate+check on its next access.

-
-#[derive(BorshSerialize, BorshDeserialize)] -struct MigrationWitness { v1_count: u64 } - -// Migrate returns (State, Witness) so the check has a v1 baseline -#[app::migrate] -fn migrate_v1_to_v2() -> (DocV2, MigrationWitness) { - let old = read_v1(); - let v1_count = old.items.len().unwrap_or(0) as u64; - (DocV2 { items: old.items, .. }, MigrationWitness { v1_count }) -} - -// Check: old scalars + new collections + optional witness → bool -#[app::migration_check] -fn check_migration(_old: DocV1, new: DocV2, w: MigrationWitness) -> bool { - matches!(new.items.len(), Ok(n) if n as u64 == w.v1_count) -} -
-

Both the check and any witness must be deterministic pure functions of the v1 state — they run independently on every node and must reach the same verdict. See the migrations guide §7 for the full contract.

-

Determinism — no node-local inputs. A migration check (and its witness) must never read env::time_now(), RNG, env::executor_id(), or any other node-local state. Two nodes evaluating different wall-clocks would reach opposite verdicts and fork the network — exactly the divergence Counter::increment / RGA::insert already panic on during a migrate (see store.html, nonce/clock section). Treat the inputs as the only legal source of truth: derive the verdict purely from old, new, and the borsh witness carried out of #[app::migrate]. Use the witness to smuggle any v1-derived quantity (counts, hashes) the check needs, rather than recomputing it from ambient state.

-
-
- -
-#[app::destroy] — cleanup hook -
-

Called on application removal. Can perform resource cleanup, final event emission, or state archival before the context is torn down.

-
-#[app::destroy] -pub fn cleanup(&self) { - app::emit!(Event::AppRemoved {}); - app::log!("Application destroyed, state archived"); -} -
-
-
- -
-#[app::private] — internal helper methods -
-

Marks a method as internal to the application. Private methods are not exported as WASM entry points and cannot be called via the admin API or JSON-RPC. They can only be called by other methods within the same impl block. Useful for shared validation, computation, or state access helpers.

-
-#[app::logic] -impl MyApp { - #[app::private] - fn validate_key(&self, key: &str) -> bool { - !key.is_empty() && key.len() < 256 - } - - pub fn set(&mut self, key: String, value: String) -> app::Result<()> { - if !self.validate_key(&key) { app::bail!("invalid key"); } - self.items.insert(key, value.into())?; - Ok(()) - } -} -
-
-
- -
-app::emit! — emit events -
-

Emits events from logic methods. Supports fire-and-forget or handler-triggered patterns.

-
-// Fire-and-forget (delivered to SSE/WS clients): -app::emit!(Event::ItemSet { key: "foo", value: "bar" }); - -// With handler (executed on receiving nodes, not author): -app::emit!((Event::ItemSet { key: "foo", value: "bar" }, "on_item_set")); - -// Handler must be commutative, idempotent, and pure: -pub fn on_item_set(&mut self, key: &str, value: &str) { - self.counters.increment(); // safe: CRDT op -} -
-
-
- -
-app::err! / bail! / log! — error handling and logging -
-

Error and logging macros that integrate with the WASM host function ABI.

-
-use thiserror::Error; - -#[derive(Debug, Error)] -#[error("not found: {0}")] -pub struct NotFound(String); - -pub fn get(&self, key: &str) -> app::Result<String> { - app::log!("looking up {key}"); - let Some(val) = self.items.get(key)? else { - app::bail!(NotFound(key.to_owned())); - }; - Ok(val.get().clone()) -} -
-
-
- -
-
State / lifecycle
-
Logic / export
-
Events / logging
-
-
- - -
-

App Lifecycle

-

From writing application code to executing methods in a live context.

- -
1
-

Write App with SDK

-

Define your state struct with #[app::state], implement methods with #[app::logic], declare events with #[app::event]. Use CRDT collections (UnorderedMap, UnorderedSet, LwwRegister) for conflict-free replicated state.

-
-
2
-

Build with cargo-mero

-

Run cargo mero build to compile to .wasm. The build tool runs proc-macro expansion, generates the ABI manifest, and targets wasm32-unknown-unknown. The output is a single .wasm binary ready for installation.

-
-
3
-

Install Application

-

Use meroctl app install to upload the .wasm binary to a running node. The node stores it as a blob, computes its BlobId (content hash), and registers it in the application catalog.

-
-
4
-

Create Context

-

Create a context (group + application binding) via the admin API or meroctl context create. This associates the installed application with a new group, sets up storage, and subscribes to the context's gossipsub topic.

-
-
5
-

Execute Methods via JSON-RPC

-

External clients call methods on the running context via JSON-RPC 2.0. Each call is deserialized, executed in the WASM runtime, and produces a delta that is broadcast to all peers via gossip. State converges automatically.

-
- - - - - - - - Write App - SDK macros - - - - - cargo mero build - .wasm output - - - - - meroctl install - blob store - - - - - Create Context - group + app - - - - - JSON-RPC Execute - call methods - -
- - -
-

CRDT Collections

-

Conflict-free replicated data types available to application state. All implement the Mergeable trait for automatic convergence across peers.

- -
-
-

UnorderedMap<K, V>

-

Key-value store with per-key merge semantics. Supports nested maps — inner maps merge recursively. Values must implement Mergeable. Used for most application state.

-
-use calimero_storage::collections::UnorderedMap;

-struct State {
-    users: UnorderedMap<AccountId, UserProfile>,
-    nested: UnorderedMap<String, UnorderedMap<String, Value>>,
-} -
-
-
-

UnorderedSet<T>

-

Add-wins set. Concurrent additions and removals resolve in favor of additions. Elements must be Eq + Hash + Borsh. Ideal for membership lists, tags, and unique collections.

-
-use calimero_storage::collections::UnorderedSet;

-struct State {
-    members: UnorderedSet<AccountId>,
-    tags: UnorderedSet<String>,
-} -
-
-
- -
-
-

LwwRegister<T>

-

Last-write-wins register with logical timestamp. Concurrent writes resolve by highest timestamp. Simple but effective for single-value fields where latest value is always preferred. A drop-stamping guard on value_mut() prevents an incorrect last-writer timestamp from being recorded on mutable-borrow drop (fixed in 0.11.0-rc.6).

-
-use calimero_storage::collections::LwwRegister;

-struct State {
-    title: LwwRegister<String>,
-    config: LwwRegister<AppConfig>,
-} -
-
-
-

RGA (Replicated Growable Array)

-

Sequence CRDT for ordered collections and text. Supports concurrent insert, delete, and move operations. Preserves user intent for collaborative editing scenarios.

-
-// Used internally for text/sequence state
-// Supports: insert_at, delete_at, move
-// Conflict resolution: position-based -
-
-
- -
-
-

Counter

-

Distributed counter generic over ALLOW_DECREMENT: bool. The default (Counter<false>, aliased GCounter) is grow-only — each node increments its own slot and value() returns the sum. Set the flag to true (alias PNCounter) to also support decrement, at the cost of a second per-node map. Naturally commutative and idempotent.

-
-
-

Vector

-

Ordered append-only list. Items are pushed to the end. Concurrent pushes from different nodes both get appended.

-
-
-

UnorderedMap

-

Key-value store with LWW (Last-Write-Wins) semantics per entry. Entries inserted, updated, and removed independently across nodes.

-
-
- -

Storage Primitives

-

Beyond the public CRDT collections above, the storage layer offers wrappers that constrain who can write what. Most are signature-based: after your method returns, the context manager signs the executor's actions with their identity key, and peers verify those signatures at merge time using the runtime's ed25519_verify host function. WASM code itself does not produce signatures — it only stamps placeholders that get filled in post-execution. FrozenStorage is the exception: it has no per-identity check, immutability is structural (enforced by content-addressing). You don't need to call any signing API; pick the primitive whose guarantee matches your data and the trust model follows from the type. See crates/store for the full enforcement details.

- - -
what guarantee do you need? - ├── none, public → UnorderedMap / Vector / UnorderedSet / LwwRegister / Counter - ├── immutable, content-addressed → FrozenStorage<T> - └── identity-bound writes ↓ - ├── one slot per user, disjoint → UserStorage<T> - ├── one shared slot, named writer set → SharedStorage<T> (alias: PermissionedStorage<T, WriterSetAcl>) - ├── one shared slot, single transferable owner → Ownable<T> - ├── shared keyspace, per-entry author → AuthoredMap<K,V> - └── shared sequence, per-entry author → AuthoredVector<V>
- -

See crates/store for the full comparison table including storage stamps, mutation rules, and when composition does (and does not) substitute for these primitives.

- -
-
-

UserStorage<T>

-

Per-user slot keyed by PublicKey. The executor can only write into their own slot; reads are unrestricted. Use for per-member preferences, profiles, or private-but-replicated state.

-
-use calimero_storage::collections::UserStorage;

-struct State {
-    profile: UserStorage<Profile>,
-} -
-
-
-

SharedStorage<T>

-

A single value writable by any signer in a mutable writer set. Any current writer can rotate the set unless it is frozen. Use for group-managed config or shared documents with a controlled author list.

-
-use calimero_storage::collections::SharedStorage;

-struct State {
-    policy: SharedStorage<Policy>,
-} -
-
-
- -
-
-

FrozenStorage<T>

-

Content-addressable immutable storage. insert returns a SHA-256 hash; reads are by hash. Same value always yields the same hash; entries cannot be updated. Use for attachments, snapshots, or any data that should be append-only and de-duplicated. Note: unlike the others in this section, FrozenStorage's authorization is structural — anyone can insert, immutability is enforced by content-addressing, no per-identity signature is checked at merge time.

-
-use calimero_storage::collections::FrozenStorage;

-struct State {
-    attachments: FrozenStorage<Vec<u8>>,
-} -
-
-
-

AuthoredMap / AuthoredVector

-

Shared keyspace with per-entry ownership. Any member can insert/push, but only the original author can update, remove (Map), or tombstone (Vector) their own entries. Use for posts, comments, or any append-friendly collection where authorship matters.

-
-use calimero_storage::collections::{AuthoredMap, AuthoredVector};

-struct State {
-    posts: AuthoredMap<PostId, Post>,
-    feed: AuthoredVector<Event>,
-} -
-
-
- -
-
-

Ownable<T>

-

A single-owner variant of SharedStorage — the writer set is always size 1. Ownership is transferable (transfer_ownership(new_owner)) and renounce-able (renounce_ownership(), which freezes the value). The only_owner() call is a fail-fast API guard; the actual security boundary is merge-time signature verification. See calimero-components for the full component model.

-
-use calimero_storage::collections::Ownable;

-struct State {
-    config: Ownable<LwwRegister<String>>,
-} -
-
-
-

Op-granular writer capabilities (OpMask)

-

The SharedStorage writer set now supports per-writer OpMask bits (INSERT, UPDATE, DELETE, ADMIN). Use grant_capability(who, mask) to restrict a writer to append-only (OpMask::APPEND) or deny deletes. The mask is enforced at merge time by the ProtocolAuthorizer after signature verification — a writer with DELETE not set cannot delete even by crafting a raw delta. A writer with no explicit mask resolves to FULL (all operations permitted).

-
-use calimero_storage::entities::OpMask;

-// Grant write-but-not-delete to a collaborator:
-self.shared.grant_capability(collaborator, OpMask::APPEND)?; -
-
-
- -
-

Mergeable Trait

-

Universal merge interface implemented by all CRDT types. The runtime calls merge() automatically when applying incoming deltas from peers. Custom types can implement this trait for domain-specific merge logic.

-
-pub trait Mergeable {
-    fn merge(&mut self, other: Self) -> MergeResult;
-    fn is_empty(&self) -> bool;
-} -
-
- -
-
Key-value (Map)
-
Collection (Set)
-
Register (LWW)
-
Sequence (RGA)
-
Authorization-aware
-
Trait (Mergeable)
-
-
- - -
-

Mergeable State Lint

-

A compile-time check, applied by #[app::state] and #[derive(Mergeable)], that rejects field types which cannot participate in CRDT merging. Without it, those fields silently fail to converge across replicas — the worst class of distributed-systems bug, because there is no error and no log line, just data that quietly disagrees.

- -
-

The problem

-

In Calimero, two nodes can independently update the same logical state and later sync. Convergence requires every persistent field to know how to merge with another copy of itself — that's the contract Mergeable encodes. Std collections (HashMap, BTreeMap, Vec, ...) and bare primitives (String, u64, ...) have no such contract; before this lint, they would compile fine, get persisted as opaque blobs, and silently diverge.

-

A particularly subtle variant: LwwRegister<HashMap<K, V>>. The wrapper makes the whole field Mergeable, so the type system accepts it — but merging happens at the register level (last-write-wins on the entire map), not at the map entry level. Two nodes inserting different keys concurrently will see one set of inserts win and the other vanish.

-
- -
-

What the lint rejects

-

Walking the AST of every field in a #[app::state] struct (and every field of every #[derive(Mergeable)] struct), recursing into generic arguments:

-
    -
  • Forbidden anywhere in the type tree: HashMap, BTreeMap, HashSet, BTreeSet, LinkedList, VecDeque. These have no merge semantics at any depth — including inside LwwRegister<_>.
    • -
  • Forbidden at the field root only: bare Vec, bare String, bare primitives (u8..u128, i8..i128, f32/f64, bool, char). Inside CRDT collections their use is fine — e.g. UnorderedMap<String, _> uses String as a key, which is just bytes; only the value side has to be Mergeable.
    • -
  • Allowed transparently: Option<T> and Box<T> pass through to their inner type; the lint applies as if the wrapper weren't there.
    • -
-
- -
-

Example diagnostic

-
error: (calimero)> `HashMap` is not allowed here — std collections are not CRDTs and would silently diverge across replicas. Use `UnorderedMap<K, V>` from `calimero_storage::collections` instead. - - If you genuinely need a non-CRDT type, skip `#[app::state]` / `#[derive(Mergeable)]` and implement `Mergeable` by hand — but understand that any non-CRDT field will not converge across replicas. - --> tests/compile_fail/state_hashmap_in_lww.rs:12:24 - | -12 | items: LwwRegister<HashMap<String, String>>, - | ^^^^^^^
-
- -
-

#[derive(Mergeable)]

-

Companion proc-macro for user-defined structs that need to live inside CRDT collections (UnorderedMap<_, V>, Vector<V>, ...). Runs the same field-level lint and generates a field-by-field Mergeable impl that delegates to each field's own merge(). Re-exported as calimero_sdk::app::Mergeable.

-
-use calimero_sdk::app::Mergeable;
-use calimero_storage::collections::{Counter, LwwRegister};

-#[derive(Mergeable)]
-pub struct UserStats {
-    visits: Counter,
-    name: LwwRegister<String>,
-} -
-

Enums are rejected by the derive — there is no canonical merge rule for "left side is VariantA, right side is VariantB". Wrap in LwwRegister<MyEnum> for last-write-wins, or implement Mergeable by hand.

-
- -
-

Escape hatch

-

There is no attribute opt-out. The only way to put a non-CRDT field in persistent state is to not use #[app::state] / #[derive(Mergeable)] and implement Mergeable by hand. This is intentional: any non-CRDT field will not converge, and we want that decision to require explicit code, not a flag. The hand-rolled impl on FileRecord in apps/blobs is a worked example — atomic LWW on an uploaded_at timestamp, justified inline.

-
- -
- Implementation: crates/sdk/macros/src/forbidden_types.rs. Negative tests: crates/sdk/tests/compile_fail/. Originally landed in PR #2276. -
-
- - -
-

Sample Apps

-

Reference applications demonstrating SDK features, CRDT patterns, and migration workflows.

- -
-
-

kv-store

-

Basic key-value store using UnorderedMap<String, LwwRegister<String>>. Demonstrates #[app::state], #[app::logic], and simple CRUD operations. The canonical "hello world" for Calimero apps.

- apps/kv-store -
-
-

collaborative-editor

-

Real-time collaborative text editor. Uses RGA for character-level CRDT operations. Demonstrates event emission for cursor position sharing and multi-user editing.

- apps/collaborative-editor -
-
-

access-control

-

Role-based access control. Uses UnorderedMap<AccountId, UnorderedSet<Role>> for per-user role sets. Demonstrates #[app::private] for internal admin checks.

- apps/access-control -
-
-

blobs

-

Binary large object storage. Demonstrates the blob API for storing and retrieving arbitrary binary data via content-addressed BlobId handles.

- apps/blobs -
-
-

nested-crdt-test

-

Nested CRDT stress test. Uses UnorderedMap<K, UnorderedMap<K, UnorderedSet<V>>> to verify recursive merge behavior under concurrent modifications.

- apps/nested-crdt-test -
-
-

team-metrics

-

Dual-implementation app: one using SDK macros, one with a custom ABI. Demonstrates both approaches side-by-side for comparison.

- apps/team-metrics -
-
-

private_data

-

Private data patterns with encrypted fields and selective disclosure. Uses LwwRegister for per-field encryption keys and #[app::private] for access gates.

- apps/private_data -
-
-

migrations

-

5-version migration suite demonstrating #[app::migrate] across schema changes. Each version adds fields, renames keys, or restructures the state tree.

- apps/migrations -
-
-

e2e-kv-store

-

End-to-end test harness for the kv-store app. Runs multi-node scenarios with concurrent writes, verifies CRDT convergence, and validates sync protocol correctness.

- apps/e2e-kv-store -
-
-

custom-key-store

-

Demonstrates using a custom key type in a Calimero collection. Shows how to implement AsRef<[u8]> (the SDK's StorageKey requirement) for arbitrary structs or numeric types, which are otherwise rejected at compile time as collection keys.

- apps/custom-key-store -
-
-
- -

Developer Guide

- -
-

Handler Safety Requirements

-

Event handlers may execute in parallel across nodes. All handlers must be:

-
-
-

Commutative

-

Order-independent. Counter increments are safe; operations that depend on prior state (create then update) are not.

-
-
-

Independent

-

No shared mutable state. Each handler should use unique keys. Two handlers writing the same map key causes a race condition.

-
-
-

Idempotent

-

Safe to retry. CRDT operations are naturally idempotent. External API calls (payments, notifications) are not.

-
-
-

Pure

-

No external side effects. Only modify CRDT state and emit logs. HTTP calls, file I/O, and non-deterministic operations are forbidden.

-
-
-
- -
-

Common Patterns

-
-
-

Counter

-

Use Counter (G-Counter) for distributed counting. Each node tracks increments separately; value() returns the sum.

-
-
-

Key-Value

-

Use UnorderedMap<K, LwwRegister<V>> for key-value storage. Last-write-wins on concurrent updates to the same key.

-
-
-

Events

-

Use app::emit! for fire-and-forget, or app::emit!((event, "handler_name")) to trigger a handler on receiving nodes.

-
-
-
- -
-

Available Macros

-
-#[app::state] — Mark application state struct -#[app::state(emits = Event)] — With event type -#[app::state(version = N)] — Set schema target (auto-generates migrate_my_entries) -#[derive(app::Migrate)] — Auto-derive migration body (with #[migrate(…)] annotations) -#[app::logic] — Mark implementation block -#[app::init] — Mark constructor -#[app::view] — Declare method read-only (shared lock, parallel execution) -#[app::event] — Mark event enum -#[app::migrate] — Mark migration function -#[app::migration_check] — Pre-commit guard; false → logical abort, zero residue -app::emit!(event) — Emit event -app::log!("msg") — Logging -app::bail!(Error::X) — Early return with error -
-
- -
-

Environment Functions

-
-use calimero_sdk::env; - -let executor = env::executor_id(); // [u8; 32] -let context = env::context_id(); // [u8; 32] -let now = env::time_now(); // u64 nanoseconds -env::log("Hello from WASM"); -
-
- -
-
- - - diff --git a/architecture/crates/server.html b/architecture/crates/server.html deleted file mode 100644 index 8cf32d9114..0000000000 --- a/architecture/crates/server.html +++ /dev/null @@ -1,547 +0,0 @@ - - - - -Server & API — Calimero Core Architecture - - - -
-
-
Home/System Overview/Server & API
- -

Server & API

-

calimero-server + calimero-server-primitives

- -
-
5
HTTP surfaces
-
60+
admin endpoints
-
Axum
framework
-
2
auth modes
-
- - -

Purpose

-

A single Axum HTTP server that binds multiple listeners and mounts all public-facing surfaces: Admin REST API, JSON-RPC 2.0, WebSocket, Server-Sent Events, Prometheus metrics, an optional embedded auth provider (mero-auth), and a static admin dashboard SPA. All business logic is delegated to ContextClient and NodeClient façades.

-crates/server - - -
-

HTTP Surfaces

-

The server exposes five distinct surface types on the same port. Each is mounted as a separate Axum router and merged into the main application.

- - - - - - - - - - Axum Server - single port · TLS optional · tower middleware - - - - - - - - - - - Admin REST API - /admin-api/* - - Protected + public routes - Context management - Group operations - App lifecycle - Blob management - Identity & health - - 30+ routes - - - - JSON-RPC 2.0 - /jsonrpc POST - - execute method - → ContextClient - Spec-compliant - Batch requests - Typed error codes - - WASM - - - - WebSocket - /ws upgrade - - NodeEvent stream - Context subscriptions - Filtered by context - Ping/pong keepalive - - realtime - - - - SSE - /sse persistent - - Persistent sessions - Same event stream - HTTP/1.1 compatible - Auto-reconnect - - events - - - - Prometheus - /metrics - - Text exposition - Request latencies - Active connections - Context counts - - monitoring - - -
-
Admin REST
-
JSON-RPC
-
WebSocket
-
SSE
-
Prometheus
-
-
- - -
-

Admin API Endpoints

-

The Admin REST API provides comprehensive management of contexts, groups, applications, blobs, and node identity. All protected routes require authentication (JWT or proxy).

- -
- Contexts -
-
GET /contexts
List all contexts visible to this node
-
POST /contexts
Create a new context (initializes storage, deploys app)
-
DELETE /contexts/:id
Delete a context and all associated data
-
GET /contexts/:id
Get context details (app, members, state hash)
-
GET /contexts/:id/storage
Inspect raw storage contents
-
GET /contexts/:id/identities
List context identities and their roles
-
POST /contexts/:id/capabilities/grant
Grant a capability to a member
-
POST /contexts/:id/capabilities/revoke
Revoke a capability from a member
-
POST /contexts/:id/sync
Trigger manual sync for a context
-
-
- -
- Groups -
-
GET /groups
List all groups
-
POST /groups
Create a new group
-
GET /groups/:id
Get group info (settings, visibility, member count)
-
PUT /groups/:id
Update group metadata
-
DELETE /groups/:id
Delete a group
-
GET /groups/:id/members
List group members with roles
-
POST /groups/:id/members
Add a member to the group
-
DELETE /groups/:id/members/:mid
Remove a member from the group
-
PUT /groups/:id/members/:mid/roles
Update member roles
-
PUT /groups/:id/members/:mid/aliases
Set member aliases
-
PUT /groups/:id/members/:mid/caps
Set member capabilities
-
GET /groups/:id/contexts
List contexts belonging to this group
-
POST /groups/:id/invite
Generate group invitation
-
POST /groups/:id/join
Join a group via invitation
-
GET /groups/:id/signing-key
Get the group's signing key
-
POST /groups/:id/upgrade
Propose a group app upgrade
-
POST /groups/:id/sync
Trigger group governance sync
-
PUT /groups/:id/settings
Update group settings
-
POST /groups/:id/nest
Nest a child group under this parent
-
POST /groups/:id/unnest
Unnest a child group from parent
-
GET /groups/:id/subgroups
List subgroups of this group
-
-
- -
- Namespaces -
-
GET /namespaces
List all namespaces
-
POST /namespaces
Create a new namespace (root group)
-
GET /namespaces/:id
Get namespace details (app, members, contexts, subgroups)
-
DELETE /namespaces/:id
Delete a namespace (requires no registered contexts)
-
POST /namespaces/:id/invite
Create namespace invitation (supports recursive for all subgroups)
-
POST /namespaces/:id/join
Join a namespace via invitation
-
GET /namespaces/:id/identity
Get this node's namespace identity (public key)
-
GET /namespaces/:id/groups
List groups within this namespace
-
POST /namespaces/:id/groups
Create a new group within this namespace — provisions subgroup signing key and group encryption key before applying the GroupCreated op; key-provisioning failures now return a 500 error response rather than silently succeeding. Accepts an optional visibility field ("open" or "restricted", default "restricted") to set the group's visibility at creation time.
-
GET /namespaces/for-application/:app_id
List namespaces for a specific application
-
-
- -
- Applications -
-
POST /applications/install
Install an application from a package registry
-
POST /applications/install-dev
Install from local path (dev mode, no registry)
-
GET /applications
List installed applications
-
GET /applications/:id
Get application details (version, size, contexts)
-
DELETE /applications/:id
Uninstall an application
-
-
- -
- Blobs -
-
POST /blobs
Upload a binary blob (content-addressed, returns BlobId)
-
GET /blobs/:id
Download blob content
-
GET /blobs/:id/info
Get blob metadata (size, hash, refcount)
-
GET /blobs
List all blobs
-
DELETE /blobs/:id
Delete a blob (if refcount = 0)
-
-
- -
- Packages -
-
GET /packages
List packages in the registry
-
GET /packages/:id/versions
List all versions of a package
-
GET /packages/:id/latest
Get the latest version metadata
-
-
- -
- Identity & Other -
-
POST /identity/context
Generate a new context identity (Ed25519 keypair)
-
GET /health
Health check (returns 200 if node is running)
-
GET /is-authed
Check if the current request is authenticated
-
GET /certificate
Get the node's TLS certificate info
-
GET /peers
List connected peers with connection info
-
GET /tee/info
TEE enclave information (if running in TEE)
-
POST /tee/attest
Request a TEE attestation (returns mock quote when --mock-tee is active)
-
POST /tee/verify
Verify a TEE attestation from another node (accepts mock quotes when --mock-tee is active)
-
-
-
- - -
-

Authentication

-

The server supports two authentication modes. The choice is made at startup via configuration.

- -
-
-

Proxy Mode default

-

No in-process JWT handling. The server assumes an external reverse proxy (e.g. nginx, Envoy, Cloudflare Access) terminates TLS and validates credentials before forwarding requests. The server trusts all incoming requests as authenticated.

-

Suitable for production deployments behind an identity-aware proxy with mTLS or OAuth2 termination.

-
-
-

Embedded Mode mero-auth

-

Full in-process JWT issuing and verification via the mero-auth crate. Mounts additional routes at /auth for login flows (challenge-nonce based). Issues JWT tokens on successful authentication.

-

Protected routes are guarded by an AuthGuardLayer (Tower middleware). Tokens can be passed via:

-
    -
  • Authorization: Bearer <token> header
    • -
  • ?token=<token> query parameter
    • -
-
-
- - - - - - - - - Client - request - - - - - - AuthGuardLayer - validate JWT · extract claims - or pass-through (proxy mode) - - - - - - Route Handler - admin / rpc / ws / sse - - - - - - ContextClient / NodeClient - business logic via actor facades - - - - /auth endpoints - login · challenge · verify · refresh - - - embedded only - -
- - -
-

Execute Caller-Identity Enforcement

-

Both the JSON-RPC /jsonrpc surface and the WebSocket /ws surface share a common execute_request path in crates/server/src/execute.rs. Before any WASM method is invoked, the handler verifies that the authenticated caller's public key is a known member of the target context.

- -
-
-

Before (vulnerable)

-

The executor identity was always auto-resolved to the node's own namespace identity for the context, regardless of which authenticated key made the request. Any valid token could invoke mutating methods on any context the node participated in.

-
-
-

After (fixed)

-

execute_request takes a caller: &PublicKey extracted from the verified AuthenticatedKey extension. It calls ctx_client.has_member(context_id, caller) first; a non-member receives FunctionCallError("Caller is not a member of this context") and the WASM runtime is never reached.

-
-
- -

How the key is threaded per transport

-
    -
  • JSON-RPC: AuthGuardLayer injects AuthenticatedKey(pk) into Axum request extensions after JWT verification. The handle_request extractor pulls it out and passes it through the Request::handle trait to execute_request.
    • -
  • WebSocket: Auth is connection-level (HTTP upgrade). ws_handler extracts AuthenticatedKey at upgrade time and stores the public key in ConnectionStateInner::caller. Each subsequent execute message reads the stored key and passes it to execute_request.
    • -
- -
-

Executor identity vs. caller identity

-

After the membership check passes, the executor identity used to invoke the WASM method is still auto-resolved to the node's owned namespace identity for that context (the key whose private material the node holds). The caller's key gates access; the node's owned key drives execution. This distinction matters for applications that inspect the executor identity inside WASM.

-
-
- - -
-

Server → Actor Connection

-

The server holds an AdminState struct that is shared across all route handlers via Axum's state extraction. All business logic flows through the ContextClient and NodeClient façades — the server never accesses storage or actors directly.

- -
-pub struct AdminState {
-    store: Arc<dyn Database>, // read-only queries (list, get)
-    ctx_client: ContextClient, // execute, create, join, governance
-    node_client: NodeClient, // peers, blobs, boot
-} -
- - - - - - - - - AdminState - - - - store: Arc<dyn Database> - - - ctx_client: ContextClient - - - node_client: NodeClient - - - - - - - - - Storage (read-only) - list contexts, get metadata, query state - - - - ContextManager - via LazyRecipient<ContextMessage> - - - - NodeManager - via LazyRecipient<NodeMessage> - - - All mutations flow - through actor façades. - Server never writes - to storage directly. - - -
- Request lifecycle example -
-
1
-

HTTP Request Arrives

-

Axum receives a POST /admin-api/contexts request. Tower middleware extracts JWT claims (embedded mode) or passes through (proxy mode).

-
-
2
-

Route Handler

-

The route handler extracts AdminState and request body. Constructs the appropriate parameters for the operation.

-
-
3
-

Façade Call

-

Calls state.ctx_client.create_context(params).await. This sends a ContextMessage::CreateContext to the ContextManager actor via LazyRecipient.

-
-
4
-

Response

-

The actor processes the message, returns a Result. The route handler serializes it to JSON and sends the HTTP response.

-
-
-
-
- -

Route Catalog

- -
-

Admin API Routes

-
-— Application Management — -POST /admin-api/dev/install-application -POST /admin-api/dev/install-application-from-url -GET /admin-api/dev/applications -GET /admin-api/dev/applications/:app_id - -— Context Management — -POST /admin-api/contexts -GET /admin-api/contexts -GET /admin-api/contexts/:ctx_id -DELETE /admin-api/contexts/:ctx_id -PUT /admin-api/contexts/:ctx_id/application - -— Group Management — -POST /admin-api/groups -GET /admin-api/groups -GET /admin-api/groups/:group_id -DELETE /admin-api/groups/:group_id -POST /admin-api/groups/:group_id/members -DELETE /admin-api/groups/:group_id/members -GET /admin-api/groups/:group_id/members - -— Identity & Node — -GET /admin-api/identity -GET /admin-api/node-info -GET /admin-api/peers -
-
- -
-

Auth Modes

-
-
-

Embedded Auth

-

Set network.server.embedded_auth = true in config. The server runs the mero-auth JWT service internally. Admin Dashboard SPA is served from /admin/. Frontend assets are baked in at build time via CALIMERO_AUTH_FRONTEND_DIR.

-
-
-

Proxy Auth

-

Run mero-auth as a separate service behind a reverse proxy (nginx/Traefik). The proxy validates tokens via /auth/validate before forwarding to the node. Node remains auth-unaware.

-
-
-
- -
-

Subgroup Creation Behaviour (POST /namespaces/:id/groups)

-

As of the latest change, the create_group_in_namespace handler follows a strict key-first ordering:

-
    -
  1. The subgroup signing key is written to SigningKeysRepository.
    2. -
  2. The group encryption key is written to GroupKeyring.
    3. -
  3. Only after both writes succeed is sign_apply_and_publish_namespace_op called to apply the GroupCreated op.
    4. -
-

Key-provisioning failures are now hard errors. If either key write fails, the request is aborted immediately and the caller receives a 500-style error response. Previously, failures were warn-logged and the request continued, leaving the group in an inconsistent state.

-

Namespace root TEE members are now fully admitted into Restricted subgroups at creation time, because the signing key is available before the op is applied.

-

Visibility field: The request body (CreateGroupInNamespaceBody) accepts an optional visibility string. "open" sets restricted = false on the GroupCreated op; "restricted" (or absent/unrecognized) sets restricted = true. This allows callers to create born-Open subgroups in a single REST call without a follow-up update.

-

Op-store mirroring: After sign_apply_and_publish_namespace_op successfully authors the GroupCreated op, ScopeProjections::persist_namespace_head_ops is called to persist the op into the unified op-store, ensuring the new group's creation is durably recorded and visible to op-store consumers.

-
- -
-

Op-Store Mirroring for Group Reparenting (POST /namespaces/:id/nest & unnest)

-

After sign_apply_and_publish_namespace_op successfully authors a GroupReparented op in the reparent_group admin handler, ScopeProjections::persist_namespace_head_ops is now called to persist the op into the unified op-store. This mirrors the same behaviour introduced for GroupCreated in create_group_in_namespace, ensuring that all structural group hierarchy changes are durably recorded and visible to op-store consumers immediately after the operation succeeds.

-
- -
-

Wire-Fixture Round-Trip Canary Tests

-

Golden JSON files under crates/server/primitives/fixtures/wire/ capture the canonical on-wire shape of six HTTP request/response DTOs:

-
    -
  • CreateContextRequest / CreateContextResponseData
    • -
  • ReparentGroupApiRequest / ReparentGroupApiResponse
    • -
  • ExecutionRequest / ExecutionResponse (JSON-RPC)
    • -
-

A Rust integration test (wire_fixtures.rs) deserializes each fixture into its Rust type and re-serializes it. Any field rename, removal, or type change causes a diff and fails the test. Run UPDATE_FIXTURES=1 cargo test -p calimero-server-primitives --test wire_fixtures to regenerate the golden files; do this whenever an HTTP DTO field changes, then update the matching SDK type accordingly.

-
- -
-

Route-Manifest Guard (endpoints.json)

-

A committed endpoints.json file enumerates all 72 admin HTTP routes. A Rust integration test (route_manifest.rs) parses .route("...", ...) calls from admin/service.rs and asserts they match the manifest exactly — adding a new route without updating the manifest causes a test failure.

-

Run UPDATE_MANIFEST=1 cargo test -p calimero-server --test route_manifest to regenerate endpoints.json after adding or removing routes.

-
- -
-

Endpoint Coverage Enforcement (check-endpoint-coverage.sh)

-

The check-endpoint-coverage.sh script diffs endpoints.json manifest routes against concrete paths recorded by the mero-js e2e run (:param and *glob pattern matching). Known gaps are tracked in coverage-baseline.json:

-
    -
  • Routes present in the baseline are accepted as known gaps and reported as a burndown backlog.
    • -
  • Any newly uncovered route not in the baseline fails the build.
    • -
  • The baseline currently contains only /admin-api/ (the root catch-all), meaning 71 of 72 admin routes must be exercised by e2e tests.
    • -
-

New endpoints without e2e coverage must either be added to coverage-baseline.json with a documented reason, or have a mero-js e2e test added.

-
- -
-

Fleet-Join Cold-Start: NoPeersSubscribedToTopic is Non-Fatal

-

When a node joins a fleet namespace and no peers are yet subscribed to the gossipsub topic (empty mesh at cold start), publish_on_namespace_now returns a NoPeersSubscribedToTopic error. This is now treated as a non-fatal expected condition:

-
    -
  • An info-level log is emitted instead of an error.
    • -
  • The handler falls through to the re-announce polling loop, which retries until peers are available.
    • -
  • The caller does not receive an HTTP 500, and the node does not unsubscribe from the topic.
    • -
  • All other publish errors remain fatal and continue to abort the request.
    • -
-

This is the normal outcome on the very first fleet-join when the gossipsub mesh has not yet formed. The re-announce loop handles propagation once peers connect.

-
- -
-

--mock-tee Flag (Dev/Test Only)

-

merod run gains a --mock-tee flag (also configurable via the MEROD_MOCK_TEE environment variable). When active:

-
    -
  • The fleet-join and attest handlers produce and accept mock attestation quotes instead of requiring real TDX hardware.
    • -
  • A loud tracing::warn is emitted on startup to signal that mock TEE mode is active.
    • -
  • An additional warn fires if a Phala KMS provider is configured alongside mock TEE (likely misconfiguration — real attestation is disabled).
    • -
  • The flag is threaded through NodeConfig → AdminState as AdminState::mock_tee; it is never persisted to the on-disk config.
    • -
-
-

⚠ Security Warning

-

--mock-tee is intended for development and testing only. It is refused at startup (bail!) if the node's TeeConfig has real attestation configured (TeeConfig::has_real_attestation returns true). It must never be used in production environments.

-
-
-
-
- - - - - - diff --git a/architecture/crates/store.html b/architecture/crates/store.html deleted file mode 100644 index e4f0b5488f..0000000000 --- a/architecture/crates/store.html +++ /dev/null @@ -1,592 +0,0 @@ - - - - -Storage Layer — Calimero Core Architecture - - - -
-
-
Home/System Overview/Storage Layer
- -

Storage Layer

-

calimero-store + calimero-store-rocksdb + calimero-dag + calimero-storage

- -
-
11
column families
-
26+
group key prefixes
-
4
crates
-
AES-GCM
optional encryption
-
- - -

Purpose

-

The storage layer provides a column-family key-value abstraction over RocksDB. The core Database trait exposes has, get, put, delete, iter, and apply(Transaction). Keys are typed via generic_array, giving compile-time guarantees on key size and column assignment. An optional AES-GCM encryption layer transparently encrypts values at rest.

- -
-pub trait Database {
-    fn has(&self, col: Column, key: &[u8]) -> Result<bool>;
-    fn get(&self, col: Column, key: &[u8]) -> Result<Option<Slice>>;
-    fn put(&self, col: Column, key: &[u8], value: &[u8]) -> Result<()>;
-    fn delete(&self, col: Column, key: &[u8]) -> Result<()>;
-    fn iter(&self, col: Column) -> Result<DBIterator>;
-    fn apply(&self, tx: Transaction) -> Result<()>;
-} -
- - -
-

Column Architecture

-

All persistent data is partitioned into 11 column families. Each maps to a dedicated RocksDB column family with independent compaction and bloom filters. The Group column is the most complex, containing 26+ logical key prefixes for governance state including namespace identity, namespace governance ops, group hierarchy, and group encryption keys. A new UnifiedOp column was added to serve as the unified causal-log op-store during the dual-write transition (see below).

- - - - - - - - Column Families (11 total) - - - - Meta - schema version, node id - - - Config - node configuration - - - Identity - keypairs, public keys - - - State - shared context state KV - - - PrivateState - per-node private KV - - - - Delta - causal deltas (DAG) - - - Blobs - binary content store - - - Application - WASM binaries, metadata - - - Alias - human-readable aliases - - - - Generic - uncategorized data - - - - UnifiedOp - unified causal-log op-store · key: scope(32) ‖ op_id(32) · value: borsh-serialized Op (Identity codec) · separate from Delta during dual-write transition - - - - - - - - Group Column — Prefix-Partitioned Keys - Each prefix isolates a logical namespace within the single Group column family - - - - - group::info - group metadata - - - group::member - members - - - group::role - role assignments - - - group::cap - capabilities - - - group::alias - member aliases - - - - group::context - linked contexts - - - group::invite - invitations - - - group::signing_key - keys - - - group::parent_ref - hierarchy - - - group::child_index - tree - - - - group::oplog - operation log - - - group::oplog_head - DAG heads - - - group::state_hash - root hash - - - group::nonce - monotonic - - - group::settings - config - - - - group::upgrade - app upgrades - - - group::epoch - governance epoch - - - group::sync_state - sync - - - group::pending_op - queued - - - group::applied_op - applied - - -
-
Membership
-
Resources
-
OpLog / Hashing
-
Lifecycle
-
-

Informal group::… labels in the diagram map to the typed keys and single-byte prefixes in Storage Schema (Group column): group::info → GroupMeta (0x20); group::member → GroupMember (0x21); group::role → role data on GroupMember values (0x21); group::context → GroupContextIndex (0x22) / reverse index ContextGroupRef (0x23); group::upgrade → GroupUpgradeKey (0x24); group::signing_key → GroupSigningKey (0x25); group::cap → GroupMemberCapability (0x26); group::settings (defaults) → GroupDefaultCaps (0x29); migration markers → GroupContextLastMigration (0x2B); group::nonce → GroupLocalGovNonce (0x2C); group::alias (member) → GroupMemberAlias (0x2D), group/context names → GroupAlias (0x2E), GroupContextAlias (0x2F); group::oplog → GroupOpLog (0x30); group::oplog_head → GroupOpHead (0x31); member–context links → GroupMemberContext (0x32), GroupContextMemberCap (0x33). Some cells (e.g. group::invite, group::state_hash) are illustrative and do not match a single prefix—use the schema tables as the source of truth.

-
- - -
-

Key Model

-

All keys are statically typed via Key<T>, a newtype over GenericArray<u8, T::Size>. The AsKeyParts / FromKeyParts traits define how a key is decomposed into column + byte components, giving compile-time column assignment.

- -
-pub struct Key<T: KeyParts>(GenericArray<u8, T::Size>);

-pub trait AsKeyParts {
-    type Size: ArrayLength<u8>;
-    const COLUMN: Column;
-    fn as_key(&self) -> Key<Self>;
-}

-pub trait FromKeyParts: AsKeyParts {
-    fn from_key(key: Key<Self>) -> Self;
-} -
- -

Key Types by Column

- -
Meta
SchemaVersionKey, NodeIdKey — fixed singleton keys
-
Config
NodeConfigKey — serialized node configuration
-
Identity
ContextIdentityKey(ContextId) — maps context → keypair
-
State
ContextStateKey(ContextId, [u8]) — shared KV scoped to context
-
PrivateState
PrivateStateKey(ContextId, [u8]) — private KV (never synced)
-
Delta
DeltaKey(ContextId, DeltaId) — stores CausalDelta payload (legacy; kept separate from UnifiedOp during dual-write transition)
-
UnifiedOp
ScopeUnifiedOp(scope: [u8;32], op_id: [u8;32]) — 64-byte key mirroring ContextDagDelta; value is a borsh-serialized calimero_op::Op stored raw via the Identity codec. Callers own serialization because calimero-store cannot depend on calimero-op. Kept distinct from Column::Delta to prevent key collisions; the two columns are intended to merge once legacy delta rows retire in a later cutover slice.
-
Blobs
BlobKey(BlobId) — content-addressed binary data
-
Application
ApplicationKey(ApplicationId), ApplicationBlobKey — WASM binary + metadata
-
Alias
AliasKey(scope, name) — human-readable name → id mapping
-
Group
GroupInfoKey, GroupMemberKey, GroupRoleKey, GroupCapKey, GroupOpLogKey, etc. — prefix-partitioned (see above)
-
- - -
-

calimero-dag

-

A generic, in-memory causal DAG used for both context state deltas and governance operation logs. Provides topological ordering, pending queues for out-of-order delivery, and missing-parent detection for catch-up.

-crates/dag - -
-
-

CausalDelta<T>

-
-pub struct CausalDelta<T> {
-    id: DeltaId,
-    parents: Vec<DeltaId>,
-    payload: T,
-    timestamp: HLC,
-    expected_root_hash: Hash,
-} -
-

Each delta records its causal parents, forming a partial order. The expected_root_hash enables fast consistency checks — a peer can verify it arrived at the same state after applying a delta.

-
-
-

DagStore<T>

-
-pub struct DagStore<T> {
-    deltas: HashMap<DeltaId, CausalDelta<T>>,
-    applied: HashSet<DeltaId>,
-    pending: Vec<CausalDelta<T>>,
-    heads: HashSet<DeltaId>,
-} -
-

Tracks all known deltas, which have been applied, which are pending (missing parents), and the current DAG head set. New deltas promote heads automatically.

-
-
- -

DeltaApplier<T> trait

-
-pub trait DeltaApplier<T> {
-    fn apply_delta(&mut self, delta: &CausalDelta<T>) -> Result<()>;
-    fn restore_applied_delta(&mut self, delta: &CausalDelta<T>) -> Result<()>;
-} -
- -

Key Operations

-
-
1
-

Topological Ordering

-

Before applying, all pending deltas are sorted in topological order (parents before children). This ensures deterministic replay regardless of arrival order.

-
-
2
-

Pending Queue

-

If a delta's parents haven't been seen yet, it enters the pending queue. When missing parents arrive, queued deltas are automatically drained and applied.

-
-
3
-

restore_applied_delta

-

Used during node restart to rebuild the in-memory DAG from persisted deltas without re-executing the payload (state is already in storage).

-
-
4
-

get_missing_parents

-

Returns the set of delta IDs referenced as parents but not yet received. Used by the sync protocol to request specific deltas from peers.

-
-
-
- - -
-

calimero-storage

-

Provides CRDT collections used by the WASM runtime for conflict-free replicated state. Each collection implements the Mergeable trait for automatic conflict resolution during sync.

-crates/storage - -
-
-

UnorderedMap

-

Observed-remove map. Concurrent puts to the same key are resolved by LWW using HybridTimestamp. Deletions are tracked as tombstones until causally stable.

-
-
-

UnorderedSet

-

Observed-remove set. Add/remove conflicts resolved in favor of add (add-wins semantics). Internally backed by an UnorderedMap with unit values.

-
-
-

LwwRegister

-

Last-writer-wins register. Stores a single value with a HybridTimestamp. On merge, the value with the highest timestamp wins.

-
-
- -

Core Traits

-
-pub struct HybridTimestamp(Timestamp); // from uhlc: NTP64 wall-clock + 128-bit node ID tiebreaker

-pub trait Mergeable {
-    fn merge(&mut self, other: &Self) -> Result<(), MergeError>;
-} -
- -

The Mergeable trait is the fundamental building block — any type that implements it can be used as a CRDT value in the storage layer. The runtime's host functions call merge when applying remote deltas.

-
- - -

RocksDB Implementation

-

The calimero-store-rocksdb crate provides the concrete Database implementation backed by RocksDB.

-crates/store/rocksdb - -
-
-

Column Family Mapping

-

Each Column enum variant maps 1:1 to a RocksDB column family. CFs are created at DB open time. Each has independent compaction, bloom filters (10 bits/key), and block cache partitions.

-
-
-

WriteBatch Transactions

-

The Transaction type accumulates puts and deletes, then is atomically committed via WriteBatch. Guarantees all-or-nothing semantics for multi-key operations like delta application. Transaction::merge combines two transactions by keying the inner map on entry.key() (wrapped in Slice) rather than on the operation's value bytes, and handles both Operation::Put and Operation::Delete with a single insert(entry.key().to_owned(), op.clone()) — a previous version incorrectly used the value bytes as the map key and panicked on Operation::Delete via unreachable!().

-
-
-

Snapshot Iteration

-

Iterators are backed by RocksDB snapshots for consistent point-in-time reads. Prefix iteration uses set_iterate_range for efficient scans within a column family.

-
-
-

Pinned Gets

-

Uses get_pinned_cf for zero-copy reads where possible. The returned Slice borrows directly from the block cache, avoiding allocation for large values.

-
-
- -

CRDT Collections

-

The calimero-storage crate provides application-level CRDT collections built on top of the storage layer. These are what SDK applications use for state management.

- -
-

Available Collections

-
-
-

UnorderedMap

-

Key-value store with LWW (Last-Write-Wins) semantics per entry. Entries can be inserted, updated, and removed independently across nodes.

-
-
-

Vector

-

Ordered append-only list. Items are pushed to the end. Positional inserts and removes use index-based CRDT logic.

-
-
-

Counter

-

Generic over ALLOW_DECREMENT: bool. Default Counter<false> (alias GCounter) is grow-only: each node increments its own slot and value() returns the sum across all nodes. Counter<true> (alias PNCounter) layers a second per-node map to also support decrement. Both variants are commutative and idempotent.

-
-
-
- -
-

Storage Primitives

-

Beyond the public CRDT collections above (UnorderedMap, Vector, UnorderedSet, LwwRegister, Counter) the storage layer provides wrappers that constrain who can write what. Three flavours of constraint:

-
    -
  • Signature-based (UserStorage, SharedStorage, AuthoredMap, AuthoredVector) — the context manager signs each mutating action with the executor's private key (sign_authorized_actions in crates/context/src/handlers/execute/mod.rs) after WASM returns its outcome. Peers verify the signature at merge time in Interface::apply_action in crates/storage/src/interface.rs using the runtime's ed25519_verify host function and reject the action with InvalidSignature on mismatch.
    • -
  • Structural (FrozenStorage) — no per-identity signature; immutability after first write is enforced by content-addressing. Once the SHA-256 hash → value mapping is published, attempts to overwrite the same hash are rejected.
    • -
  • None (the public collections) — anyone in the context can write, update, or remove any entry. Listed below for contrast.
    • -
- -

Comparison

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PrimitiveStorage stampKeyspaceWrites newMutates existingReads
UnorderedMap<K,V>
(public baseline — no auth)
—K → Vanyoneanyoneeveryone
UserStorage<T>User { owner = executor_id }PublicKey → T, disjoint per-user slotsonly into your own slotonly your own sloteveryone (get_for_user)
FrozenStorage<T>
(structural — no signature check)
FrozenHash(value) → T, content-addressedanyonenobody — immutableeveryone
SharedStorage<T>Shared { writers, frozen }one slot with T (T often a nested map)signer ∈ writerssigner ∈ writers; writers rotatable if !frozeneveryone
AuthoredMap<K,V>User { owner } per-entryshared keyspace, K → Vanyone (becomes owner)only ownereveryone
AuthoredVector<V>User { owner } per-entryshared sequence, index → Vanyone (push, becomes owner)only owner (update / tombstone)everyone
- -

Picking one

- -
what guarantee do you need? - ├── none, public → UnorderedMap / Vector / UnorderedSet / LwwRegister / Counter - ├── immutable, content-addressed → FrozenStorage<T> - └── identity-bound writes ↓ - ├── one slot per user, disjoint → UserStorage<T> - ├── one shared slot, named writer set → SharedStorage<T> (alias: PermissionedStorage<T, WriterSetAcl>) - ├── one shared slot, single transferable owner → Ownable<T> (alias: PermissionedStorage<T, OwnerAcl>) - ├── shared keyspace, per-entry author → AuthoredMap<K,V> - └── shared sequence, per-entry author → AuthoredVector<V>
- -
-
-

UserStorage<T>

-

Per-user slot keyed by PublicKey. The executor can only write into their own slot (env::executor_id()); reads are unrestricted. Internally an UnorderedMap<PublicKey, T>.

-
-
-

SharedStorage<T>

-

A single value writable by any signer in a mutable writers set. Any current writer can rotate the set unless it is frozen. The context manager signs each write after WASM returns its outcome; peers verify the signature against the stored writer set at merge time. See ADR 0001 for the rotation-during-concurrent-write contract.

-
-
-
-
-

FrozenStorage<T>

-

Content-addressable immutable storage. insert returns a SHA-256 hash; reads are by hash. Same value always yields the same hash, and entries cannot be updated once written. Internally an UnorderedMap<Hash, FrozenValue<T>> with first-write-wins semantics.

-
-
-

AuthoredMap<K, V>

-

Shared keyspace map with per-entry ownership. Any member can insert a new key; only the inserter can update or remove their own entries. Each entry carries a StorageType::User { owner } stamp set from the executor's public key at insert time.

-
-
-
-
-

AuthoredVector<V>

-

Ordered shared vector with per-entry ownership. Any member can push; only the pusher can update or tombstone their entry. There is no physical remove — shifting indices would complicate concurrent-push merge semantics. Use tombstone(idx) to retract a slot.

-
-
-

Merge-time enforcement (signature-based primitives)

-

Local update/remove calls short-circuit non-owner attempts so bugs surface in-process. The load-bearing check happens at merge time in Interface::apply_action:

-
    -
  • Signature — the runtime's ed25519_verify host function is called against the entry's stored owner (or, for SharedStorage, against the current writer set). Mismatch returns InvalidSignature.
    • -
  • Replay nonce — incoming nonce must be strictly greater than the stored value; equal nonces are rejected as NonceReplay (i.e. the comparison is incoming > stored, not ≥). The nonce is the action's SignatureData.nonce, set from env::time_now() (a wall-clock nanosecond timestamp from SystemTime::now()) at action build time. Caveat: wall-clock time is not strictly monotonic across NTP slews, leap-second corrections, VM-host clock changes, or process restarts. Two consecutive writes on the same node can collide or step backward under those conditions, in which case the second write is rejected as NonceReplay; conversely a node with a far-future clock can write a nonce that effectively locks the entity until later honest writes catch up. Per the inline comment in interface.rs, the nonce check is itself a transitional v2 mechanism that the project plans to retire after a soak period in favour of DAG-causal verification.
    • -
  • Per-entity scope — the stored nonce lives on the individual storage entity (the row keyed by entity id), not on the owner globally. For AuthoredMap each map key is its own entity; for UserStorage each per-user slot; for SharedStorage the single value.
    • -
  • Entity binding — a signed action targeting one entity cannot be replayed against a different entity even when the same key signed both: the entity id is part of the bytes hashed by Action::payload_for_signing(), so the signature is bound to that specific entity.
    • -
-
-
- -
-
-

PermissionedStorage<T, A> / Ownable<T>

-

Policy-parameterised wrapper over the underlying WriterSetCell<T> storage primitive. The type parameter A: Authorizer is a zero-sized policy marker that decides, at the API surface, whether the current executor may perform a given Op. Two built-in policies ship:

-
    -
  • WriterSetAcl — any member of the writer set may perform any op (the behaviour of plain SharedStorage). Aliased as SharedStorage<T>.
    • -
  • OwnerAcl — the single writer is the owner; Ownable<T> is the ergonomic alias. Supports transfer_ownership and renounce_ownership.
    • -
-

The security boundary is unchanged: merge-time signature verification against the writer set. The Authorizer is fail-fast UX sugar — it lets a method reject an unauthorised caller early, but cannot substitute for correct storage-type placement (data must live inside the wrapper to be protected).

-
-use calimero_storage::collections::{Ownable, PermissionedStorage, WriterSetAcl};

-struct State {
-    config: Ownable<LwwRegister<String>>, // single owner, transferable
-    policy: PermissionedStorage<LwwRegister<String>, WriterSetAcl>,
-} -
-
-
-

OpMask — operation-granular capabilities

-

The writer set on a Shared entity has been extended from BTreeSet<PublicKey> to BTreeMap<PublicKey, OpMask>. Each writer now carries a bitmask of permitted operations, enforced at merge time by the ProtocolAuthorizer after signature verification:

-
-bitflags! {
-    pub struct OpMask: u8 {
-        const INSERT = 0b0000_0001; // create new entity
-        const UPDATE = 0b0000_0010; // modify existing
-        const DELETE = 0b0000_0100; // delete
-        const ADMIN = 0b0000_1000; // rotate writers / grant / revoke
-        const APPEND = INSERT | UPDATE; // write, no delete
-        const WRITE = INSERT | UPDATE | DELETE;
-        const FULL = WRITE | ADMIN;
-    }
-} -
-

A key with no explicit mask resolves to OpMask::FULL — preserving today's "in the set ⇒ anything" behaviour. grant_capability(who, mask) sets a per-principal mask; concurrent conflicting grants merge by bitwise AND (revoke-wins, fail-safe). WRITE/DELETE/ADMIN bits are merge-enforced; INSERT vs UPDATE is deferred (needs exists_at_cut).

-
-
- -

Key distinctions

-
-
-

UserStorage vs AuthoredMap

-

Both stamp entries with StorageType::User { owner }. The difference is keying:

-
    -
  • UserStorage<T>: key is the public key, slots are disjoint per user.
    • -
  • AuthoredMap<K,V>: key is application-defined, owner is recorded in the entry's metadata. Two users can compete to insert the same key; first writer wins, subsequent updates are owner-only.
    • -
-
-
-

SharedStorage vs AuthoredMap

-

Both allow mutation after creation. The difference is the granularity of the writer set:

-
    -
  • SharedStorage<T>: one collection-level writer set governs one logical T. Several named members can co-author the same value.
    • -
  • AuthoredMap<K,V>: per-entry writer set (currently size-1 = single author). The writer varies per key.
    • -
-
-
- -

Why composition doesn't replace these primitives

-
-
-

AuthoredMap is not UnorderedMap<K, SharedStorage<V>>

-

You can nest SharedStorage inside an UnorderedMap for per-key multi-writer values, but the outer map is still public — outer keys can be inserted, overwritten, or removed by anyone. AuthoredMap puts the ownership stamp on the entry itself within a shared keyspace, so K cannot be replaced by non-owners.

-
-
-

SharedStorage is not UnorderedMap<K, UserStorage<V>>

-

You can nest UserStorage inside an UnorderedMap for per-key single-author values, but again the outer map is public (key-level tampering remains possible). And there is no way to model "several named people co-own this value" without a writer set — exactly what SharedStorage provides.

-
-
-
- -
-

Merge Semantics

-

All collections implement the Mergeable trait. When state deltas arrive from peers, the storage layer calls merge() on each affected entry. The merge is:

-
    -
  • Commutative — merge(A, B) = merge(B, A)
    • -
  • Associative — merge(merge(A, B), C) = merge(A, merge(B, C))
    • -
  • Idempotent — merge(A, A) = A
    • -
-

These properties guarantee eventual consistency regardless of message ordering or duplication.

-
- -
-
- - - diff --git a/architecture/crates/sync.html b/architecture/crates/sync.html deleted file mode 100644 index 58f1a31b51..0000000000 --- a/architecture/crates/sync.html +++ /dev/null @@ -1,783 +0,0 @@ - - - - -Sync Engine — Calimero Core Architecture - - - -
-
-
Home/System Overview/Sync Engine
- -

Sync Engine

-

Sync protocols from calimero-node · calimero-node-primitives

- - -

Purpose

-

Two independent synchronization systems keep peers converged: application state sync (context DAG) and group governance sync (governance DAG). The SyncManager runs periodic and on-demand sync cycles. Protocols are selected dynamically via handshake negotiation based on divergence metrics.

- -
-
4
app sync protocols
-
3
governance mechanisms
-
30s
heartbeat interval
-
N
concurrent syncs
-
- - -
-

Protocol Selection

-

Peers exchange a SyncHandshake to compare state. The comparison metrics determine which protocol minimizes bandwidth and latency for the current divergence level.

- - - - - - - - - - - Peer A - root_hash, dag_heads - entities, depth - - - - Peer B - root_hash, dag_heads - entities, depth - - - - - - - - SyncHandshake - Compare root_hash + dag_heads - entity count + tree depth - - - - - - - - divergence? - - - - - - - - - Hash Comparison - Merkle tree walk - Small diffs, few nodes - LOW bandwidth - - small - - - - - - - Level-wise - Wide-tree traversal - Broad, shallow divergence - MEDIUM bandwidth - - broad - - - - - - - Snapshot - Full state transfer - Paged chunks + DeltaBuffer - HIGH bandwidth - - large - - - - - - - - Delta Sync - Direct DAG exchange - Bloom filter + subtree prefetch - OPTIMAL bandwidth - - direct DAG - - - - - Hash Comparison - - Level-wise - - Snapshot - - Delta Sync - -
- - -
-

Application State Protocols

-

Four protocols for synchronizing context application state. Each is optimized for a different divergence scenario.

- -
- Hash Comparison — Merkle tree walk -
-

The lightest sync protocol. Both peers share their Merkle tree root hash. If roots differ, they exchange hashes at progressively deeper levels to locate the divergent subtrees. Only the changed subtrees are transferred.

-

Algorithm

-
1
-

Root Compare

-

Exchange root hashes. If equal, sync is a no-op. If different, descend to the next level.

-
-
2
-

Level Descent

-

Exchange hash nodes at the current depth. Mark subtrees with matching hashes as "synced" and skip them. Continue descending into differing subtrees.

-
-
3
-

Leaf Transfer

-

At leaf level, exchange the actual data entries for divergent nodes. Apply incoming entries to local storage and update the Merkle tree.

-
-

Best for: Small, localized changes — a few keys updated in a large state tree.

- crates/node/src/sync -
-
- -
- Level-wise — wide-tree traversal -
-

Optimized for broad, shallow divergence where many branches differ but the tree isn't deep. Instead of drilling into individual branches, this protocol processes entire levels in parallel.

-

Algorithm

-
1
-

Level Exchange

-

Both peers serialize their entire tree at the current level and exchange the batch. This avoids the round-trip overhead of per-branch negotiation.

-
-
2
-

Batch Diff

-

Compare the received level against the local level. Identify additions, deletions, and modifications in a single pass.

-
-
3
-

Descend Parallel

-

For each differing node, descend to the next level. Process all descendants in parallel rather than sequentially.

-
-

Best for: Many keys changed across different parts of the tree — bulk updates, migrations.

-
-
- -
- Snapshot — full state transfer -
-

Fallback protocol for maximum divergence or when a peer has no prior state (new joiner). Transfers the entire state as paged chunks with a DeltaBuffer to capture mutations during transfer.

-

Algorithm

-
1
-

Snapshot Begin

-

Source peer takes a consistent read snapshot and begins chunking the state into pages of snapshot_chunk_size bytes. A DeltaBuffer is opened to capture any writes that occur during the transfer.

-
-
2
-

Chunk Transfer

-

Pages are streamed to the target peer. Each page includes a sequence number and hash for integrity verification. The target applies pages to a staging area.

-
-
3
-

DeltaBuffer Replay

-

After all pages are transferred, the accumulated DeltaBuffer entries (mutations that occurred during the snapshot) are replayed on top of the restored state.

-
-
4
-

Commit

-

The staging area is atomically swapped into the live state. The Merkle tree is rebuilt from the new state.

-
-

Best for: New peers joining a context, or peers that have been offline for a very long time.

-
-
- -
- Delta Sync — direct DAG exchange -
-

Direct exchange of causal delta entries between peers. Uses bloom filters to efficiently identify which deltas the remote peer is missing, and subtree prefetch to minimize round-trips.

-

Algorithm

-
1
-

Bloom Filter Exchange

-

Each peer builds a bloom filter of their DAG entry hashes and exchanges it. This allows each side to quickly identify entries the other is likely missing without enumerating the full DAG.

-
-
2
-

Delta Request

-

Based on bloom filter results, the receiver requests specific delta entries by hash. The sender responds with the full CausalDelta payloads.

-
-
3
-

Subtree Prefetch

-

When a requested delta has parent hashes that the receiver also lacks, the sender proactively includes the parent chain (subtree prefetch) to reduce additional round-trips.

-
-
4
-

Apply + Verify

-

Received deltas are verified for causal consistency (all parents present) and applied to the local DAG in topological order. The Merkle tree is incrementally updated.

-
-

Best for: Moderate divergence with known recent activity — the most commonly used protocol during normal operation.

-
-
-
- - -
-

Group Governance Sync

-

Governance operations form a separate DAG and use dedicated sync mechanisms. Three complementary approaches ensure governance state converges across all group members.

- -
-

Projection-Authoritative Gossip Write Authorization (authorize_delta_at_edge_projected)

-

The gossip state-delta path (handle_state_delta) calls authorize_delta_at_edge_projected in verify.rs. This function is now the sole authorizer for all delta authorization paths — the live authorize_delta_at_edge function backed by acl_view_at has been deleted from verify.rs and is no longer re-exported from state_delta/mod.rs. No live resolver is consulted on any path; no divergence cross-check or shadow logging is performed.

-

authorize_delta_at_edge_projected accepts a caller-supplied resolve closure that returns a CutMembership enum. All call sites — gossip apply, parent-fetch (request_missing_deltas / verify_fetched_parent), snapshot-replay (replay_buffered_delta), and DAG-catchup in SyncManager — now pass a closure backed by the shared resolve_cut_membership renderer (see below).

-

The three CutMembership variants map directly to authorization outcomes:

-
    -
  • Member(role) → Authorized — projection confirms membership with a known role; write proceeds normally.
    • -
  • NotMember → MembershipReject — projection definitively rejects; delta is discarded.
    • -
  • Incomplete → Buffer — governance ancestry is not yet fully folded (no embedded needed set; the buffer's needed heads are populated from the governance position's heads); delta is held in the pending buffer and re-evaluated when governance catches up.
    • -
-

The entire previous co-authorizer / shadow-logging block — live authorize_delta_at_edge call, divergence markers, role-shadow logging (unified_projection_divergence / plane=data-write-role), decision-shadow logging (data_write_decision_shadow), and the MembershipReject grant-override via projection_member_at_cut_authoritative — has been removed.

-

DeltaAuthOutcome::MembershipReject no longer carries a group: ContextGroupId field. Code matching on this variant must be updated to omit the field (use .. if forward-compatibility is needed).

-

Scope: This change now applies to all authorization paths. Parent-fetch (request_missing_deltas and verify_fetched_parent) and snapshot-replay (replay_buffered_delta) gain a new node_state: &NodeState parameter to carry the projection reference. The DAG-catchup head-pull in SyncManager is similarly updated.

-

Visibility: projection_member_at_cut is refactored to delegate to resolve_cut_membership (discarding the role) so both helpers cannot drift. resolve_cut_membership is pub(crate) and is the single projection verdict renderer.

-
-// Sole authorization entry point for all delta paths (verify.rs)
-// authorize_delta_at_edge (live/acl_view_at-backed) has been deleted
-pub fn authorize_delta_at_edge_projected<F>(
-    ctx: &DeltaContext,
-    resolve: F,
-) -> DeltaAuthOutcome
-where
-    F: FnOnce() -> CutMembership

-// Shared projection verdict renderer (state_delta/mod.rs)
-// Refreshes scope projection (write lock), then reads member + role
-// under a single read guard for epoch consistency.
-// Called by gossip-apply, parent-fetch, snapshot-replay, DAG-catchup.
-pub(crate) fn resolve_cut_membership(
-    node_state: &NodeState,
-    ctx: &DeltaContext,
-) -> CutMembership

-// CutMembership — Incomplete carries no embedded needed set
-pub enum CutMembership {
-    Member(Role),    // → Authorized
-    NotMember,      // → MembershipReject
-    Incomplete,     // → Buffer (governance ancestry still folding; no needed set)
-}

-// MembershipReject no longer carries a group field
-// DeltaAuthOutcome::MembershipReject (no group: ContextGroupId) -
-

Operators: Gossip deltas whose governance ancestry is not yet folded are now held in the pending buffer rather than discarded. No unified_projection_divergence or data_write_decision_shadow warnings are emitted from any write-auth path. These log markers no longer appear for the data-write plane and any alerts configured on them for this path may be retired. Doc comments in peer_identity_cache.rs and namespace_sync.rs have been updated to reference projection-based membership resolution rather than the deleted live resolver.

-
- -
-

Projection Rebuild Indirection — ops_for_namespace

-

ScopeProjections::ops_for_namespace is the new stable indirection point for loading ops at every projection-rebuild site. It currently delegates directly to collect_namespace_ops (the governance-DAG walk) — it does not yet read from the unified op-store (load_scope_ops). The C2.2b flip to the op-store is explicitly deferred until late-decrypted membership is fixed.

-

Root cause for the deferral: an encrypted MemberAdded op can be applied to the governance DAG before its group key arrives. The apply-time dual-write freezes it as Noop in the op-store, while collect_namespace_ops re-decrypts at read time once the key lands. Using the op-store as the projection source would therefore silently drop members and break joiner sync.

-

All three internal projection-rebuild call sites in scope_projection.rs — backfill_namespace, member_now_ephemeral/ephemeral_fold, and the scope_root reconstruction — and the call site in refresh_projection_for_cut in the node crate now call ops_for_namespace instead of collect_namespace_ops directly. collect_namespace_ops remains as the implementation that ops_for_namespace currently delegates to.

-

Inside refresh_projection_for_cut, after ops_for_namespace returns the governance-DAG fold, ScopeProjections::check_op_store_completeness is called with those ops before apply_backfill. This adds one op-store read on cold/refresh cuts (warm cuts skip via namespace_to_refresh returning None). The result is ignored by the caller; the gate is purely diagnostic and never affects whether or how the backfill is applied. The projection still folds via collect_namespace_ops.

-
-// Stable indirection layer — all projection-rebuild sites call this
-// Currently delegates to collect_namespace_ops (governance-DAG walk)
-// Will be flipped to load_scope_ops (op-store) once late-decrypt is fixed
-pub(crate) fn ops_for_namespace(
-    &self,
-    namespace: &Namespace,
-) -> Result<Vec<Op>>

-// Observe-only completeness gate — called in refresh_projection_for_cut
-// after ops_for_namespace, before apply_backfill. One op-store read on
-// cold/refresh cuts; skipped on warm cuts. Result is always ignored.
-pub(crate) fn check_op_store_completeness(
-    &self,
-    ops: &[Op],
-) -> Result<()>

-// Implementation (unchanged); called only via ops_for_namespace
-fn collect_namespace_ops(
-    &self,
-    namespace: &Namespace,
-) -> Result<Vec<Op>>

-// Removed from ScopeProjections:
-// shadow_compare_op_store — cross-check was unreliable (see above)
-// scope_root_from_governance_dag — companion to shadow_compare_op_store -
-

Operators: No shadow_compare_op_store log output is emitted from the backfill path. Any monitoring rules that relied on post-backfill op-store comparison metrics from this path may be retired. The ops_for_namespace indirection is the future flip point; watch for a follow-up change that routes it to load_scope_ops once late-decrypted membership handling is resolved. check_op_store_completeness is diagnostic only — its result is discarded and it does not block or alter the backfill.

-
- -
-
- -
-

Op-Store Refresh on Late Key Delivery (recover_missing_group_keys) — sole persist trigger

-

SyncManager::recover_missing_group_keys is the single runtime chokepoint for direct group-key delivery. Immediately after a successful key delivery and the existing drain/reconcile steps, ScopeProjections::repersist_namespace_ops is called for the affected namespace_id. This ensures that any MemberAdded ops that were frozen as Noop in the op-store (because their group key had not yet arrived at apply time) are re-decrypted and written into the op-store immediately when the key lands — without waiting for the next projection rebuild cycle.

-

ScopeProjections::persist_namespace_head_ops — a thin alias that previously documented a per-site 'C3 Stage 1/2/3' persist pattern — has been deleted. All direct call sites (create_group, delete_group, join_context, join_subgroup_inheritance, remove_group_members, join_namespace, reparent_group, create_group_in_namespace) have been removed. repersist_namespace_ops called from recover_missing_group_keys is now the sole trigger for op-store persistence of governance ops.

-

Because all late-arriving keys flow through recover_missing_group_keys, no other call site needs to trigger repersist_namespace_ops for the key-delivery case. The call is placed after drain/reconcile so the projection is already updated before the op-store is refreshed.

-
-// Called in SyncManager after each successful key delivery
-pub(crate) fn repersist_namespace_ops(
-    &self,
-    namespace_id: NamespaceId,
-) -> Result<()> -
-

Operators: After this change, late-decrypted MemberAdded ops are persisted to the op-store as soon as their key arrives rather than on the next projection rebuild. No new log markers are emitted; the existing key-delivery trace spans cover this path.

-
- -
-

Post-Backfill Shadow Compare (refresh_projection_for_cut) — Removed

-

After apply_backfill writes the governance-DAG fold into the maintained projection inside refresh_projection_for_cut, a post-fold cross-check against the op-store was previously triggered via shadow_compare_op_store. This call and both shadow_compare_op_store and scope_root_from_governance_dag have been removed from ScopeProjections.

-

The cross-check was flawed: the maintained projection holds live ingest_op-fed ops that may not yet be in the op-store (an encrypted MemberAdded op can be applied to the governance DAG before its group key arrives, causing the apply-time dual-write to freeze as Noop in the op-store while collect_namespace_ops re-decrypts at read time once the key lands). The op-store gap was therefore silently masked by the live apply path, making the shadow compare unreliable. No observational backfill check of this kind is performed.

-
- -
-

Projection-Authoritative Drain (drain_governance_pending)

-

State deltas that arrived before their governance ancestry was locally available are buffered and re-evaluated by drain_governance_pending once new governance ops are applied. This drain now uses member_at_cut_authoritative (projection) as its sole decision authority, replacing the previous acl_view_at (live DAG walk) primary path.

-

The three projection outcomes map directly to the three drain outcomes:

-
    -
  • Some(true) → re-apply — delta is re-submitted to the normal apply path. No cross-check is performed; the re-apply decision is purely from member_at_cut_authoritative.
    • -
  • Some(false) → drop — delta is discarded; the author is definitively not a member at that cut. acl_view_at is called after the drop decision solely to label the drop metric ("removed" vs "never_member"); no unified_projection_divergence warn is emitted.
    • -
  • None → re-buffer — governance ancestry is still incomplete; the delta stays pending for the next drain cycle.
    • -
-

Because the projection never returns an error, the previous Err match arm that dropped a buffered delta and recorded a "lookup_error" metric has been eliminated entirely.

-

acl_view_at Usage

-

acl_view_at is called only in the Some(false) (drop) arm, and only to obtain a best-effort metric label. It is not called in the Some(true) arm (no grant-plane cross-check) and not called on None (to avoid false positives while ancestry is still being folded). No unified_projection_divergence warning is emitted on any drain path.

-

Metric Changes

-

The governance_drain_outcome metric label for drop events is best-effort. It is derived from acl_view_at's response in the drop arm: "removed" if live says Removed, "never_member" if live says NeverMember, and the fallback "not_member" if the live resolver is inconclusive. The "lookup_error" label is no longer emitted. Grant-arm (Some(true)) drop metrics are unaffected as acl_view_at is no longer called there.

-

The debug log emitted when a delta is re-buffered no longer includes a needed_count field, because the projection returns a boolean None rather than a structured variant enumerating missing DAG heads.

-
-// Drain outcomes — now driven by projection
-// Some(true) → re-apply
-// Some(false) → drop (metric label: "removed" | "never_member" | "not_member")
-// None → re-buffer (no needed_count in log)
-pub fn member_at_cut_authoritative(
-    ctx: &DeltaContext,
-    cut: &Cut,
-) -> Option<bool> -
-

Operators: No unified_projection_divergence warns are emitted from the drain path. Update any metric dashboards to handle the new "not_member" label and the absence of "lookup_error" on governance_drain_outcome. Deltas that previously reached the drain via the live-path MembershipReject drop-with-no-recovery will now instead arrive in the drain via the gossip path's Incomplete → Buffer outcome, so drain throughput may increase during governance catch-up periods.

-
- -
-

Inbound-Sync Peer Authorization (peer_is_group_member)

-

A new private method SyncManager::peer_is_group_member encapsulates membership authorization for inbound sync peers. It resolves the current governance heads via ScopeProjections::namespace_current_heads and calls projection_member_at_cut to obtain the projection's verdict. If the projection returns None (cold or partially folded governance), the method falls back to the live result (MembershipRepository::is_member) so no peer is spuriously rejected while governance ancestry is still being caught up. When the projection returns Some(x) it is used as the authoritative decision with no cross-check warn emitted.

-

Two call sites inside SyncManager that previously called MembershipRepository::new(store).is_member(...) directly — the materialization-wait verification and the inherited-member check in verify_inbound_member — now call self.peer_is_group_member(...), so the projection-first logic applies uniformly to all inbound peer authorization.

-
-// Authorizes an inbound sync peer: projection-first, live fallback
-async fn peer_is_group_member(
-    &self,
-    group_id: GroupId,
-    peer_id: PeerId,
-) -> Result<bool>

-// projection returns None → fall back to live (projected.unwrap_or(live))
-// Some(x) → authoritative; no cross-check warn emitted -
-

Operators: A None projection result is not an error — it means governance ancestry is still being folded and the live result is used as a safe temporary answer. No unified_projection_divergence warning is emitted on this path.

-
- -
-
-

Real-time Gossip

- primary -

SignedGroupOpV1 operations are published via gossipsub on dedicated group topics. Each operation is verified on ingress — signature check, capability check, parent hash validation. Valid operations are fed directly into the local DagStore.

-
-// Gossip on group topic
-pub fn publish_group_op(
-    topic: &GroupTopic,
-    op: &SignedGroupOpV1,
-) -> Result<()> -
-
-
-

Heartbeat Comparison

- periodic -

GroupStateHeartbeat is broadcast every 30 seconds. Contains the group's current dag_heads (latest operation hashes). Peers compare heads and trigger catch-up if they differ.

-
-pub struct GroupStateHeartbeat {
-    group_id: GroupId,
-    dag_heads: Vec<Hash>,
-    op_count: u64,
-} -
-
-
- -
-
-

Stream Catch-Up

- on-demand -

When heartbeat comparison reveals a mismatch, a direct stream is opened to the divergent peer. The peer exchanges GroupDeltaRequest specifying known heads, and receives a GroupDeltaResponse with the missing operations scanned from the remote OpLog.

-
-pub struct GroupDeltaRequest {
-    group_id: GroupId,
-    known_heads: Vec<Hash>,
-}

-pub struct GroupDeltaResponse {
-    ops: Vec<SignedGroupOpV1>,
-    has_more: bool,
-} -
-
-
-

Startup Recovery

- boot -

On node startup, reload_group_dags reads the persistent OpLog from RocksDB and reconstructs all group DAGs in memory. The DAG heads are then compared with peers via the first heartbeat cycle to catch up on any operations missed while offline.

-
-// Called at node boot
-pub async fn reload_group_dags(
-    store: &RocksDBStore,
-) -> Result<HashMap<GroupId, DagState>> -
-
-
- - - - - - - - GroupOp Created - sign + apply local - - - - - Gossip Broadcast - group topic - - - - - Peer Ingestion - verify + DagStore - - - - - Heartbeat Check - dag_heads compare - - - - - Stream Catch-Up - DeltaReq / DeltaResp - - -
-
Real-time gossip
-
Heartbeat periodic
-
Stream catch-up
-
Startup recovery
-
-
- - -
-

SyncManager

-

Long-running async task (not an Actix actor) that orchestrates all synchronization. Runs periodic sync cycles and handles on-demand sync triggered by incoming streams.

- -

Configuration

-
-pub struct SyncConfig {
-    timeout: Duration,             // per-sync timeout
-    interval: Duration,            // periodic sync interval
-    frequency: u32,                // syncs per interval
-    max_concurrent: usize,         // max parallel sync streams
-    snapshot_chunk_size: usize,    // bytes per snapshot page
-    delta_sync_threshold: u64,     // entity count for delta vs snapshot
-} -
- -

Core Loop

-
-
-
1
-

start() — Spawn Loop

-

The start() method spawns the main sync loop as a Tokio task. It runs indefinitely, waking on a timer tick or when an incoming stream is opened by a remote peer.

-
-
2
-

Periodic Tick

-

On each interval tick, iterates over all active contexts. For each context, builds a SyncState (current root_hash, dag_heads, entity count) and selects a peer for synchronization.

-
-
3
-

Peer Selection

-

Selects a sync peer based on: most recently seen heartbeat, lowest latency, and whether they reported a different state hash. Avoids peers already in an active sync session.

-
-
-
-
4
-

Concurrent Futures

-

Sync operations run as bounded concurrent futures via FuturesUnordered, capped at max_concurrent. Each future handles one context-peer sync session from handshake through completion.

-
-
5
-

handle_opened_stream()

-

When a remote peer opens a stream, reads the StreamMessage::Init frame, extracts the InitPayload, and dispatches to the appropriate sync protocol handler based on the negotiated protocol type.

-
-
-
- -

Stream Wire Types

-
-
-pub enum StreamMessage {
-    Init(InitPayload),
-    Message(MessagePayload),
-    OpaqueError,
-} -
-
-
-pub struct InitPayload {
-    context_id: ContextId,
-    protocol: SyncProtocol,
-    handshake: SyncHandshake,
-} -
-
-pub enum MessagePayload {
-    HashNodes(Vec<HashNode>),
-    LevelData(LevelBatch),
-    SnapshotChunk(PageData),
-    DeltaEntries(Vec<CausalDelta>),
-    BloomFilter(BloomData),
-    // DagHeadsResponse carries an optional governance scope root
-    // None = responder could not fold scope (non-group ctx or store fault)
-    DagHeadsResponse {
-        scope_root: Option<Hash>,
-    },
-} -
-
-
- -

Post-Sync Governance Reconciliation (handle_dag_sync)

-
-

Governance divergence check moved to handle_dag_sync

-

Governance reconciliation after a sync session is now owned exclusively by handle_dag_sync in the sync manager, not by individual protocol backends. After execute returns Ok(Some(_)) — meaning a data backend actually ran (Hash Comparison, LevelWise, Snapshot, Delta Sync, or any future backend) — the manager re-reads the post-sync local scope_root and compares it byte-for-byte against the peer's handshake scope_root. If they differ, pull_namespace_governance is called immediately. A store fault on the post-sync root read causes the check to be skipped with a warn and deferred to the next tick.

-

Ok(None) is excluded: when execute returns Ok(None) (i.e., SyncProtocol::None — entities were already in sync), the post-sync check does not run, avoiding a redundant store read and scope_root fold on every converged tick. The pre-sync GovDiverged path (described in the ProtocolDispatch section) handles that case.

-

All backends covered: Previously, only Hash Comparison and LevelWise could trigger a governance pull (via a gov_divergence_detected flag on their stats structs). Snapshot and Delta Sync never triggered such a pull, leaving peers that converged entity state via those backends stuck with a diverged governance plane. The manager-level check fires after all data backends.

-

gov_divergence_detected: bool has been removed from both HashComparisonStats and LevelWiseStats. HC and LevelWise still compute the ScopeVerdict for root_hash_verified and the scope_root_governance_divergence warning log, but no longer propagate a divergence flag to the selector.

-
-// Post-sync check in handle_dag_sync — fires on Ok(Some(_)) only
-// if post_sync_scope_root != peer_handshake_scope_root {
-// let ops = self.pull_namespace_governance(context_id, peer).await;
-// info!(marker = "gov_divergence_pull_triggered", ops_pulled = ops, ...);
-// }

-// gov_divergence_detected removed from both stats structs
-pub struct HashComparisonStats {
-    // ... existing fields (no gov_divergence_detected) ...
-}

-pub struct LevelWiseStats {
-    // ... existing fields (no gov_divergence_detected) ...
-} -
-

Operators: The gov_divergence_pull_triggered info marker is now emitted by the sync manager after any data-backend sync session, not only HC/LevelWise. Any dashboards or alerts previously relying on gov_divergence_detected in protocol stats should be retired; watch the manager-level gov_divergence_pull_triggered and gov_divergence_pull_complete markers instead.

-
- -

pull_namespace_governance — inherent method & op-count return

-
-

SyncManager::pull_namespace_governance — inherent method, returns usize

-

pull_namespace_governance is an inherent async method on SyncManager. It is best-effort: resolves the namespace ID for the given context via namespace_sync::resolve_namespace_id, calls sync_namespace_from_peer targeting the specific peer, and returns the number of governance ops delivered. All early-return and error paths return 0.

-

pull_namespace_governance is no longer a method on the ProtocolDispatch trait. The trait method and its implementation have been removed. The manager calls the inherent method directly from handle_dag_sync (post-sync path) and from the pre-sync entities-agree path.

-

sync_namespace_from_peer and pull_namespace_governance previously returned Result<()>. They now return Result<usize> — the number of governance ops delivered by the backfill response. The SyncDriverDispatch::sync_namespace_from_peer wrapper discards the count with let _ops = ....

-

Pre-sync governance divergence (entities-agree path): When select_protocol returns SyncProtocol::None (entity roots agree, no entity walk needed) and the peer advertised a non-None scope_root, the sync manager computes a scope_verdict. If the verdict is ScopeVerdict::GovDiverged, it calls pull_namespace_governance and returns early, bypassing the entity walk. The gov_divergence_pull_complete debug log now includes an ops_pulled field. The store read for scope_verdict is gated on peer_scope_root being Some: when a peer sends scope_root: None the datastore is never touched.

-

Post-sync path (all backends): After execute returns Ok(Some(_)), handle_dag_sync re-reads the local scope_root, compares it to the peer's handshake scope_root, and calls pull_namespace_governance when they differ. The returned op count is included in the gov_divergence_pull_triggered info log. The pull is self-limiting: once governance converges, roots match and no further pull is issued. The warn! log in run_initiator_impl changed from "awaiting governance sync to propagate the rotation" to "pulling governance from the peer to propagate the rotation".

-
-// Inherent method — called by pre-sync and post-sync paths
-// Returns the number of governance ops delivered (0 on error/no-op)
-async fn SyncManager::pull_namespace_governance(
-    &self,
-    context_id: ContextId,
-    peer: PeerId,
-) -> Result<usize>

-// sync_namespace_from_peer now returns usize (op count)
-// SyncDriverDispatch wrapper discards with: let _ops = ...;

-// Pre-sync entities-agree path (SyncProtocol::None + peer scope_root Some)
-// if scope_verdict == ScopeVerdict::GovDiverged {
-// let ops = self.pull_namespace_governance(context_id, peer).await?;
-// debug!(marker = "gov_divergence_pull_complete", ops_pulled = ops, ...);
-// return; // bypass entity walk
-// }

-// Post-sync path in handle_dag_sync — fires on Ok(Some(_)) only
-// if post_sync_scope_root != peer_handshake_scope_root {
-// let ops = self.pull_namespace_governance(context_id, peer).await?;
-// info!(marker = "gov_divergence_pull_triggered", ops_pulled = ops, ...);
-// } -
-

Operators: The gov_divergence_pull_triggered info marker (post-sync) and gov_divergence_pull_complete debug marker (pre-sync) both now include an ops_pulled field. ops_pulled == 0 indicates a best-effort failure or an already-converged peer. Monitor both markers together to verify governance convergence. The pull_namespace_governance trait method on ProtocolDispatch no longer exists; any custom ProtocolDispatch implementations must remove it.

-
- -

scope_root Shadow Divergence

-
-

local_scope_root helper & scope_root_shadow_divergence marker

-

A new private helper local_scope_root looks up the group that owns a context and then calls ScopeProjections::group_scope_root_ephemeral to fold the current governance scope root. Returns None for non-group contexts or store faults. A TODO(perf, C1+) note marks the current full DAG walk per session as temporary — it should be replaced with a read from the maintained projection before C1 makes this path load-bearing.

-

DagHeadsResponse gains a scope_root: Option<Hash> field. The responder side in delta_request.rs and hash_comparison.rs calls local_scope_root and includes the result in every DagHeadsResponse sent. The deterministic sync simulator in hash_comparison_protocol.rs sends None because it has no governance projection.

-

query_peer_dag_state return type extended: The return type of query_peer_dag_state was extended from Option<(Hash, Vec<[u8; DIGEST_SIZE]>)> to Option<(Hash, Vec<[u8; DIGEST_SIZE]>, Option<Hash>)>. The scope_root field from DagHeadsResponse, previously matched as scope_root: _ (discarded), is now captured and threaded back to all callers. This allows the pre-sync entities-agree path to inspect the peer's governance scope root without an additional round-trip.

-

query_peer_current_root now returns (root_hash, Option<scope_root>) and is pub(crate) so the LevelWise module can call it directly, sharing a single implementation of the mid-session DagHeadsRequest/DagHeadsResponse round-trip. After the HC session completes, the convergence verdict (stats.root_hash_verified) on the re-query path now uses scope_root as the authoritative signal when both peers resolve one. Both scope roots are bound by destructuring (Some(local), Some(peer)); if either is None (cold projection or non-group context), the check falls back to bare entity-root comparison against the freshly re-queried peer root (not the stale handshake root). Transport faults on the re-query are logged at debug and fall back to the handshake root; an older peer returning Ok(None) falls back silently. This means two nodes can no longer declare convergence when they share identical entity data but have diverged governance state (e.g., after a hash-neutral membership rotation).

-

The LevelWise initiator follows the same re-query pattern: at end-of-session it calls query_peer_current_root to obtain the peer's current entity root plus scope_root, then applies the same scope_root-preferring convergence decision described above (falling back to the freshly re-queried entity root, not the stale handshake root).

-

The LevelWise responder loop now has an explicit InitPayload::DagHeadsRequest arm. When triggered, it re-reads the local entity root after all leaf/tombstone merges, folds the governance projection (local_scope_root), and replies with a DagHeadsResponse carrying both root_hash and scope_root. On a non-group or cold projection, scope_root is None, signalling the initiator to fall back to entity-root comparison. Previously this arm fell through to the catch-all _ that broke out of the loop.

-

The C0 observe-only block that emitted scope_root_shadow_divergence when entity roots agreed but scope roots differed — without acting on the divergence — has been removed. It is superseded by the authoritative scope_root convergence check and the new scope_root_governance_divergence warning.

-

A new WARN log with marker = "scope_root_governance_divergence" is emitted by both the Hash Comparison and LevelWise paths when: (1) both peers resolved a scope_root, (2) root_hash_verified is false, and (3) the bare entity roots agree. This distinguishes pure ACL/governance-plane divergence — where the entity tree-walk has nothing to reconcile — from a data-plane divergence. All other non-convergence cases emit the general warning with peer_hash taken from the re-queried root (replacing the old stale remote_hash).

-
-// Resolves the governance scope root for a context (SyncManager)
-// TODO(perf, C1+): replace full DAG walk with maintained-projection read
-fn local_scope_root(
-    store: &Store,
-    context_id: ContextId,
-) -> Option<Hash>

-// Returns entity root + optional governance scope root
-// pub(crate) — shared by Hash Comparison and LevelWise initiators
-pub(crate) async fn query_peer_current_root(
-    &self,
-    peer_id: PeerId,
-    context_id: ContextId,
-) -> Result<(Hash, Option<Hash>)>

-// query_peer_dag_state — extended return type (used in select_protocol)
-// Now returns Option<(Hash, Vec<[u8; DIGEST_SIZE]>, Option<Hash>)>
-// Third element is scope_root from DagHeadsResponse (previously discarded)
-// scope_root: None → peer is cold/non-group; skip scope_verdict check
-// scope_root: Some(h) → used by pre-sync GovDiverged detection

-// Authoritative convergence verdict (run_initiator_impl re-query path,
-// used by both Hash Comparison and LevelWise):
-// Both scope roots bound by destructuring (Some(local), Some(peer))
-// if both Some → require scope_root equality
-// if either None → fall back to freshly re-queried entity-root comparison
-// root_hash_verified = match (local_scope_root, peer_scope_root) {
-// (Some(l), Some(p)) => l == p,
-// _ => local_root_hash == peer_current_root, // re-queried, not stale
-// }

-// LevelWise responder: explicit DagHeadsRequest arm in run_responder_loop
-// Re-reads entity root post-merge, folds local_scope_root, replies with
-// DagHeadsResponse { root_hash, scope_root }. scope_root=None for non-group
-// or cold projection → initiator falls back to entity-root comparison.

-// WARN emitted (HC and LevelWise) when both scope_roots are Some,
-// root_hash_verified is false, but bare entity roots agree:
-// marker=scope_root_governance_divergence
-// entity_root=<8-byte hex>, local_scope=<8-byte hex>, peer_scope=<8-byte hex>
-// Other mismatches: general non-convergence warn with peer_hash from re-query -
-

Operators: The scope_root field is additive and optional — responders that cannot fold the scope send None, and the initiator falls back to bare entity-root comparison in that case, so the protocol is fully backward compatible. The C0 scope_root_shadow_divergence WARN marker is no longer emitted; update any log-aggregator rules or alerts that referenced it. Watch instead for the new scope_root_governance_divergence WARN marker on both Hash Comparison and LevelWise warn logs: a hit indicates real ACL/membership divergence on a context whose entity data is already in sync — these are awaiting governance sync propagation and require no entity-level reconciliation. Add a separate alert rule for this marker on LevelWise paths. Downstream code matching on MessagePayload::DagHeadsResponse must be updated to handle or ignore the scope_root field.

-
-
- - -
-

Transport

-

The sync transport layer wraps libp2p streams with framing, encryption, and authentication.

- -
-
-

SyncTransport

-

Wraps a raw libp2p stream with length-delimited framing. Each frame is a Borsh-serialized StreamMessage. Handles backpressure via Tokio's AsyncWrite flow control.

-
-pub struct SyncTransport {
-    framed: Framed<Stream, LengthCodec>,
-    encryption: EncryptionState,
-} -
-
-
-

EncryptionState

-

Manages per-stream encryption state. After the initial handshake, streams are encrypted with a shared key derived from the context's encryption domain. Supports both Encrypted and Plaintext modes (configurable per context).

-
-pub enum EncryptionState {
-    Plaintext,
-    Encrypted {
-        cipher: ChaCha20Poly1305,
-        nonce_counter: u64,
-    },
-} -
-
-
- -
-

Challenge Domain

-

Each sync stream authenticates with a challenge derived from the context's encryption domain. The initiating peer signs a challenge nonce, and the responder verifies membership before accepting the stream. This prevents unauthorized peers from syncing context data.

-
- -
-

--mock-tee flag (dev/test only) — ⚠ NOT FOR PRODUCTION

-

merod run gains a --mock-tee flag (also settable via the MEROD_MOCK_TEE environment variable). When active, the fleet-join and attest handlers produce and accept mock attestation quotes instead of requiring real TDX hardware. This is intended exclusively for local development and CI environments where TDX hardware is unavailable.

-

The flag is refused at startup (bail!) if the node's TeeConfig has real attestation configured (TeeConfig::has_real_attestation returns true). A loud tracing::warn is emitted on startup whenever mock-tee is active. An additional warning fires if a Phala KMS provider is configured but real attestation is disabled, indicating a likely misconfiguration.

-

The flag is threaded through NodeConfig::mock_tee → AdminState::mock_tee (via AdminState::new) and is never persisted to the node config file.

-
-// CLI / env
-// merod run --mock-tee | MEROD_MOCK_TEE=1 merod run

-// Guard — refuses mock-tee when real KMS attestation is configured
-impl TeeConfig {
-    pub fn has_real_attestation(&self) -> bool { /* ... */ }
-}

-// Propagation path (never persisted)
-RunCommand::mock_tee: bool
-  → NodeConfig::mock_tee: bool
-    → AdminState::mock_tee: bool  // via AdminState::new -
-

Operators: Never set --mock-tee or MEROD_MOCK_TEE in production deployments. The startup bail! guard prevents accidental use when TeeConfig::has_real_attestation is true, but nodes without a real KMS configured will not be protected by that guard — rely on deployment policy and the emitted startup warning instead.

-
- - - - - - - - SyncManager - open stream - - - - - Challenge Auth - sign + verify - - - - - Encryption - ChaCha20Poly1305 - - - - - Framed Stream - length-delimited - - - - - StreamMessage - Borsh serialized - - -
-
Transport framing
-
Encryption
-
Authentication
-
Wire format
-
-
- -
-
- - - diff --git a/architecture/crates/tools.html b/architecture/crates/tools.html deleted file mode 100644 index aa50a27f85..0000000000 --- a/architecture/crates/tools.html +++ /dev/null @@ -1,598 +0,0 @@ - - - - -Tools — Calimero Core Architecture - - - -
-
-
Home/System Overview/Tools
- -

Developer Tools

-

mero-abi (calimero-abi) + mero-sign + merodb — CLI utilities for ABI, signing, and database inspection

- - -

Overview

-

Three tools live in the tools/ directory. They are not part of the runtime — these are developer and operator utilities for working with WASM artifacts, bundle signing, and database debugging.

- -
-
3
CLI tools
-
tools/
directory
-
0
runtime deps
-
clap
CLI framework
-
- -
-

Tool Suite

-

Each tool is a standalone binary crate excluded from workspace versioning.

- -
-
-

calimero-abi

- mero-abi -

Extract, inspect, and validate WASM ABI artifacts. Reads abi.json produced at build time, inspects WASM binary structure via wasmparser.

-
-
-

mero-sign

- CLI -

Ed25519 bundle manifest signing with did:key identifiers. Uses RFC 8785 JSON Canonicalization Scheme for deterministic signing.

-
-
-

merodb

- CLI -

RocksDB debug, inspect, export, and migrate tool. YAML-driven migration plans, DAG export, column-level data inspection, optional GUI.

-
-
-
- - -
-

calimero-abi (mero-abi)

-

CLI for working with Calimero WASM ABI artifacts. Reads abi.json produced at build time, inspects WASM binary structure via wasmparser.

- -

CLI Commands

-
-
extract WASM_FILE [-o OUTPUT] [--verify]
-
Copies ABI from resolved abi.json to output file (default: same basename .abi.json). --verify kept for API compat but unused.
-
-
-
types WASM_FILE [-o OUTPUT]
-
Writes only the types field from abi.json (default: .types.json).
-
-
-
state WASM_FILE [-o OUTPUT]
-
Parses manifest, extracts state schema, sets schema_version to "wasm-abi/1", writes JSON.
-
-
-
inspect WASM_FILE
-
Parses WASM sections, export count, checks for get_abi_ptr / get_abi_len / get_abi exports, custom sections, calimero_abi_v1 presence.
-
- -

Key Files

-
-
-

extract.rs

-

Core extraction logic — find_and_read_abi_json, extract_abi, extract_types_schema, extract_state_schema.

-
-
-

inspect.rs

-

WASM binary parser — inspect_wasm walks sections, identifies exports and custom sections.

-
-
- -
- Extract Flow -
-
1
-

Locate abi.json

-

find_and_read_abi_json resolves the ABI file path relative to the WASM file, reading the JSON into memory.

-
-
2
-

Parse & Validate

-

Deserialize into ABI structure via serde_json. The manifest includes method signatures, types, and state schema.

-
-
3
-

Write Output

-

Depending on subcommand, write the full ABI, types-only, or state schema to the output path. State output includes schema_version: "wasm-abi/1".

-
-
-
- -
- wasmparser 0.118 - clap - serde_json - eyre - calimero-wasm-abi -
- -tools/calimero-abi -
- - -
-

mero-sign

-

Sign Calimero bundle manifest.json files with Ed25519. Uses RFC 8785 JSON Canonicalization Scheme (JCS) via serde_json_canonicalizer. Signer IDs are did:key identifiers derived from the public key (multicodec 0xed01 + base58btc multibase z).

- -

CLI Commands

-
-
sign manifest -k KEY
-
Load key file, set signerId, strip signature and _-prefixed fields from canonicalization, SHA-256 of canonical bytes, Ed25519 sign, write signature object in-place.
-
-
-
generate-key [-o OUTPUT]
-
Generate new Ed25519 keypair. Writes JSON with private_key, public_key, signer_id.
-
-
-
derive-signer-id -k KEY
-
Print did:key signer_id from key file.
-
- -

Signing Flow

- - - - - - - - - manifest.json - input file - - - - - - Strip _fields - + signature field - - - - - - JCS - RFC 8785 canonicalize - - - - - - SHA-256 - digest - - - - - - Ed25519 - sign digest - - - - - - Write signature - back to manifest - - - manifest.json → strip _fields + signature → JCS canonicalize → SHA-256 → Ed25519 sign → write signature back - - - - key.json - Ed25519 private key - - - -

Key Types

-
-
-struct KeyFile {
-    private_key: String, // hex-encoded Ed25519 secret
-    public_key: String,  // hex-encoded Ed25519 public
-    signer_id: String,  // did:key multibase identifier
-} -
-
-struct SignatureObject {
-    algorithm: String,  // "Ed25519"
-    publicKey: String,  // hex-encoded public key
-    signature: String,  // base64url-encoded signature
-} -
-
- -
- did:key Derivation -
-
1
-

Multicodec Prefix

-

Prepend the Ed25519 multicodec prefix 0xed01 to the raw 32-byte public key.

-
-
2
-

Base58btc Encode

-

Encode the prefixed bytes using base58btc (Bitcoin alphabet).

-
-
3
-

Multibase Wrap

-

Prepend multibase prefix z → final identifier: did:key:z6Mk...

-
-
-
- -
- ed25519-dalek - clap - serde_json - serde_json_canonicalizer - bs58 - base64 - sha2 -
- -tools/mero-sign -
- - -
-

merodb

-

Debug and inspect Calimero's RocksDB layout. Export column data, validate integrity, export DAG, run YAML-driven migrations, optional GUI.

- -

CLI Commands

-
-
schema [-o FILE]
-
Emit JSON schema of DB structure (stdout if no -o).
-
-
-
export --db-path PATH [--all | --columns COLS] [--state-schema-file SCHEMA] [-o FILE]
-
Opens DB read-only, exports selected columns. State decoding uses schema file or infers from DB.
-
-
-
validate --db-path PATH [-o FILE]
-
Validate database integrity — checks column families, key structure, value deserialization.
-
-
-
export-dag --db-path PATH [-o FILE]
-
Export context DAG as JSON — traverses the Delta column to reconstruct the full DAG structure.
-
-
-
migrate --plan PLAN --db-path PATH [--target-db] [--backup-dir] [--dry-run] [--apply] [--report FILE]
-
YAML migration plan with copy / delete / upsert / verify steps. Default is dry-run; --apply writes.
-
-
-
gui [--port 8080]
-
Local HTTP UI for DB exploration (requires --features gui).
-
- -

Column Support

- - - - - - - - - RocksDB Column Families - - - - Meta - context metadata - parse_context_meta - - - Config - node configuration - parse_config - - - Identity - key identities - parse_identity - - - State - application state - parse_state - - - Delta - DAG deltas - parse_dag_delta - - - - Blobs - binary blobs - parse_blob - - - Application - app metadata - parse_application - - - Alias - name lookups - parse_alias - - - Generic - catch-all - parse_generic - - - Key/value parsing dispatched per column family — each column has a typed parser function - - -
-
Core columns (Meta / Config / Identity)
-
State columns (State / Delta)
-
Data columns (Blobs / Application)
-
Auxiliary columns (Alias / Generic)
-
- -

Key Functions

-
-
-

open_database

-

Opens RocksDB in read-only mode, discovers column families, returns CF handles for selective access.

-
-
-

parse_key / parse_value

-

Decode raw byte keys to JSON. parse_value dispatches to typed parsers like parse_context_meta, parse_dag_delta, etc.

-
-
- -
- Migration Plan Format -
-
-# migration.yaml
-version: 1
-steps:
-  - action: copy
-    from_column: State
-    to_column: State_v2
-    transform: null
-  - action: delete
-    column: State
-    key_prefix: "0x..."
-  - action: upsert
-    column: Meta
-    key: "version"
-    value: "2"
-  - action: verify
-    column: Meta
-    expect_key: "version" -
-

Default mode is --dry-run — logs actions without writing. Use --apply to execute. --backup-dir creates a snapshot before applying. --report writes a JSON summary.

-
-
- -
- rocksdb - clap - serde_json - serde_yaml - borsh - sha2 - jaq-* - calimero-primitives - calimero-storage - calimero-store - calimero-wasm-abi -
- -tools/merodb -
- - -
-

Dependencies

-

All three tools are excluded from workspace versioning. They depend on shared Calimero crates but do not participate in the runtime dependency graph.

- - - - - - - - - - calimero-abi - ABI extraction & inspection - - - mero-sign - bundle manifest signing - - - merodb - DB debug & migration - - - - clap - - - serde_json - - - sha2 - - - - wasmparser - - - ed25519-dalek - - - rocksdb - - - - calimero-wasm-abi - - - calimero-primitives - - - calimero-store - - - calimero-storage - - - - - - - - - - - - - - - - - - - - - - - - - - All tools excluded from workspace versioning — standalone binaries with shared crate dependencies - - -
-
Primary / unique dependency
-
Shared external dependency
-
calimero-primitives
-
calimero-store
-
calimero-storage
-
-
- -

merod

-

Node daemon binary. Initializes configuration, manages the datastore, and runs all actors (Node, Context, Network, Server).

- -
-

Commands

-
-merod init — Initialize node: config.toml, identity, datastore -merod config — Modify config.toml fields (dot-path syntax) -merod run — Start the node (all actors, server, P2P) -
- -
-Init options -
-
-merod --home /data --node mynode init \ - --server-port 2428 \ - --swarm-port 2528 \ - --boot-network calimero-dev \ - --group-governance local -
-

--group-governance local is the default (and only) governance mode.

-

For read-only nodes, add --mode read-only.

-
-
-
- -
-

TEE Mode

-

When [tee] configuration is present, merod fetches a storage encryption key from a KMS at startup using TDX attestation. See TEE Mode for full details.

-

Note: The NEAR_API_KEY environment variable is no longer referenced by merod and should be removed from any runbooks or environment configurations.

- -

--mock-tee DEV/TEST ONLY

-

merod run accepts a --mock-tee flag (also settable via the MEROD_MOCK_TEE environment variable). When active, the fleet-join and attest handlers produce and accept mock attestation quotes instead of requiring real TDX hardware.

-
    -
  • Refused at startup if TeeConfig::has_real_attestation returns true — i.e. the node's [tee] config has real KMS attestation configured. The node will bail! rather than start in an insecure state.
    • -
  • A loud tracing::warn is emitted on startup whenever mock-tee is active. An additional warning fires if a Phala KMS provider is configured but real attestation is disabled, flagging likely misconfiguration.
    • -
  • The flag is threaded through RunCommand::mock_tee → NodeConfig::mock_tee → AdminState::mock_tee (via AdminState::new) and is never persisted to config.toml.
    • -
  • TeeConfig::has_real_attestation is the predicate used as the deny-list guard — it returns true when the TEE configuration references real attestation infrastructure.
    • -
-

⚠ Must never be used in production. This flag exists solely for development and testing on machines without TDX hardware.

-
- -

meroctl

-

Operator CLI for managing a running merod node. Connects via the admin API.

- -
-

Command Groups

-
-
-

app

-

install, list, get — manage applications (bundles and raw WASM).

-
-
-

context

-

create, list, update, delete — context lifecycle. Join contexts via group join-context. The create command no longer accepts a --protocol flag.

-
-
-

group

-

create (with --parent-group-id), list, get, delete, invite, join, members (add/remove/list/set-role/set-caps/get-caps), contexts, join-context.

-
-
-

call

-

Execute a WASM method: meroctl call METHOD --context ID --as KEY --args JSON

-
-
-

blob

-

upload, get, delete — binary large object management.

-
-
-

peers

-

List connected P2P peers.

-
-
-

node

-

Node identity and status information.

-
-
-
- -
-

Usage Example

-
-# Install an application bundle -meroctl --node mynode app install --path app.mpk - -# Create a context (--protocol flag has been removed) -meroctl --node mynode context create --application-id APP_ID - -# Call a method -meroctl --node mynode call set_item \ - --context CTX_ID --as MEMBER_KEY \ - --args '{"key": "hello", "value": "world"}' -
-

The context client-keys output table contains Signing Key and Created At columns only — the Wallet Type column has been removed.

-
- -
-
- - - diff --git a/architecture/dependency-explorer.html b/architecture/dependency-explorer.html deleted file mode 100644 index caca3cefed..0000000000 --- a/architecture/dependency-explorer.html +++ /dev/null @@ -1,465 +0,0 @@ - - - - -Dependency Explorer — Calimero Core Architecture - - - - - -
-
-
Home/Dependency Explorer
- -

Dependency Explorer

-

Interactive force-directed graph of all workspace crate dependencies

- -
- - -
- -
-
cli
-
core
-
types
-
data
-
foundation
-
wasm
-
- - - -
- -
- -
-
- - - - - diff --git a/architecture/error-flows.html b/architecture/error-flows.html deleted file mode 100644 index b8619bb539..0000000000 --- a/architecture/error-flows.html +++ /dev/null @@ -1,706 +0,0 @@ - - - - -Error Flows — Calimero Core Architecture - - - - -
-
-
Home/Error Flows
- -

Error Flows

-

What happens when things go wrong

- - -
-
-

1. Network Partition

- network - dag-merge -
-

Two groups of nodes lose connectivity. Each partition continues applying local ops independently. On reconnection the system automatically detects divergence and converges.

- - - - - Partition A - - Node 1 - - Node 2 - ops A1 → A2 → A3 - - - - Partition B - - Node 3 - - Node 4 - ops B1 → B2 - - - - PARTITION - - - - - - reconnect - - - - Heartbeat: divergent dag_heads - - - - - - - SyncManager catchup - - - - DAG Merge: Noop op reconciles heads - A3 + B2 → M (both partitions converge) - - - - - -
-
-

Trigger

-

Network connectivity lost between two subsets of group members. Each subset can still communicate internally.

-
-
-

During Partition

-

Each partition continues signing and applying SignedGroupOps locally. Ops have different parent hashes since they diverged from a shared ancestor.

-
-
-

Detection

-

On reconnection, periodic GroupStateHeartbeat messages reveal divergent dag_heads and/or differing root_hash. SyncManager triggers a catchup stream.

-
-
-

Recovery

-

Missing ops are exchanged via GroupDeltaRequest/GroupDeltaResponse. A merge Noop op with multiple parents reconciles the DAG heads. All nodes converge to identical state.

-
-
- -
-

Config & Metrics

-
    -
  • heartbeat_interval — 30s default, controls detection latency
    • -
  • max_dag_heads — capped at 64, triggers merge when exceeded
    • -
  • sync_catchup_ops — counter metric for ops transferred during catchup
    • -
-
-
- - -
-
-

2. Invalid Signature

- crypto - gossip -
-

A remote node receives a SignedGroupOpV1 whose Ed25519 signature does not verify. The message is silently discarded with no state change on the receiver.

- - - - - Malicious / Buggy - invalid signature - - - - - SignedGroupOpV1 - - - - Receiving Node - verify_signature() - ✗ FAIL - - - - - Silently rejected - no apply · no forward - - - - gossipsub handles sender reputation scoring — no explicit penalty in governance layer - - -
-
-

Trigger

-

Receive a SignedGroupOpV1 via gossipsub where verify_signature() returns Err. Could be tampered payload, wrong key, or serialization corruption.

-
-
-

What Happens

-

Op is rejected immediately. Not applied to GroupStore, not appended to DagStore, not forwarded to other peers. No state mutation on the receiving node whatsoever.

-
-
-

Recovery

-

No recovery needed — the system is already in a correct state. gossipsub's peer scoring mechanism may lower the sender's reputation if this happens repeatedly.

-
-
-

Design Rationale

-

Silent rejection is deliberate: no error response leaks information to potential attackers. The signature check is the first gate — all other validation (nonce, state_hash, auth) happens after.

-
-
-
- - -
-
-

3. Stale state_hash (Optimistic Lock)

- state - concurrency -
-

Two admins sign operations against the same state_hash concurrently. The first applied successfully mutates the hash, causing the second to fail validation. This prevents silent divergence.

- - - - - state_hash = 0xABC… - - - - Admin A - signs against 0xABC - - - - Admin B - signs against 0xABC - - - - - - - - - - - - - - ✓ Applied successfully - state_hash → 0xDEF… - - - - ✗ state_hash mismatch - 0xABC ≠ 0xDEF - - - - - - Re-read state → re-sign → retry - - - - If state_hash is all-zeros → validation skipped (explicit bypass) - - -
-
-

Trigger

-

Two admins concurrently read the same group state, both sign ops with state_hash = current_hash, and submit them roughly simultaneously.

-
-
-

First Op Wins

-

The first op to reach apply_local_signed_group_op passes validation, mutates the group state, and updates state_hash to a new value.

-
-
-

Second Op Rejected

-

The second op's state_hash no longer matches. apply_local_signed_group_op returns an error. The op is never applied or gossipped.

-
-
-

Recovery

-

The rejected admin re-reads current state (including the other admin's changes), re-signs with the updated state_hash, and resubmits. If state_hash is set to all-zeros, the check is explicitly bypassed.

-
-
- -
-

Config & Metrics

-
    -
  • state_hash_conflicts — counter metric for rejected ops due to stale state_hash
    • -
  • [u8; 32]::default() — all-zeros hash signals explicit bypass (used for Noop, bootstrap)
    • -
-
-
- - -
-
-

4. Out-of-Order Op Delivery

- dag - gossip -
-

A node receives a SignedGroupOp whose parent_op_hashes reference operations not yet seen locally. The op enters a pending queue until parents arrive.

- - - - - time → - - - - Op C received - parent: B (unknown!) - - - - - → DagStore pending queue - - - - Op A received - parent: genesis ✓ - - - ✓ applied - - - - Op B received - parent: A ✓ - - - ✓ applied - - - - - B arrives → unblocks C from pending - topological order: A → B → C applied - - - - - - -
-
-

Trigger

-

gossipsub delivers ops non-deterministically. A child op arrives before its parents. The node cannot apply it because the parent hashes are not in the local DagStore.

-
-
-

Pending Queue

-

The op enters DagStore's pending queue, indexed by its missing parent hashes. The op is held but not discarded.

-
-
-

Parent Arrival

-

When missing parents arrive (via gossip or catchup stream), the DagStore checks the pending queue for any ops now unblocked. These are applied in topological order.

-
-
-

Abuse Prevention

-

parent_op_hashes is capped at 256 entries per op. Ops exceeding this are rejected. Stale pending ops are cleaned up periodically.

-
-
- -
-

Config & Metrics

-
    -
  • MAX_PARENT_OP_HASHES — 256, hard cap per SignedGroupOp
    • -
  • pending_ops_count — gauge metric for ops awaiting parents
    • -
  • pending_ops_resolved — counter for ops successfully unblocked
    • -
-
-
- - -
-
-

5. WASM Out-of-Memory

- wasm - runtime -
-

The WASM runtime enforces VMLimits (max memory pages, stack size). If a WASM module exceeds these limits, execution is trapped and no state delta is produced.

- - - - - WASM Module - memory.grow() - - - - - - - VMLimits exceeded - max_memory_pages - max_stack_size - - - - - - - Wasmer trap - panic caught - - - - - - - VMLogic - Outcome { error } - - - - ✗ No state delta produced - - - ✗ No gossip broadcast - - - ✓ Error returned to caller - - - - Storage layer is never touched — no partial writes, no rollback needed. WASM sandbox provides clean isolation. - - -
-
-

Trigger

-

WASM module calls memory.grow() beyond VMLimits.max_memory_pages, or stack depth exceeds max_stack_size, or fuel runs out.

-
-
-

Trap & Catch

-

Wasmer's runtime traps with a panic. VMLogic catches the panic at the host boundary and converts it into a structured Outcome with an error variant.

-
-
-

No Side Effects

-

No state delta is produced. No changeset is committed to storage. No gossip message is broadcast. The execution is reported as failed in the Outcome returned to the caller.

-
-
-

Recovery

-

Caller receives the error and can retry with smaller input, or the WASM application can be upgraded to use less memory. Node state is completely unaffected.

-
-
- -
-

Config & Metrics

-
    -
  • VMLimits.max_memory_pages — configurable per-context memory ceiling
    • -
  • VMLimits.max_stack_size — stack depth limit
    • -
  • wasm_oom_traps — counter metric for OOM trap events
    • -
  • wasm_execution_failures — counter for all execution failures
    • -
-
-
- - -
-
-

6. Missing DAG Parents (State Delta)

- dag - sync -
-

A node receives a StateDelta whose parent_ids reference deltas not yet in the local DeltaStore. The delta enters a pending queue until parents are fetched.

- - - - - StateDelta received - parent_ids: [X, Y] - - - - - - - DeltaStore lookup - parent X = ✗ unknown - - - - - - - → DeltaStore pending - - - - NodeManager - DeltaRequest { delta_id: X } - - - - - - - - - Parent X arrives → unblock - - - - - Pending deltas applied in topological order - - -
-
-

Trigger

-

A StateDelta arrives via gossip with parent_ids referencing deltas the node has not yet received or applied.

-
-
-

Pending Queue

-

Delta enters DeltaStore pending, indexed by missing parent IDs. The delta is stored but not applied.

-
-
-

Fetch Missing

-

NodeManager sends a DeltaRequest stream to peers who likely have the missing parent. Peers respond with the missing delta payload. If the initial peer does not resolve every missing parent, SyncManager iterates additional mesh peers for the context up to parent_pull_additional_peers, bounded by parent_pull_budget; unresolved pending parents after the budget surface as a sync-session error so callers (e.g. join_context) fail loud rather than reporting success on a partial DAG.

-

The same pattern applies to governance DAG ops. When a NamespaceGovernanceDelta arrives via gossip with missing parents, the node immediately opens a NamespaceBackfillRequest stream to the gossip source (rather than waiting for the periodic NamespaceStateHeartbeat cycle) and falls through to the same cross-peer iteration if the first peer cannot drain the pending chain.

-
-
-

Resolution

-

When all parents arrive, pending deltas are drained in topological order. Stale pending deltas that are never resolved are cleaned up periodically.

-
-
- -
-

Config & Metrics

-
    -
  • pending_deltas_count — gauge for deltas awaiting parents
    • -
  • delta_request_sent — counter for DeltaRequest messages emitted
    • -
  • stale_pending_cleanup_interval — periodic cleanup timer
    • -
  • parent_pull_additional_peers — max additional mesh peers tried after the initial sync peer (default 3, applies to data-delta and namespace governance parent pulls)
    • -
  • parent_pull_budget — wall-clock budget for the cross-peer parent-pull loop (default 10s)
    • -
-
-
- - -
-
-

7. Snapshot Sync Failure

- sync - stream -
-

During a snapshot transfer the connection drops. The DeltaBuffer may have accumulated deltas during the transfer. On retry, the system determines whether partial state is reusable or a fresh snapshot is needed.

- - - - - Source Node - snapshot sender - - - - SnapshotStream - - - - - - - - Joining Node - partial snapshot state - - - - DeltaBuffer (accumulating) - deltas arriving during transfer - - - - - Connection drops → retry - - - - New handshake: is partial state usable? - compare root_hash of partial vs source's current state - - - - - - - - ✓ Resume from partial - apply buffered deltas - - - - ✗ Fresh snapshot required - discard partial, restart - - -
-
-

Trigger

-

TCP/QUIC connection drops during a SnapshotStreamRequest transfer. The joining node has received partial snapshot data.

-
-
-

DeltaBuffer

-

While the snapshot was transferring, new deltas continued arriving via gossip. These are held in a DeltaBuffer with finite capacity. If the buffer fills, oldest deltas are dropped (sync_buffer_drops metric).

-
-
-

Retry Handshake

-

On reconnection, a new handshake compares the partial state's root_hash against the source's current state. If compatible, the transfer can resume from where it left off.

-
-
-

Recovery

-

If partial state is usable: apply buffered deltas and continue. If not: discard partial state, request a fresh snapshot from scratch. Invariant I6: buffer has finite capacity, drops oldest if full.

-
-
- -
-

Config & Metrics

-
    -
  • sync_buffer_capacity — max deltas held during snapshot transfer
    • -
  • sync_buffer_drops — counter for deltas dropped due to buffer overflow
    • -
  • snapshot_retry_count — counter for snapshot transfer retries
    • -
  • snapshot_transfer_bytes — histogram of snapshot sizes
    • -
-
-
- - -
-
-

8. Cascade on Member Removal

- cascade - governance -
-

When a member is removed from a group, all their context identities and join tracking across every context in that group are cleaned up in one deterministic cascade operation.

- - - - - MemberRemoved - GroupOp applied - - - - - - - GroupStore scan - GroupMemberContext index lookup - - - - - - - - - - Context A - identity removed - - - - Context B - identity removed - - - - Context C - identity removed - - - - … Context N - identity removed - - - - Join tracking cleaned up for each context (GroupMemberContext rows deleted) - - - - Deterministic: same op on any node → identical cascade — verified by convergence tests - - -
-
-

Trigger

-

A MemberRemoved GroupOp is applied to the GroupStore. The member is removed from the group's member list.

-
-
-

Index Scan

-

GroupStore scans the GroupMemberContext index to find all contexts the removed member had joined via this group. This index is maintained automatically when members join contexts within the group.

-
-
-

Cascade Removal

-

For each found context: the member's context identity is removed, their join tracking row is deleted. If the member was in 100 contexts, all 100 get cleaned up atomically within the single op apply.

-
-
-

Convergence Guarantee

-

The cascade is fully deterministic — the same MemberRemoved op on any node produces identical deletions. This is verified by the convergence test suite (21 two-node tests).

-
-
- -
-

Config & Metrics

-
    -
  • cascade_contexts_removed — counter for context memberships cleaned up per MemberRemoved op
    • -
  • cascade_duration_ms — histogram for time spent in cascade cleanup
    • -
  • No upper limit on contexts per member — cascade is O(n) in number of contexts the member joined
    • -
-
-
- -
-
- - - diff --git a/architecture/example-chat.html b/architecture/example-chat.html deleted file mode 100644 index 9f8deed1b3..0000000000 --- a/architecture/example-chat.html +++ /dev/null @@ -1,156 +0,0 @@ - - - - -Example: Chat App — Calimero Core Architecture - - - -
-
-
Home/Example: Chat App
-

Example: Chat App

-

Slack-like messaging with subgroups, ReadOnly channels, and cross-context mentions

- -
-

Group Hierarchy

-

Org tree for Acme Corp. Subgroups nest under the root; each group owns its own contexts and governance topic.

-
-Acme Corp root
-├── Engineering
-│   ├── Platform Team
-│   └── Frontend Team
-├── Marketing
-└── Leadership
-
-

Membership flows downward: anyone in a parent group is treated as a member of descendant groups (with direct child membership overriding inheritance where applicable).

-

Admin authority is structural, not access-based. A root admin can create or delete subgroups, manage members at the org level, and perform other governance ops on the tree — but to access contexts in a child subgroup, they must be a member of that subgroup. Private channels are placed in dedicated subgroups with restricted membership.

-
-
-

Flows down

-

Member of Acme Corp is a member of Engineering, Platform Team, and Marketing. With auto_join (default), all contexts in those groups are accessible.

-
-
-

Restricting access

-

Private channels live in dedicated subgroups with restricted membership. Only members explicitly added to the subgroup can access its contexts.

-
-
-
- -
-

Context Layout

-

Any group member can join contexts in that group. ReadOnly role: can join contexts and read, but mutations are rejected by the platform — ideal for company-wide announcements and auditors. Private channels are placed in dedicated subgroups with restricted membership.

- -

Root (Acme Corp)

-
profiles
auto_join — DMs, @mentions, notification sink
-
general
Group context — public lobby
-
announcements
Owned by Leadership subgroup; company-wide ReadOnly access granted at root level
- -

Engineering

-
eng-general
Group context
-
incidents
Group context
-
eng-hiring
In a dedicated subgroup — eng leads only (subgroup membership)
- -

Platform Team

-
platform
Group context
-
on-call
In a dedicated subgroup — paging roster
- -

Frontend Team

-
fe-general
Group context — design + UI discussion
-
design-crit
In a dedicated subgroup — invite-only critique
- -

Marketing

-
campaigns
Group context
-
brand-review
In a dedicated subgroup — brand council
- -

For announcements, the context lives in the Leadership subgroup where members have full write access. The root group grants ReadOnly role so the whole org sees updates without chat noise or edits.

-
-Cross-context mentions -
-

When a user types @alice in eng-general, the chat WASM can issue an xcall into the profiles context (same org, Open with auto_join) to increment Alice’s mention_counts counter. Alice’s client subscribes to profiles for notification badges without exposing other Restricted rooms.

-
-
-
- -
-

WASM State Design

-

Chat app state is CRDT-backed so offline edits merge cleanly. Types mirror Calimero’s WASM data primitives.

-
-
-

ChatChannel

-
-struct ChatChannel {
-    messages: Vector<Message>,
-    topic: LwwRegister<String>,
-    pinned: UnorderedMap<String, LwwRegister<bool>>,
-    reactions: UnorderedMap<String, UnorderedMap<String, Counter>>,
-}
-
-
-
-

Message

-
-struct Message {
-    id: String,
-    author: String,
-    content: String,
-    timestamp: u64,
-}
-
-

UserProfiles (profiles context)

-
-struct UserProfiles {
-    profiles: UnorderedMap<String, LwwRegister<String>>,
-    status: UnorderedMap<String, LwwRegister<String>>,
-    mention_counts: UnorderedMap<String, Counter>,
-}
-
-
-
-
    -
  • Vector for messages — ordered log; concurrent sends get deterministic merge.
    • -
  • LwwRegister for topic and pinned flags — single winner per key; last writer wins with tie-break.
    • -
  • Counter for reactions and mention_counts — additive; safe concurrent increments from many clients.
    • -
  • UnorderedMap for keyed maps — per-message reaction bags and profile fields merge by key.
    • -
-
-Why not a CRDT for message text edits? -
-

This sketch treats each Message as immutable once assigned an id. Edits could be modeled as new tombstone-aware entries or a separate LwwRegister per message id if the product needs Slack-style “edited” labels.

-
-
-
- -
-

Permission Matrix

-

How five personas interact with the Acme tree. Structural power ≠ read access to child Restricted contexts.

-
CEO
Acme Corp Admin — structural control everywhere (subgroups, org members). Cannot access eng-hiring or other private subgroup contexts without being added to those subgroups.
-
VP Engineering
Engineering Admin — full governance under Engineering, Platform Team, Frontend Team; manages eng subgroup membership.
-
Senior engineer
Engineering Member + CAN_CREATE_CONTEXT + CAN_INVITE_MEMBERS — can spin up channels and invite, subject to group policy.
-
External auditor
Acme Corp ReadOnly — reads channels in groups they belong to; no writes.
-
Board observer
Leadership ReadOnly; added to specific subgroups for board-updates only — no automatic access to other Leadership channels.
-
- -
-

Platform Features Used

-
-

Subgroups

Org hierarchy under one root; membership inherited down the tree so engineers automatically exist in Platform / Frontend subgroups.

-

ReadOnly

Announcements and auditor feeds: join + read without mutating application state.

-

Capabilities

CAN_CREATE_CONTEXT lets trusted members create new channels without full admin.

-

Subgroup access control

Private channels live in dedicated subgroups with controlled membership. Group membership = context access.

-

xcall

Cross-context calls to bump mention_counts in profiles when someone is @mentioned from any channel.

-

CRDTs

Vector messages, Counter reactions & mentions, LwwRegister topic and pins.

-
-
- -
-

Privacy Boundary

-

The important invariant: the CEO can delete the Engineering subgroup but cannot access eng-hiring contexts. Access is enforced by subgroup membership; the CEO must be explicitly added to the eng-hiring subgroup to access its contexts.

-

If executive oversight is required, the CEO must be added to the relevant subgroup by its admin — or policy can require a separate compliance context with its own governance. The platform keeps structural hierarchy and confidentiality orthogonal by design.

-
- -
-
- - - diff --git a/architecture/example-docs.html b/architecture/example-docs.html deleted file mode 100644 index 1738f49d86..0000000000 --- a/architecture/example-docs.html +++ /dev/null @@ -1,182 +0,0 @@ - - - - -Example: Collaborative Docs — Calimero Core Architecture - - - -
-
-
Home/Example: Collaborative Docs
-

Example: Collaborative Docs

-

Google Docs-style editing with RGA, subgroups, ReadOnly viewers, and cross-context indexing

- -
-

Primary state

Each document context owns CollabDoc in WASM; peers sync via CRDT deltas over the normal context channel.

-

Secondary index

doc-index is a sibling context: searchable metadata and activity feeds without coupling security domains.

-
- -
-

Group hierarchy

-
-Acme Workspace // root
-├── Engineering Docs → Architecture RFCs
-├── Marketing Docs
-├── HR & People
-└── Company Handbook -
-

Each document is its own context (WASM app + CRDT state). Subgroups segment collections without merging doc state. Sensitive folders sit under HR as a dedicated subgroup: HR admins control membership; the CEO keeps org-wide structural powers but cannot access those doc contexts without being added to the HR subgroup.

-
    -
  • One context per doc — replication, permissions, and WASM bytecode are scoped per document; moving a doc between folders is a registry update in doc-index, not a state merge.
    • -
  • Nested groups — Engineering Docs and Architecture RFCs are separate groups linked under Acme; RFC admins need not be Marketing admins.
    • -
  • Policy isolation — Subgroup membership on HR contexts enforces confidentiality even when the CEO is root-group admin elsewhere.
    • -
-
-Why not one giant doc context? -
-

Sharding by document caps sync volume, isolates failure blast radius, and lets each context pick its own visibility (Open handbook vs Restricted postmortem) without splitting CRDT streams manually.

-
-
-
- -
-

Context layout

-

Contexts are registered under their subgroup; group membership determines access and ReadOnly vs Member roles. The index context lives at the workspace root so every member can resolve “what exists” without opening each WASM state.

-

Registry and documents

-
Root · doc-index
auto_join — cross-workspace metadata registry (recent edits, titles, folder graph).
-
Marketing Docs · doc-brand-voice
Group members can join; marketing leads are Member.
-
Engineering Docs · doc-api-guide
Group members can join — shared API reference.
-
Engineering Docs · doc-postmortem-42
In a dedicated subgroup — incident responders only.
-
Architecture RFCs · doc-rfc-subgroups
Group members can join — design notes for nested groups.
-
Architecture RFCs · doc-rfc-security
In a dedicated subgroup — security reviewers only.
-
HR & People · doc-comp-bands
In HR subgroup — CEO explicitly not a subgroup member.
-
HR & People · doc-hiring-plan
In HR subgroup — recruiting + HR.
-
Company Handbook · doc-employee-handbook
Root group context; HR = Member (edits), all employees = ReadOnly (reads).
-
-Folder graph vs WASM -
-

The folders map in DocIndex mirrors the subgroup tree for UI navigation. Individual doc WASM remains unaware of folder moves until it receives an xcall or governance event updating its alias metadata.

-
-
-
- -
-

WASM state — RGA editing

-
-struct CollabDoc {
-  title: LwwRegister<String>,
-  content: ReplicatedGrowableArray, // character-level CRDT (RGA)
-  cursors: UnorderedMap<String, LwwRegister<CursorState>>,
-  comments: UnorderedMap<String, Vector<Comment>>,
-  edit_count: Counter,
-}

-struct DocIndex {
-  documents: UnorderedMap<ContextId, DocumentMeta>,
-  folders: UnorderedMap<FolderId, FolderEntry>,
-  recent_edits: Vector<EditEvent>,
-} -
-

What the index stores

-
    -
  • DocumentMeta — title snapshot, owning subgroup id, optional alias for deep links.
    • -
  • EditEvent — compact audit trail (actor, monotonic counter, HLC) for “activity” sidebars without shipping character payloads.
    • -
  • FolderEntry — ordered child ids so the SPA can render the same tree the governance graph exposes.
    • -
-

Character identity

-
-struct CharId {
-  hlc: HybridLogicalClock, // wall + lamport component
-  seq: u32, // per-replica disambiguator
-}
-struct RgaAtom { id: CharId, left: Option<CharId>, value: char, tombstone: bool } -
-

RGA. Each character carries a CharId (HLC timestamp + per-replica sequence), is positioned after an explicit left neighbor id, and deletions are tombstones (presence bit), so concurrent inserts at different anchors commute. Two users at the same insertion point both win: total order is deterministic from (left_neighbor, CharId).

-
-RGA merge intuition -
-

Replicas exchange ops; merging never drops acknowledged characters. Tombstones suppress deleted ids in the materialized string while preserving merge metadata for late joiners. The visible text is a fold over live atoms in id order after resolving the left-neighbor linked list.

-
-
-
-Cursor LwwRegister -
-

Per-user cursor anchors (line, column, or id-offset) use last-writer-wins so presence UI stays lightweight; the document truth remains the RGA, not the cursor map.

-
-
-
- -
-

Concurrent editing

-

Clients optimistically apply local keystrokes; the runtime merges remote ops into the same CRDT. The four panels below are the cases product teams care about when reasoning about conflicts.

-
-

Different positions

Edits target distinct left-neighbors; both apply independently. No user sees a forced rollback.

-

Same position

RGA orders concurrent inserts with CharId after the shared predecessor; list order is stable across peers.

-

Delete + insert overlap

Deletes mark tombstones; inserts relative to live or tombstone ids compose without losing either intent.

-

Offline editing

Each client buffers local ops; on reconnect CRDT merge reconciles divergent suffixes into one convergent document.

-
-

No OT-style transform locks: intent lives in CRDT ops, so merge is associative and commutative modulo causal delivery.

-
- -
-

Permission matrix

-

Roles stack: workspace membership grants a default lane; subgroup admin overrides inside that subtree; per-context allowlists carve exceptions for Restricted docs.

-
CEO
Structural control (subgroups, policies, org-wide Open contexts). No access to HR Restricted doc contexts or comp-bands allowlist.
-
VP Engineering
Subgroup admin over Engineering Docs + Architecture RFCs; manages eng doc visibility and invites.
-
Engineer
Member on eng tree + CAN_CREATE_CONTEXT for new docs.
-
HR director
Admin on HR & People subgroup; controls membership. CEO cannot join HR subgroup contexts without being added by an HR admin.
-
Employee
ReadOnly on Acme Workspace; reads all Open docs including handbook; no write on handbook.
-
External reviewer
ReadOnly on Architecture RFCs; added per-document or per-folder allowlists for sensitive RFCs.
-
-Handbook vs RFC access -
-

Employees see handbook text through ReadOnly membership on an Open context. Reviewers never get HR paths; they only receive scoped invites on RFC contexts that opt into external eyes.

-
-
-
- -
-

Cross-context communication

-

Doc WASM stays authoritative for body text; the index context aggregates discoverability. Use xcall so every edit updates the shared registry without sharing private state across Restricted boundaries. The index never stores full RGA payloads—only pointers and audit-friendly metadata.

-

Hooks

-
    -
  • insert_text() in a doc → xcall doc-index "record_edit" with redacted metadata (context id, timestamp, actor).
    • -
  • set_title() → xcall doc-index "update_document_title".
    • -
  • ReadOnly users file comments through a small proxy context (or capability-limited handler) that validates identity then forwards an xcall into the doc’s comment Vector.
    • -
-
-// Pseudocode: doc app handler
-fn after_local_insert(ctx: &ContextId, op: RgaOp) {
-  xcall(DOC_INDEX, "record_edit", EditReceipt { ctx, op_id: op.id, ts: now_hlc() });
-}

-fn on_title_committed(ctx: &ContextId, t: &str) {
-  xcall(DOC_INDEX, "update_document_title", TitleUpdate { ctx, title: t.to_string() });
-} -
-

Read path

-
    -
  1. Client resolves doc-index for titles, paths, and recency.
    2. -
  2. Opening a row triggers context join (requires group membership; auto_join handles this automatically).
    3. -
  3. Live typing subscribes to the target doc context; index rows update asynchronously via xcall receipts.
    4. -
-

Comment proxies validate ReadOnly membership on the handbook (or RFC) context, then perform a privileged append into comments so thread data stays CRDT-consistent with the body.

-
- -
-

Platform features used

-

End-to-end, this sample exercises the pieces that turn Calimero from “synced KV” into a credible docs surface: nested governance, visibility, CRDT-rich WASM state, and controlled cross-context calls.

-
-

RGA

Character-level real-time collaborative editing with causal, convergent merges; scales per document instead of one global sequence.

-

Subgroups

Workspace → doc collections → sub-collections without fusing WASM state; mirrors how companies carve ACLs today.

-

ReadOnly

Handbook readers and external reviewers; enforced by the platform role model so UI cannot accidentally grant write RPCs.

-

Privacy boundary

HR subgroup contexts stay opaque to execs not in the subgroup; metadata xcalls carry only agreed fields (no body text leakage).

-

xcall

doc-index updates, title sync, comment proxy indirection; keeps secondary indices consistent with primary doc state.

-

CRDTs

Counter (edits), LwwRegister (title, cursors), Vector (comments), UnorderedMap (per-user cursors).

-
-
- -
-
- - - diff --git a/architecture/getting-started.html b/architecture/getting-started.html deleted file mode 100644 index 36d76f622f..0000000000 --- a/architecture/getting-started.html +++ /dev/null @@ -1,66 +0,0 @@ - - - - -Getting Started — Calimero Core Architecture - - - -
-
-
Home/Getting Started
-

Getting Started

-

From zero to a running app in 5 minutes

- -
-

Quick path

-

Install merod, run a node, install the sample kv-store app, create a context, and invoke methods with meroctl.

- -
1

Install merod

-
cargo install --git https://github.com/calimero-network/core merod
-

Or download a prebuilt binary from GitHub releases.

- -
2

Initialize a node

-
merod --home ~/.calimero --node mynode init \
-  --server-port 2428 --swarm-port 2528
- -
3

Start the node

-
merod --home ~/.calimero --node mynode run
-

The node is listening. Admin API: http://localhost:2428/admin-api

- -
4

Build and install an app

-
cd apps/kv-store && ./build.sh
-meroctl --node mynode app install --path res/kv_store.wasm
-

Note the ApplicationId printed in the output.

- -
5

Create a namespace and context

-
meroctl --node mynode namespace create --application-id <APP_ID>
-meroctl --node mynode context create --application-id <APP_ID> --group-id <NAMESPACE_ID>
-

A namespace is the root group; context create requires --group-id (pass the NamespaceId). Save the ContextId and MemberPublicKey from the response.

- -
6

Call a method

-
-
meroctl --node mynode call set --context <CTX_ID> \
-  --args '{"key":"hello","value":"world"}'
-
meroctl --node mynode call get --context <CTX_ID> \
-  --args '{"key":"hello"}'
-
-
- -
-

What’s next

-
- -

Core Concepts

Understand groups, roles, and capabilities.

 - -

SDK Reference

Build your own WASM app on Calimero.

 - -

Example Apps

See full app designs (e.g. chat) end to end.

 -
-
- -
-
- - - diff --git a/architecture/glossary.html b/architecture/glossary.html deleted file mode 100644 index 93639d5484..0000000000 --- a/architecture/glossary.html +++ /dev/null @@ -1,946 +0,0 @@ - - - - -Glossary — Calimero Core Architecture - - - - -
-
-
Home/Glossary
- -

Glossary

-

Key terms and concepts with source links

- -
- -
-
- -
- - -
#
- -
-
.mpk (bundle)
-
- Signed application archive (tar.gz) containing manifest.json, app.wasm, and optional abi.json. The deployable unit for Calimero applications. -
-
- - -
A
- -
-
AppKey
-
- Composite application identity: {package}:{signerId}. Determines the ApplicationId. Must stay stable across versions for the upgrade path to work. -
-
- -
-
ApplicationId
-
- 32-byte hash identifying a WASM application. -
Defined in: crates/primitives/src/application.rs
-
-
- - -
B
- -
-
BlobId
-
- Content-addressed identifier for binary blobs. -
Defined in: crates/primitives/src/blobs.rs
-
-
- -
-
Borsh
-
- Binary Object Representation Serializer for Hashing. Deterministic binary serialization format used for state deltas, governance payloads, store entries, and wire protocol messages. -
-
- -
-
BroadcastMessage
-
- Borsh-serialized gossipsub payload enum (StateDelta, HashHeartbeat, SignedGroupOpV1, etc.). -
Defined in: crates/node/primitives/src/sync/snapshot.rs
-
-
- - -
C
- -
-
Capabilities
-
- Per-member u32 bitmask controlling group-level permissions (CAN_CREATE_CONTEXT, CAN_INVITE_MEMBERS, CAN_JOIN_OPEN_SUBGROUPS, MANAGE_MEMBERS, MANAGE_APPLICATION, CAN_CREATE_SUBGROUP, CAN_DELETE_SUBGROUP, CAN_MANAGE_VISIBILITY, CAN_MANAGE_METADATA). -
-
- -
-
CausalDelta
-
- A single DAG entry: delta_id, parents, payload, HLC, expected_root_hash. -
Defined in: crates/dag/src/lib.rs
-
-
- -
-
Column
-
- Logical partition in the KV store (Meta, Config, Identity, State, Delta, Blobs, Application, Alias, Generic, Group). -
Defined in: crates/store/src/db.rs
-
-
- -
-
Context
-
- An instance binding a Group to an Application. Has its own identity, state DAG, and member list. -
Defined in: crates/primitives/src/context.rs
-
-
- -
-
ContextClient
-
- Async facade over LazyRecipient<ContextMessage> for invoking context operations. -
Defined in: crates/context/primitives/src/client.rs
-
-
- -
-
ContextId
-
- 32-byte hash identifying a context. -
Defined in: crates/primitives/src/context.rs
-
-
- -
-
ContextManager
-
- Actix actor managing contexts, groups, governance DAGs, and application lifecycle. -
Defined in: crates/context/src/lib.rs
-
-
- -
-
ContextMessage
-
- Top-level message enum for ContextManager with 20+ variants. -
Defined in: crates/context/primitives/src/messages.rs
-
-
- -
-
CRDT
-
- Conflict-free Replicated Data Type. Data structures where replicas can be updated independently and merged to produce a consistent result regardless of message ordering. -
-
- - -
D
- -
-
DAG (Directed Acyclic Graph)
-
- Causal ordering structure used for both application state deltas and governance operations. -
Defined in: crates/dag/src/lib.rs
-
-
- -
-
DagStore
-
- In-memory causal DAG with pending queue, topological ordering, and head tracking. -
Defined in: crates/dag/src/lib.rs
-
-
- -
-
DeltaApplier
-
- Async trait for applying deltas to storage. Implemented by GroupGovernanceApplier. -
Defined in: crates/dag/src/lib.rs
-
-
- -
-
DeltaBuffer
-
- Temporary storage for state deltas that arrive before their context is initialized or while a sync session is active. -
Defined in: crates/node/primitives/src/delta_buffer.rs
-
-
- -
-
did:key
-
- DID method that encodes an Ed25519 public key directly in the identifier string. Format: did:key:z{base58btc(0xed01 || public_key)}. -
-
- - -
E
- -
-
Ed25519
-
- Elliptic-curve digital signature algorithm used for governance operations, bundle manifests, and challenge-response authentication. -
-
- - -
F
- -
-
Fleet node / fleet-join
-
- A TEE replica node entitled to serve a namespace as a hardware-attested, read-only replica. It joins via the attestation announce flow — broadcasting a TeeAttestationAnnounce on the namespace topic, getting verified against the namespace TeeAdmissionPolicy, and being admitted as a ReadOnlyTee member via MemberJoinedViaTeeAttestation. Fleet-join is the end-to-end admission of such a node; eviction triggers a self-purge. -
TEE Fleet HA
-
-
- - -
G
- -
-
G-Counter
-
- Grow-only Counter CRDT. Each node increments its own slot; value() returns the sum. Calimero's Counter type follows this pattern. -
-
- -
-
GarbageCollector
-
- Periodic actor that cleans up stale data (unused blobs, expired entries). Default interval: 12 hours. -
-
- -
-
gossipsub
-
- libp2p publish/subscribe protocol. Calimero uses context/<hex> topics for state deltas and group/<hex> topics for governance operations. -
-
- -
-
Governance epoch
-
- Reserved field on state deltas for future staleness checks. Will reject deltas from removed members. Currently stored but not enforced. -
-
- -
-
Namespace
-
- A root group (no parent) that serves as the application instance boundary and identity scope. Each namespace gets its own Ed25519 keypair (auto-generated on first create/join). Subgroups and contexts within the namespace share that identity. Different namespaces have different keys. All groups within a namespace share a single governance DAG. Operations are either cleartext RootOps (structural changes, key delivery) or encrypted GroupOps (membership, capabilities). New members receive group keys via ECDH-wrapped KeyDelivery. -
Store key: NamespaceIdentity (0x37) • API: GET/POST /admin-api/namespaces
-
-
- -
-
Service
-
- A named WASM module within a multi-service application bundle. Each service has its own bytecode and optional ABI. Contexts specify which service they run via service_name. Single-service bundles do not require a service name. -
Store type: ServiceMeta • Manifest field: services
-
-
- -
-
Group
-
- Governance unit within a namespace, owning one or more Contexts. Manages membership and capabilities via signed operations. Subgroups inherit the root group's application. Access control is group-membership-based. -
Defined in: crates/context/primitives/src/group.rs
-
-
- -
-
GroupGovernanceApplier
-
- DeltaApplier implementation bridging the DAG to GroupStore for governance ops. -
Defined in: crates/context/src/governance_dag.rs
-
-
- -
-
GroupGovernanceDelta
-
- Gossip message carrying a signed governance operation with explicit DAG metadata (delta_id, parent_ids) for replication and catch-up. -
-
- -
-
NamespaceGovernanceDelta
-
- Gossip message carrying a SignedNamespaceOp on namespace topics. Published when namespace governance operations are created. Receivers verify, apply to the namespace DAG, and may trigger key delivery. -
-
- -
-
SignedNamespaceOp
-
- A signed namespace governance operation containing a NamespaceOp (either a cleartext RootOp or an encrypted GroupOp). Signed with the node's namespace identity private key. Forms a single causal DAG per namespace. -
Defined in: crates/context/primitives/src/local_governance/mod.rs
-
-
- -
-
KeyDelivery
-
- A cleartext RootOp in the namespace governance DAG. Delivers an ECDH-wrapped group encryption key to a newly joined member. The sender wraps the key using ECDH(sender_sk, joiner_pk); the joiner unwraps using ECDH(joiner_sk, sender_pk). Delivers the per-group GroupKeyring key — not the node's KMS storage key (see the two key families entry). -
-
- -
-
GroupKeyring (group encryption keys)
-
- The per-group symmetric encryption keys used to encrypt/decrypt a group's GroupOps and shared state. Each new member receives them via ECDH-wrapped KeyDelivery, they are rotated when a member is removed, and (as of #2776) they are deleted from a fleet node's disk on self-purge. Distinct from the node's KMS storage key (see the two key families entry). -
-
- -
-
two key families (storage key vs. group keys)
-
- Calimero TEE deployments involve two unrelated key families that are commonly conflated: -

- (1) KMS storage / disk key — per-node, fetched from the KMS only after the node's TEE measurement (MRTD) is verified. Encrypts the node's on-disk datastore. One per node; never delivered over governance. -

- (2) Per-group GroupKeyring encryption keys — per-group, ECDH-delivered to members via KeyDelivery, rotated on member removal, and (as of #2776) deleted on self-purge. Encrypt group operations and shared state. See TEE Fleet HA. -

- Mnemonic: the KMS key protects this node's disk; the GroupKeyring keys protect a group's shared data across all members. -
-
- -
-
NamespaceIdentity
-
- A per-namespace Ed25519 keypair automatically generated on first create/join and persisted in the datastore (key prefix 0x37). Used for signing namespace governance ops and unwrapping group keys via ECDH. Shared across all subgroups and contexts in the namespace. -
Store key: NamespaceIdentity (0x37)
-
-
- -
-
TopicManager
-
- Deduplication-aware gossipsub subscription manager. Uses a RwLock<HashSet> to track active subscriptions and avoid redundant network calls. Supports namespace, group, and context topics. -
Defined in: crates/node/primitives/src/topic_manager.rs
-
-
- -
-
GroupMemberRole
-
- Enum distinguishing the four member roles within a group: Admin, Member, ReadOnly, and ReadOnlyTee. ReadOnly and ReadOnlyTee both bar state mutation, but ReadOnlyTee is reserved for hardware-attested fleet replica nodes (directly-rowed, never inherited). -
Defined in: crates/primitives/src/context.rs
-
-
- -
-
GroupMutationKind
-
- Lightweight notification enum broadcast after a governance op is applied. Used for real-time UI notifications. -
-
- -
-
GroupOp
-
- Non-exhaustive enum of governance operations (MemberAdded, ContextRegistered, etc.). 25+ variants. -
Defined in: crates/context/primitives/src/local_governance.rs
-
-
- -
-
GroupStateHeartbeat
-
- Periodic gossip message (default 30s) carrying DAG heads and member count. Peers compare heads and request missing ops on divergence. -
-
- -
-
GroupStore
-
- Authoritative persistence layer for group state. Handles apply, OpLog, nonces, cascades. -
Defined in: crates/context/src/group_store.rs
-
-
- - -
H
- -
-
HashHeartbeat
-
- Periodic gossip message carrying root_hash + dag_heads for divergence detection. Part of BroadcastMessage. -
-
- -
-
HybridTimestamp (HLC)
-
- Logical clock combining wall-clock time with a counter for causal ordering. -
Defined in: crates/storage/src/logical_clock.rs
-
-
- - -
I
- -
-
InitPayload
-
- Stream protocol init message enum (BlobShare, KeyShare, DeltaRequest, SnapshotStreamRequest, GroupDeltaRequest, etc.). -
Defined in: crates/node/primitives/src/sync/wire.rs
-
-
- - -
J
- -
-
JCS
-
- JSON Canonicalization Scheme (RFC 8785). Produces deterministic byte ordering for JSON. Used by mero-sign before hashing bundle manifests. -
-
- -
-
JSON-RPC
-
- JSON-RPC 2.0 protocol for external method calls on contexts. Clients POST to /jsonrpc with method name and params. -
-
- - -
K
- -
-
Kademlia
-
- Distributed hash table protocol from libp2p used for peer discovery, bootstrap routing, and record storage. -
-
- -
-
KMS
-
- Key Management Service. Releases the per-node storage / disk key to a TEE node only after verifying its MRTD measurement (see TEE Mode). This is distinct from per-group GroupKeyring encryption keys (see the two key families entry). Supported: mero-kms-phala for Phala Cloud. -
-
- - -
L
- -
-
LazyRecipient
-
- Deferred Actix actor address wrapper using Arc<OnceCell<Recipient>>. Breaks circular init dependencies. -
Defined in: crates/utils/actix/
-
-
- -
-
libp2p
-
- Modular P2P networking framework providing transport (TCP/QUIC), discovery (Kademlia, mDNS), messaging (gossipsub), and stream protocols. -
-
- -
-
LWW (Last-Write-Wins)
-
- Merge strategy where concurrent updates are resolved by keeping the latest timestamp. Used by LwwRegister and per-entry in UnorderedMap. -
-
- - -
M
- -
-
manifest
-
- JSON metadata inside an .mpk bundle: package, version, signerId, wasm/abi paths, migrations, and Ed25519 signature. -
-
- -
-
MemberCapabilities
-
- Struct holding the five capability bit constants for group-level authorization. -
-
- -
-
MemberJoinedViaTeeAttestation
-
- The governance operation a verifier publishes to admit a fleet node as a ReadOnlyTee member, after its TeeAttestationAnnounce has been validated against the namespace TeeAdmissionPolicy. It is the attestation-gated analogue of an ordinary member-add: it directly rows the node into the target namespace (never inherited) rather than delivering an invitation. -
TEE Fleet HA
-
-
- -
-
Mergeable
-
- Trait for CRDT merge operations. Implemented by Map, Set, LwwRegister. -
Defined in: crates/storage/
-
-
- -
-
mero-sign
-
- Tool for Ed25519 key generation and bundle manifest signing using JCS + SHA-256 + Ed25519. -
-
- -
-
meroctl
-
- Operator CLI for a running node. Command groups: app, context, group, call, blob, peers, node. -
-
- -
-
merod
-
- Calimero node daemon. Commands: init (create config), config (modify settings), run (start all actors and servers). -
-
- -
-
merodb
-
- Debug CLI for direct RocksDB inspection: schema, export, validate, DAG export, YAML migration plans. -
-
- -
-
Multiaddr
-
- libp2p address format encoding layered protocol stacks (e.g. /ip4/127.0.0.1/tcp/2428/p2p/12D3KooW...). -
-
- - -
N
- -
-
NetworkClient
-
- Async facade over LazyRecipient<NetworkMessage>. -
Defined in: crates/network/primitives/src/client.rs
-
-
- -
-
NetworkEvent
-
- Events from the libp2p swarm (ListeningOn, Subscribed, Message, StreamOpened, BlobRequested, etc.). -
Defined in: crates/network/primitives/src/messages.rs
-
-
- -
-
NetworkManager
-
- Actix actor wrapping the libp2p Swarm. -
Defined in: crates/network/src/lib.rs
-
-
- -
-
NodeClient
-
- Async facade over LazyRecipient<NodeMessage>. -
Defined in: crates/node/primitives/src/client.rs
-
-
- -
-
NodeManager
-
- Central Actix actor orchestrating network events, blob cache, heartbeats. -
Defined in: crates/node/src/lib.rs
-
-
- -
-
Nonce
-
- Cryptographic nonce for replay protection. Per-signer monotonic counter in governance ops. -
-
- - -
O
- -
-
OpLog
-
- Persistent log of applied SignedGroupOp entries, keyed by (group_id, sequence). -
Defined in: crates/store/src/key/group.rs
-
-
- -
-
OpHead
-
- Latest sequence number + current DAG heads for a group. -
Defined in: crates/store/src/key/group.rs
-
-
- - -
P
- -
-
PeerId
-
- libp2p peer identifier derived from the node's public key. Uniquely identifies a node in the P2P network. -
-
- -
-
PrivateState
-
- Node-local storage column for data that should not be synchronized across peers. -
-
- -
-
PublicKey
-
- Ed25519 public key used for identity and signing. -
Defined in: crates/primitives/src/identity.rs
-
-
- - -
Q
- -
-
QUIC
-
- UDP-based transport with built-in encryption and multiplexing. Calimero nodes listen on both TCP and QUIC. -
-
- -
-
quote hash (replay protection)
-
- A per-group dedup record of a TDX quote already used to admit a fleet node. The verifier keys on the quote's hash so the same quote cannot admit a node twice within the same group, even if re-broadcast. Combined with the per-quote nonce in report_data, it closes the replay window on the TeeAttestationAnnounce → MemberJoinedViaTeeAttestation path. -
TEE Fleet HA
-
-
- - -
R
- -
-
ReadOnly
-
- Group member role that allows joining contexts and reading state but prevents all state mutations. Enforced at both local and remote nodes. A regular (non-attested) role; contrast with ReadOnlyTee, the hardware-attested fleet-node variant. -
-
- -
-
ReadOnlyTee
-
- Read-only TEE fleet-node role. Like ReadOnly it bars all state mutation, but it is directly-rowed and NEVER inherited through subgroups: a fleet replica is only ReadOnlyTee in the exact namespace it was admitted to. Admission is gated on hardware TDX attestation (see TeeAttestationAnnounce / MemberJoinedViaTeeAttestation), and on eviction the node hard-purges its local rows and keys (see self-purge). A variant of GroupMemberRole. -
Defined in: crates/primitives/src/context.rs • TEE Fleet HA
-
-
- -
-
report_data
-
- The 64-byte free-form field bound into a TDX quote at generation time. For a fleet node's attestation it is nonce(32) || Sha256(pubkey): a fresh 32-byte nonce concatenated with the SHA-256 of the announcing node's public key. Binding the pubkey hash into the hardware-signed quote proves the quote belongs to that key; the nonce gives freshness for replay protection. -
-
- -
-
RGA (Replicated Growable Array)
-
- Sequence CRDT for character-level collaborative text editing with HLC-based CharIds and left-neighbor positioning. -
-
- -
-
RocksDB
-
- Embedded LSM-tree key-value store backing all node persistence. 11 column families accessed via calimero-store. -
-
- - -
S
- -
-
self-purge
-
- The role-scoped local cleanup a fleet node runs on eviction from ReadOnlyTee. It hard-deletes the node's own local rows and keys for the namespace — its namespace signing-key material and, as of #2776, its per-group GroupKeyring encryption keys — so no residual state or decryption capability lingers on disk. Driven by a pending-self-purge marker plus a startup reconcile sweep, so an eviction missed while offline is still completed on the next boot. Failures and sweep outcomes are observable via the self_purge_* metrics. -
Membership & Leave • TEE Fleet HA
-
-
- -
-
SHA-256
-
- Hash function used for content-addressed IDs (delta_id, blob_id), state hashes, and bundle signing. -
-
- -
-
SignedGroupOp
-
- Signed governance operation: version, group_id, parent_op_hashes, state_hash, signer, nonce, op, signature. -
Defined in: crates/context/primitives/src/local_governance.rs
-
-
- -
-
SignedGroupOpenInvitation
-
- Admin-signed invitation token for joining a group. -
Defined in: crates/context/primitives/src/local_governance.rs
-
-
- -
-
SignedGroupOpV1
-
- Legacy gossip wrapper carrying opaque Borsh-serialized SignedGroupOp payload. Still accepted for compatibility. -
-
- -
-
SignerId
-
- did:key string from the Ed25519 key used to sign bundles. Establishes cryptographic update authority. -
-
- -
-
StateDelta
-
- BroadcastMessage variant carrying encrypted state diff for a context. Part of the context DAG. -
-
- -
-
StreamMessage
-
- Wire format for sync streams: Init, Message, or OpaqueError. -
Defined in: crates/node/primitives/src/sync/wire.rs
-
-
- -
-
SubgroupCreated
-
- Governance operation that links a child group to a parent, enabling membership inheritance. -
-
- -
-
SyncManager
-
- Async task managing periodic sync. Selects from HashComparison, LevelWise, Snapshot, or Delta protocols. -
Defined in: crates/node/src/sync/manager.rs
-
-
- - -
T
- -
-
TDX
-
- Intel Trust Domain Extensions. Hardware TEE with attestation via measurement registers (MRTD, RTMR0-3). -
-
- -
-
TEE
-
- Trusted Execution Environment. Hardware-isolated runtime protecting code and data. merod supports TEE mode with KMS key fetch. -
-
- -
-
TeeAdmissionPolicy
-
- The namespace-scoped measurement allowlist a fleet node's attestation is checked against. It pins the acceptable TDX measurements — allowed_mrtd, RTMR values, TCB-status bounds — plus an accept_mock escape hatch for non-TEE test runs. Resolved to the namespace root, so the whole namespace shares one policy. Admission of a TeeAttestationAnnounce succeeds only if the verified quote satisfies this policy. -
TEE Fleet HA
-
-
- -
-
TeeAttestationAnnounce
-
- The broadcast a fleet node publishes on its namespace topic (ns/<hex>) to announce itself for admission. It carries a fresh TDX quote whose report_data is nonce(32) || Sha256(pubkey) — binding the node's public key into the hardware-signed quote and supplying a nonce for freshness. A verifier validates it against the TeeAdmissionPolicy and, on success, publishes MemberJoinedViaTeeAttestation to admit the node as ReadOnlyTee. -
TEE Fleet HA
-
-
- -
-
TopicHash
-
- Gossipsub topic identifier. Formats: context/<hex> for state, group/<hex> for governance. -
-
- - -
U
- -
-
UpgradePolicy
-
- Group setting controlling how application upgrades propagate: Automatic, LazyOnAccess, or Coordinated. -
Defined in: crates/primitives/src/context.rs
-
-
- - -
V
- -
-
auto_join
-
- Group join behavior flag (default: true). When enabled, joining a group automatically subscribes the node to all contexts in the group and its descendant subgroups. -
-
- -
-
VMContext
-
- Runtime execution context: input, context_id, executor identity (auto-resolved from namespace identity), governance_epoch. -
Defined in: crates/runtime/src/logic.rs
-
-
- -
-
VMLogic
-
- WASM host function dispatcher. Manages registers, memory, storage, events. -
Defined in: crates/runtime/src/logic.rs
-
-
- - -
W
- -
-
WASM
-
- WebAssembly. Portable bytecode for application logic. Apps compile to wasm32-unknown-unknown and run via Wasmer with Cranelift JIT. -
-
- - -
X
- -
-
xcall
-
- Cross-context fire-and-forget call from one context's WASM to another in the same namespace on the same node. Queued during execution, runs after commit. No return value, but the outcome — success, denial, or target error — is broadcast as an XCall event. Cross-namespace calls are denied; a target may opt into an entry-point allowlist with #[app::xcall] and read the verified caller via env::xcall_origin(). -
-
- -
- -
No terms match your filter.
- -
-
- - - - - diff --git a/architecture/index.html b/architecture/index.html deleted file mode 100644 index 51a5a2d959..0000000000 --- a/architecture/index.html +++ /dev/null @@ -1,195 +0,0 @@ - - - - -Calimero Core — Architecture Reference - - - -
-
- -
-

Calimero Core

-

Calimero is a peer-to-peer platform for building collaborative apps with automatic conflict-free sync, encrypted P2P networking, and group-based access control. Write your app in Rust or JavaScript, compile to WASM, and every node runs the same logic with state that converges automatically.

- -
- -
-

Start Here

-

Pick your path based on what you want to do.

-
-
1

Build an App

Get a node running, deploy your first app, and understand groups, contexts, and CRDTs.

Getting Started →

-
2

Deploy & Operate

Install merod, configure nodes, manage groups and members, set up TEE mode.

merod & meroctl →

-
3

Understand the Architecture

Dive into the internals: actors, governance DAGs, sync protocols, storage schema.

System Overview →

-
-
- -
-

System Map

- - - - meroctl CLI - operator - - Admin Dashboard - embedded SPA - - External Clients - RPC / WS / SSE - - - - - Server Layer - Axum · REST Admin API · JSON-RPC · WebSocket · SSE · Prometheus - - - - - - - - - - NodeManager - Network events, blob cache, heartbeats - Delta handling, stream routing - Actix Actor - - - - - - ContextManager - Contexts, groups, governance DAGs - Application lifecycle, upgrades - Actix Actor - - - - - - - - - NetworkManager - libp2p: gossipsub, Kad, mDNS, relay - Streams, AutoNAT v2, DCUtR - Actix Actor - - - - - - SyncManager - Hash/Level/Snapshot - Delta protocols - - - - - - WASM Runtime - Wasmer + Cranelift - 50+ host functions - - - - - - - - - - Storage Layer - RocksDB · 10 columns · typed keys · causal DAG · CRDT collections · blob store - - - - - - - - - - mero-auth - JWT issuing + verification - Pluggable providers - Challenge/nonce auth - Embedded or proxy mode - - - - - P2P Network - Other Calimero nodes - gossipsub · streams · DHT - - - - - - calimero-sdk - #[app::state] #[app::logic] → WASM → Runtime - - - - - -
-
Node
-
Context
-
Network
-
Sync
-
Runtime
-
Storage
-
Server / SDK
-
Auth
-
-

Click any box to jump to its deep-dive page.

-
- -

Crate Index

-
- ◆

Document C0 shadow plan in unified-causal-log cutover design doc

New C0 section added to docs/design/unified-causal-log-cutover-plan.md describing the shadow strategy, computation metho

 - ◆

Add AuthoredMap and AuthoredVector fields to abi_conformance app

The `AbiState` struct in the conformance app gains two new fields: `authored_counters: AuthoredMap<String, LwwRegister<u

 - ◆

Node

NodeManager actor, event handling, blob cache, heartbeats, startup orchestration.

 - ◆

Context & Groups

ContextManager actor, 20+ handlers, GroupStore, governance DAGs, upgrades.

 - ◆

Network & P2P

libp2p swarm with 10 behaviours, gossipsub, Kademlia, streams, discovery.

 - ◆

Storage

Column-family KV over RocksDB. 10 columns, typed keys, 20+ group prefixes.

 - ◆

Sync Engine

Hash comparison, level-wise, snapshot, and delta protocols. Governance catch-up.

 - ◆

WASM Runtime

Wasmer + Cranelift, VMLogic, 50+ host functions, CRDT support, profiling.

 - ◆

Server & API

Axum HTTP: REST admin, JSON-RPC, WebSocket, SSE, Prometheus, auth guard.

 - ◆

SDK & Apps

Proc macros for app development. State, logic, events, migrations. 15+ sample apps.

 - ◆

Auth

JWT service, pluggable providers, challenge flow, embedded or proxy mode.

 -
- -
-

Key Concepts

-
-
-

Context

-

A context binds a group (governance) to an application (WASM). Members join via the group, execute methods against the application, and state is replicated across all nodes via the sync engine. Each context has its own DAG of state deltas.

-
-
-

Namespace & Group

-

A namespace is a root group that serves as the application boundary and identity scope. Each namespace gets its own Ed25519 keypair. Groups within a namespace share a single governance DAG with cleartext RootOps and encrypted GroupOps. Membership, capabilities, and access control propagate via gossipsub. See Membership & Leave for the leave operations and ownership-transfer design, and Auto-Follow for opt-in context auto-join.

-
-
-

State Sync

-

Application state changes produce CausalDelta entries in a DAG. Deltas are broadcast via gossip and reconciled using multiple protocols: hash comparison (small diffs), level-wise (broad diffs), snapshot (full transfer), or direct delta exchange.

-
-
-

WASM Execution

-

Applications are compiled to WASM via the calimero-sdk. The runtime provides 50+ host functions for storage, events, CRDTs, blobs, and identity. State mutations produce deltas that propagate to all peers.

-
-
-
- -
-
- - - diff --git a/architecture/local-governance.html b/architecture/local-governance.html deleted file mode 100644 index f76c5a2ae7..0000000000 --- a/architecture/local-governance.html +++ /dev/null @@ -1,641 +0,0 @@ - - - - -Local Group Governance — Calimero Core Architecture - - - -
-
-
Home/Local Group Governance
- - -

Local Group Governance

-

Signed gossip operations, causal DAG, capability-based authorization

- -
-
Ed25519
signed ops
-
DAG
causal ordering
-
6
capability bits
-
20+
op variants
-
- - -
-

Architecture

- - - Node A - - - GroupStore - Members, capabilities - Contexts, metadata - - - DagStore - Causal ordering - Pending resolution - - - OpLog - Persistent sequence of applied ops + DAG heads - - - SignedGroupOp (v8) - group_id + parent_op_hashes - nonce + signer - op: GroupOp + Ed25519 signature - borsh · content-addressed - - - - Node B - - - GroupStore - Members, capabilities - Contexts, metadata - - - DagStore - Causal ordering - Pending resolution - - - OpLog - Persistent sequence of applied ops + DAG heads - - - - - - - gossipsub - ns/<hex> topic - - -
-
GroupStore (persistence)
-
DagStore (causal ordering)
-
OpLog (replay + catch-up)
-
SignedGroupOp (wire format)
-
-
- - -
-

Core Concept: Groups Own Contexts

- - - - Group - admin · members · policies - - - - Context A - app + state + identities - - - Context B - app + state + identities - - - Context C - app + state + identities - - - - - - - - - Admin - - - Member 1 - - - Member 2 - - Group membership → access to contexts - MemberRemoved → cascade removal from all contexts + subgroups - - -

Member Roles

-
-

Admin

Full control. Can perform all governance operations, manage members, configure the group, and write state in all contexts.

-

Member

Standard participant. Can read and write state in joined contexts. Specific governance operations require capability bits (e.g. CAN_INVITE_MEMBERS, MANAGE_MEMBERS).

-

ReadOnly

Observer. Can join contexts and read state, but state mutations are rejected by the platform. Enforced at both the local node (execute handler discards changes) and remote nodes (state deltas from ReadOnly authors are rejected on ingestion).

-
- -

At-Cut Membership Path (MemberPathAtCut)

-

MemberPathAtCut is the at-cut analogue of the live MembershipPath enum, used for deriving a member's enumeration role at a given governance snapshot. It has three variants:

-
-

None

Identity is not a member at the cut — neither directly nor via the open-subgroup inheritance chain.

-

Direct { role }

Identity holds a direct stored membership row. Carries the folded role. The direct row is checked before the admin carve-out, so an identity that is both a stored member and the genesis admin gets the stored row's role (matching live list behaviour).

-

Inherited { anchor, via_admin }

Identity is present via the open-subgroup chain. Carries the anchor group (deepest ancestor with a direct row) and whether that ancestor granted admin authority. Sufficient information to derive the enumeration role without a second lookup.

-
-

Membership authorization is now handled entirely by the unified projection. The previous live governance-cut resolver (acl_view_at, MembershipStatus, prefix_walk_membership, MembershipTransition, heads_equal, MAX_PREFIX_WALK_NODES) has been deleted from governance-store/src/membership/status.rs. The admin predicate used in the projection is narrowed to group admin OR genesis root admin of this group only — the global is_root_admin predicate is intentionally excluded so that root admins are not silently granted access to Restricted subgroups.

- -

Subgroups

-

Groups can form a tree hierarchy. A child group links to a parent via GroupParentRef. Membership and admin authority inherit downward:

-
-
-

Membership Inheritance

-

Each subgroup carries a subgroup_visibility flag — Open or Restricted (absent = Restricted, the safe default). The membership check (check_group_membership_path) walks up the ancestor chain (max depth 16) only through unbroken Open chains, anchored at the deepest ancestor where the identity holds a direct membership row. Direct membership in the target subgroup short-circuits to Direct; otherwise the walk returns Inherited{anchor, via_admin}. A Restricted ancestor is a wall — the walk stops with None.

-

At the anchor, admins inherit unconditionally; non-admins need the CAN_JOIN_OPEN_SUBGROUPS capability bit at that anchor.

-

Inherited role preservation: enumerate_inherited now returns the member's real role at the anchor group (retrieved via role_of(&anchor, &candidate)) rather than defaulting every inherited non-admin to GroupMemberRole::Member. Admin paths still resolve to Admin. Membership authorization is driven entirely by the unified projection; the former live governance-cut resolver (acl_view_at, MembershipStatus, prefix_walk_membership, MembershipTransition) has been removed.

-
-
-

Admin Authority

-

Parent admins inherit structural governance over descendant groups: add/remove members, detach contexts, delete subgroups, set target application.

-
-
- -

Open vs Restricted Subgroups

-

Restricted subgroups are sealed: only direct members can read their governance ops or context state. Their per-subgroup encryption key is delivered via KeyDelivery to direct joiners only.

-

Open subgroups align the cryptographic boundary with the access boundary: their governance ops and context state deltas are encrypted with the parent namespace's key rather than a per-subgroup key. Every namespace member already holds the namespace key, so no separate key-delivery path is needed for inheritance-eligible members. The trade-off is explicit: the read-confidentiality boundary for an Open subgroup is namespace-wide, while the join/write boundary remains capability-gated by CAN_JOIN_OPEN_SUBGROUPS.

-

Receivers don't need a wire flag for the encryption choice — the key_id resolves uniquely to either the subgroup or the namespace keyring, with the namespace keyring tried as a fallback after the subgroup keyring miss.

-

Born-Open subgroups: RootOp::GroupCreated carries a restricted boolean field (wire-breaking change; nodes must reset). When restricted: false, the group_created apply handler writes the subgroup's visibility (set_subgroup_visibility) during apply — before the OpEvent::SubgroupCreated is queued — so the subgroup is already Open when any TEE subscriber reacts, preventing transient direct ReadOnlyTee rows. restricted: true is the default and preserves legacy behavior. CreateGroupRequest gains a restricted field; the REST endpoint accepts an optional visibility field ("open"|"restricted", default "restricted").

- -

Buffered-Op Retry on GroupCreated

-

After applying a RootOp::GroupCreated, the governance layer immediately checks whether the node holds any key for the new subgroup (or the namespace key, for Open subgroups) and, if so, retries all buffered encrypted ops for that group via redrive_buffered_ops_for_group. Previously the only retry trigger was KeyDelivery; if KeyDelivery arrived before GroupCreated, the retry failed because subgroup meta was absent and no subsequent trigger existed.

-

The state_hash staleness check is now bypassed entirely when the subgroup's GroupMeta row does not yet exist, instead of calling compute_state_hash and failing with GroupNotFoundForHash. This mirrors the existing zero-hash fast-path.

-

A one-shot startup sweep (redrive_stranded_ops_sweep) runs after subscribing to op events and before the event loop. It iterates every namespace the node has an identity for, finds groups with buffered ops whose key is already held, and re-drives each one via redrive_encrypted_ops_for_group_counted. It loops until a full pass applies nothing new (bounded by MAX_PASSES=64), narrowing to only groups that made progress in the previous pass. Errors are logged and swallowed; the sweep never blocks startup.

- -

Sync-Stream Authorization

-

The responder-side stream-auth gate (DagHeadsRequest, DeltaRequest, snapshot stream) accepts both direct context membership and inheritance-eligible parent membership. Without this, an inheritance joiner's snapshot probe would be silently closed, leaving them stuck on the all-zero initial root hash. The auth gate uses the same parent-walk that powers join-time authorization, so the contract is consistent end-to-end: namespace member + CAN_JOIN_OPEN_SUBGROUPS = read + write + sync, with no extra round-trips.

- -

Restricting Access via Subgroups

-

To restrict access to specific contexts, create a subgroup and either leave its visibility unset (Restricted by default) or register those contexts under it. Only members of that subgroup (direct or inherited via the Open-chain walk above) can join its contexts.

- -

Create a subgroup with --parent-group-id on meroctl group create or via the parentGroupId field in the API. The SubgroupCreated governance op is published on the parent group's gossip topic. Flip visibility with PUT /groups/:id/settings/subgroup-visibility or meroctl group set-subgroup-visibility.

-

When joining a parent group with auto_join (default: true), the node subscribes to gossip topics and contexts in all descendant subgroups. MemberRemoved cascades to descendant subgroup contexts (skipping child groups where the member has direct membership).

-
- - -
-

Governance DAG Structure

-

Each group has a DAG of SignedGroupOp operations. Operations reference their parents by content hash, forming a causal chain that supports offline ops, concurrent admins, and merge.

- - - - - G - genesis (group create) - - - - - A1 - MemberAdded - - - - - - - - A2 - Admin A - - - - A3 - - - - B1 - Admin B - - - - B2 - - - - - - M - Merge (Noop with 2 parents) - - - - delta_id = SHA-256(signable_bytes) - parents = current DAG heads - state_hash = optimistic lock - - - Ordering guarantees: - • topological (parents first) - • pending queue for OOO - - -
-
Genesis
-
Admin A ops
-
Admin B ops (concurrent)
-
Merge op
-
-
- - -
-

SignedGroupOp Schema (v8 / Namespace v2)

-
-
-

Signable Payload

-
version
Schema version (group ops: 8; namespace ops: 2). verify_signature returns GovernanceError::SchemaVersion for any op carrying an older version — old-schema ops are undecodable rather than silently misparsed. This is a re-bootstrap boundary.
-
group_id
Target group identifier
-
parent_op_hashes
DAG ancestry (current heads at sign time)
-
signer
Ed25519 public key of the admin
-
nonce
Per-signer monotonic (dedup + replay)
-
op: GroupOp
The actual operation (see right column)
-
-
-

GroupOp Variants

-
Noop
Merge sentinel (no state change, used to merge DAG heads)
-
MemberAdded
Add member to group (admin or MANAGE_MEMBERS)
-
MemberRemoved
Remove member + cascade to all contexts in this group and descendant subgroups (admin or MANAGE_MEMBERS)
-
MemberRoleSet
Change member role (admin only)
-
MemberCapabilitySet
Set per-member capability bitfield (admin only)
-
DefaultCapabilitiesSet
Set default capabilities for new members (admin only)
-
UpgradePolicySet
Set group upgrade policy (admin only)
-
TargetApplicationSet
Set target application for the group (admin or MANAGE_APPLICATION)
-
ContextRegistered
Bind a new context to the group (admin or CAN_CREATE_CONTEXT)
-
ContextDetached
Remove context from group (admin only)
-
ContextMetadataSet
Set a context's metadata record — name + opaque data (admin or CAN_MANAGE_METADATA)
-
MemberMetadataSet
Set a member's metadata record (self, or admin / CAN_MANAGE_METADATA for any member)
-
GroupMetadataSet
Set the group's metadata record (admin or CAN_MANAGE_METADATA)
-
GroupDelete
Delete the group (admin only, requires no contexts)
-
GroupMigrationSet
Configure migration parameters (admin or MANAGE_APPLICATION)
-
JoinWithInvitationClaim
Invitation-based group join (inviter must be admin or CAN_INVITE_MEMBERS)
-
ContextCapabilityGranted
Grant per-context capability to a member (admin or MANAGE_MEMBERS)
-
ContextCapabilityRevoked
Revoke per-context capability from a member (admin or MANAGE_MEMBERS)
-
SubgroupCreated
Link a child group under this group (admin only)
-
SubgroupRemoved
Unlink a child group from this group (admin only)
-
-
-
- - -

Operation Flows

- - -
-

Create Context Flow

-
1

Admin calls create_context

API receives group_id (or auto-creates group). Uses the node's namespace identity keypair (from the datastore) if no explicit secret provided.

-
2

Handler signs GroupOps

Emits a sequence: ContextRegistered → optional ContextMetadataSet. Each is a SignedGroupOp applied locally and published via gossip.

-
3

Local apply + OpLog

group_store::apply_local_signed_group_op verifies signature, checks state_hash, updates KV rows, appends to op log, recomputes DAG heads.

-
4

Gossip propagation

Published on group/<hex> topic. Remote nodes receive via SignedGroupOpV1 broadcast, verify, and apply the same ops.

-
- - -
-

Join Group via Invitation

-
1

Admin creates invitation

create_group_invitation generates a SignedGroupOpenInvitation with optional expiration timestamp. Shared out-of-band to the joiner.

-
2

Joiner submits invitation

join_group builds GroupRevealPayloadData, signs it, submits a JoinWithInvitationClaim GroupOp.

-
3

Op applied + member added

Group store validates the claim, adds the new member, updates op log. Gossip ensures all peers converge.

-
- - -
-

Join Context (within Group)

-
1

Group membership implies context access

Context membership is implicit from group membership — no separate governance op or explicit join is required. When a member joins a group with auto_join: true (default), they automatically subscribe to all contexts in that group and its descendant subgroups.

-
2

Context identity resolved

The node uses its namespace identity as the context member identity. No per-context keypair generation needed.

-
- - -
-

Member Removal Cascade

- - - - MemberRemoved - - - - - - - Remove from group - GroupMember row deleted - - - - - - - - Context A identity removed - - - Context B identity removed - - - Join tracking cleaned up - - - - Deterministic: same op on any node produces identical cascade — verified by convergence tests - -
- - -
-

Sync & Catch-up Protocol

- - - - Node A (online) - - - - Node B (rejoining) - - - every 30s - - StateHeartbeat - - - - { group_id, dag_heads: [h1, h2] } - - - - Compare heads - missing h2? - - - - Stream request - - - - GroupDeltaRequest { delta_id: h2 } - - - - read_op_log - scan for hash - - - - - GroupDeltaResponse { payload: borsh(SignedGroupOp) } - - - - apply_signed_ - group_op - - - - converged ✓ - - - -
- - -
-

Startup Recovery

-
-
1

reload_group_dags + recover_in_progress_upgrades

On node start, for each group in the store, reads the full OpLog and rebuilds the in-memory DagStore via restore_applied_delta. Reconstructs pending parent resolution state. Re-spawns propagators for any in-progress group upgrades (crash recovery).

-
2

start_namespace_heartbeat

Starts a 30-second periodic timer. Each tick collects unique namespaces from all groups and publishes NamespaceStateHeartbeat with current namespace DAG dag_heads. Also publishes GroupStateHeartbeat per group for backward compat.

-
3

Peers respond with deltas

Peers compare heads against their own. For any ops the restarted node is missing, they serve them via NamespaceBackfillRequest/Response streams (or legacy GroupDeltaRequest/Response).

-
-
- - -
-

Namespace Governance

-

Since the namespace identity model, governance has moved to a single DAG per namespace (root group). All groups within a namespace share one governance DAG, stored in NamespaceGovOp / NamespaceGovHead storage keys. Operations are either cleartext RootOps (visible to all namespace members) or encrypted GroupOps (only readable by group members).

- -
-
-

RootOps (Cleartext)

-

Visible to all namespace members. Used for structural changes and key distribution:

-
NamespaceCreated { founder }
The namespace-root genesis op (#2474) — the first op in a namespace DAG: no parents, nonce 1 (the head defaults next_nonce to 1 when absent), signed by the namespace key (= the founder). It is self-authorizing: it establishes the founding admin and does not require a pre-existing admin. On apply it writes the root meta with admin_identity = owner_identity = founder, an Admin member row for the founder, and the default CAN_JOIN_OPEN_SUBGROUPS capability. Emitted by create_group.rs on root creation (parent_group_id == None), which previously emitted no governance op at all. Anti-hijack guards: a NamespaceCreated whose signer != founder is rejected (NamespaceCreatedRejection::SignerNotFounder); a NamespaceCreated that carries parents on a bare namespace is rejected (NamespaceCreatedRejection::NotGenesis) and does not advance the DAG head, so a subsequent parentless genesis can still apply. A parented arrival on an already-established namespace is a no-op Ok. Wire-breaking Borsh addition: appended at the end of RootOp so existing discriminants do not renumber, but consumers pinning this crate must coordinate a reset/core-rev bump.
-
GroupCreated
Atomically creates a new group AND nests it under parent_id (strict-tree invariant — no orphan creation path). Carries a restricted boolean field (wire-breaking Borsh change); apply handler writes subgroup visibility immediately when restricted: false, and triggers a buffered-op retry if the node already holds a key for the new group.
-
GroupDeleted
Cascade-deletes a group plus its entire subtree and all contained contexts in one op. Payload carries cascade_group_ids + cascade_context_ids which every peer re-enumerates and verifies for deterministic application.
-
GroupReparented
Atomic edge swap: moves a child from its current parent to a new parent within the same namespace. Replaces the previous GroupNested + GroupUnnested pair; orphan state is structurally impossible.
-
MemberJoined
Cleartext join notification (joiner doesn’t have group key yet)
-
KeyDelivery
ECDH-wrapped group key delivery to a specific member
-
AdminChanged
Changes namespace admin
-
PolicyUpdated
Extensible policy mutations
-
-
-

GroupOps (Encrypted)

-

Encrypted with the group’s key. Non-members store an opaque skeleton (causal structure preserved, content unknown):

-
Group { group_id, key_id, encrypted }
Wrapper containing an encrypted inner GroupOp
-

Inner GroupOp variants (after decryption) are the same as the per-group ops: MemberAdded, MemberRemoved, ContextRegistered, etc.

- -

Key Delivery Mechanism

-
-

1. Joiner publishes RootOp::MemberJoined (signed by joiner’s namespace identity)

-

2. Existing member sees MemberJoined on the namespace DAG

-

3. Member wraps group key via ECDH: shared = ECDH(sender_sk, joiner_pk)

-

4. Member publishes RootOp::KeyDelivery { envelope }

-

5. Joiner unwraps using ECDH(joiner_sk, sender_pk)

-

6. Joiner can now decrypt GroupOp payloads and participate fully

-
-
-
- -

Signing Key Hierarchy (Ancestor Walk)

-

Signing keys are stored at the namespace root when a namespace is created. Child groups (created via create_group_in_namespace) do not get their own copy. Instead, governance_preflight resolves signing authority by walking the parent chain:

-
-

1. Check current group for a signing key matching the requester identity

-

2. If not found, walk to parent via get_parent_group()

-

3. Repeat up to MAX_NAMESPACE_DEPTH (16) levels + final check at root

-

4. Return the first key found, or error if none exists

-
-

Revocation: Revoking a signing key at the namespace root automatically prevents all descendant groups from signing — no stale copies to clean up. Unnesting: breaks the parent link, so the child can no longer walk to the ancestor’s key.

- -

Founder Authority: Genesis-Derived, Not Trust-on-First-KeyDelivery

-

A bootstrapping replica now derives the founding admin authoritatively from the synced DAG genesis (RootOp::NamespaceCreated { founder }), instead of trust-on-first-KeyDelivery. The genesis op's founder becomes admin_identity = owner_identity on apply; the replica no longer pins whoever happened to deliver the first key as admin.

-

Seed behavior change: seed_bootstrap_admin_if_absent (the KeyDelivery-seed path used by TEE replicas) no longer pins the KeyDelivery signer as admin/owner. It now writes a non-authoritative placeholder — the all-zeros PLACEHOLDER_ADMIN_IDENTITY sentinel (the Ed25519 torsion/identity point, outside the prime-order subgroup, so no real keypair can produce it) — as both admin_identity and owner_identity, plus a GroupMemberRole::Member row (not Admin) for the deliverer, and keeps the default-capabilities bootstrap (CAN_JOIN_OPEN_SUBGROUPS). The real founder arrives via the NamespaceCreated genesis op, which recognizes the placeholder and overwrites it. Both orderings (seed-first or genesis-first) converge on the genesis-supplied founder. The seed is gated on absence so it is idempotent and never clobbers a later admin-authored override. Pre-stranded replicas need a re-seed or a gossiped DefaultCapabilitiesSet op.

-

Anti-hijack: the genesis apply is a no-op when the root meta already holds a real (non-placeholder) admin_identity and the founder differs; a NamespaceCreated whose signer != founder is rejected (NamespaceCreatedRejection::SignerNotFounder); a parented NamespaceCreated on a bare namespace is rejected (NamespaceCreatedRejection::NotGenesis) and does not advance the DAG head. A same-founder established re-arrival upgrades any seeded Member row to Admin, repairs a diverged owner_identity, and seeds default caps if absent — idempotent, never downgrades an existing Admin row.

- -

Recursive Member Removal

-

When a member is removed from a group, recursive_remove_member() cascades the removal to all descendant groups and their contexts. Removal from a child group does not affect the parent (upward isolation).

- -

Namespace Governance Epoch

-

Context state deltas carry a governance_epoch field. This is now computed from the namespace DAG heads (not per-group heads), ensuring a consistent governance state reference across all groups in a namespace. Computed via compute_namespace_governance_epoch(store, context_id).

-
- - -
-

Known Limitations

-

state_hash field removed (schema v8 / v2 flag-day)

The state_hash: [u8; 32] field has been removed from SignableGroupOp, SignedGroupOp, SignableNamespaceOp, and SignedNamespaceOp. Because the field was inside the borsh-serialised bytes that are signed and hashed to produce an op's content id, every op's content hash changes. SIGNED_GROUP_OP_SCHEMA_VERSION is now 8 and SIGNED_NAMESPACE_OP_SCHEMA_VERSION is now 2; verify_signature rejects any op carrying an older version with GovernanceError::SchemaVersion. The apply-time staleness check blocks in apply_local_signed_group_op_at_cut and apply_group_op_inner have been deleted; the per-signer nonce window is now the sole anti-replay and dedup gate. GroupHandle::compute_state_hash, NamespaceGovernance::state_hash_for_op, root_op_commits_to_namespace_state, and the pre_apply_state_hash capture in GroupGovernancePublisher have all been removed. This is a re-bootstrap boundary — nodes on older schema cannot exchange ops with nodes on v8/v2.

-

Projection-at-cut is now authoritative

PermissionChecker::is_admin and is_authorized_with_capability use the projection at the op's causal cut as the primary source of truth. The live store is only consulted when the authorizer returns None (e.g. no apply-auth context, empty parents, or incomplete fold). The previous shadow-logging path (shadow_admin, unified_projection_divergence warnings) has been removed. Each group op replayed during retry_encrypted_ops_for_group is authorized at its own parent_op_hashes cut, not the enclosing KeyDelivery op's cut.

-

Multi-admin convergence

Concurrent ops from multiple admins with the same state_hash: first applied wins per node. Different nodes may pick different winners for conflicting ops. Merge Noop resolves heads but op ordering may differ. Single-admin groups have no ambiguity.

-

Governance epoch

State deltas carry a governance_epoch field but it's not yet used to reject stale application-level deltas from removed members.

-

Context-level capability enforcement

Context-level capabilities (ManageApplication, ManageMembers) are stored and can be granted/revoked via governance ops, but are not yet checked during WASM execution. ReadOnly role enforcement is platform-level (execute handler + delta ingestion), not capability-based.

-

Gossip publish failures

If gossip publish fails, it's currently silent. Should surface errors to callers or retry.

-

Genesis founder not yet bound to namespace_id

namespace_id is currently random and not bound to the founder. A self-consistent forged genesis (NamespaceCreated with the attacker as founder) on a bare replica fed by a malicious sole sync peer cannot be prevented yet — the signer-not-founder and parented-genesis rejection guards apply, but they cannot detect a forged genesis where the attacker signs as themselves on a namespace whose id they also control. Tracked in #2932 (bind namespace_id = H(founder) so a forged genesis no longer matches the namespace id); see also #2474.

-
- - -
-

Key File Map

- -

Core Governance

-
context/primitives/src/local_governance/mod.rs
Wire types: SignedGroupOp, GroupOp, SignableGroupOp, SignedNamespaceOp, NamespaceOp (RootOp/GroupOp). Ed25519 signing/verification, content hashing. Schema version: group ops v8, namespace ops v2. The state_hash field has been removed from all four structs; sign() no longer accepts a state_hash argument; verify_signature returns GovernanceError::SchemaVersion for any op with an older version number. Also defines MemberPathAtCut — the at-cut analogue of live MembershipPath — and AclView::member_path_at_cut which returns it. Defines the AtCutAuthorizer trait (including is_admin_or_capability_at_cut and the new membership_path_at_cut), the simplified AtCutMembershipPath enum (None/Direct/Inherited) used by that method, LiveFallbackAuthorizer, and EphemeralProjectionAuthorizer. The live governance-cut resolver (acl_view_at, MembershipStatus, prefix_walk_membership, MembershipTransition, heads_equal, MAX_PREFIX_WALK_NODES) has been deleted from governance-store/src/membership/status.rs; that file now only contains role_from_invited_role and its test. Re-exports of acl_view_at and MembershipStatus have been removed from governance-store/src/membership/mod.rs, governance-store/src/lib.rs, and the group_store facade in context/src/lib.rs.
-
context/src/governance_dag.rs
GroupGovernanceApplier and NamespaceGovernanceApplier bridge calimero_dag: converts ops to CausalDelta, delegates apply to group_store.
-
context/src/group_store/mod.rs
Authoritative persistence (refactored into modules). apply_local_signed_group_op is now a shim that delegates to apply_local_signed_group_op_at_cut with the inert LIVE_FALLBACK_AUTHORIZER. apply_local_signed_group_op_at_cut is the new public entry point accepting an &dyn AtCutAuthorizer alongside the op. The apply-time state_hash staleness check block has been deleted; GroupHandle::compute_state_hash has been removed. The per-signer nonce-window load (load_nonce_window) and containment check are now the sole dedup/anti-replay gate; replayed ops return Ok(()) early. apply_group_op_mutations now threads parents and authorizer through to GroupApplyCtx::new_with_apply_auth. PermissionChecker gains with_apply_auth. PermissionChecker::is_admin and is_authorized_with_capability now query the projection at the op's causal cut via is_admin_at_cut / is_admin_or_capability_at_cut first; if the authorizer returns Some(verdict) that verdict is used directly without touching the live store. Only when the authorizer returns None (no apply-auth context, empty parents, or an incomplete fold) do the gates fall back to live MembershipRepository queries. The private shadow_admin helper and the associated shadow block in is_authorized_with_capability have been removed. apply_group_op_mutations uses signed_group_op.parent_op_hashes (the individual op's own parents) rather than self.parents (the enclosing cut's parents), so that each buffered group op replayed during retry_encrypted_ops_for_group is authorized at its own causal cut instead of the outermost KeyDelivery op's cut.
-
context/src/group_store/namespace.rs
Namespace identity resolution: resolve_namespace(), get_or_create_namespace_identity_bundle(), parent chain walking.
-
context/src/group_store/namespace_governance.rs
Namespace governance DAG application: apply_signed_namespace_op(), key unwrapping, skeleton storage for encrypted ops. GroupCreated apply now writes subgroup visibility and triggers redrive_buffered_ops_for_group when the node holds any key for the new group. The state-hash staleness check and state_hash_for_op / root_op_commits_to_namespace_state helpers have been removed; the synthesized SignedGroupOp inside decrypt_and_apply_group_op no longer copies a state_hash field (field gone). New module ops/namespace/namespace_created.rs implements the self-authorizing RootOp::NamespaceCreated apply handler with branching on placeholder/established state, anti-hijack guards, and idempotent same-founder re-arrival. New error types NamespaceCreatedRejection (SignerNotFounder, NotGenesis { parent_count }) and ApplyError::NamespaceCreatedRejected are added to errors.rs.
-
context/src/group_store/namespace_membership.rs
Namespace membership tracking and policy enforcement.
-
context/src/lib.rs
ContextManager actor. Holds per-namespace DagStores, runs recovery on startup, starts namespace heartbeat timer.
-
context/src/lifecycle.rs
Startup recovery: recover_in_progress_upgrades(), start_namespace_heartbeat() (30s periodic), redrive_stranded_ops_sweep() (one-shot at startup, re-drives buffered encrypted ops for all groups whose key is already held).
-
context/src/metrics.rs
Prometheus metrics: execution count/duration, namespace retry/decode events, membership policy rejections.
- -

Storage Layer

-
store/src/key/group/mod.rs
26+ typed key prefixes for Column::Group: GroupMeta, GroupMember, OpLog, OpHead, Nonce, Context indexes, capabilities, aliases, NamespaceIdentity (0x37), NamespaceGovOp (0x38), NamespaceGovHead (0x39), GroupKeyEntry (0x3A), GroupParentRef (0x36), GroupChildIndex (0x35). Aliases were replaced by MetadataRecord rows (0x2D–0x2F).
-
governance-store/src/lib.rs
Exports PLACEHOLDER_ADMIN_IDENTITY: [u8; 32] (all-zeros sentinel, the Ed25519 identity/torsion point) and companion placeholder_admin_identity() → PublicKey. Used by both seed_bootstrap_admin_if_absent and the genesis anti-hijack gate to distinguish an authoritative admin from a non-authoritative seed placeholder. Also exports the new GroupKeyring::delete_key_by_id(key_id: &[u8; 32]) method which deletes a single stored group key by id without the membership-removed purge precondition; intended for the namespace-root genesis rollback path.
-
store/src/types/group.rs
Value types: GroupOpHeadValue, GroupMetaValue, GroupUpgradeValue, NamespaceIdentityValue, NamespaceGovHeadValue.
- -

Network Integration

-
node/src/handlers/network_event/namespace.rs
Namespace gossip ingestion: NamespaceGovernanceDelta, NamespaceStateHeartbeat → apply ops, trigger key delivery, or backfill via P2P streams.
-
node/src/handlers/network_event/specialized.rs
Specialized node discovery, TEE attestation, join confirmation handling.
-
node/src/key_delivery.rs
Automatic key delivery: on MemberJoined, wraps group key via ECDH and publishes RootOp::KeyDelivery.
-
node/src/sync/manager/mod.rs
Responds to GroupDeltaRequest and NamespaceBackfillRequest by scanning op log and sending responses.
- -

Handlers

-
context/src/handlers/create_context.rs
Emits ContextRegistered + optional ContextMetadataSet as signed ops. Uses namespace identity for signing.
-
context/src/handlers/join_group.rs
Invitation claim flow: generates/retrieves namespace identity, subscribes to namespace topic, stream-requests join, unwraps group key, applies governance ops snapshot, auto-joins contexts.
-
context/src/handlers/apply_signed_namespace_op.rs
Namespace DagStore integration: feeds namespace ops into the namespace-level DAG.
-
context/src/handlers/apply_signed_group_op.rs
Group DagStore integration: feeds op into DAG, handles pending/applied/error states.
-
context/src/handlers/list_namespaces.rs
Namespace listing, querying, and identity resolution for the admin API.
-
context/src/handlers/upgrade_group.rs
Application upgrade propagation across group contexts.
- -

Tests

-
context/src/group_store/tests.rs
Comprehensive group store tests including namespace identity, governance, key delivery, and membership policy. Approximately 12 unit tests that covered the removed acl_view_at / MembershipStatus / prefix_walk_membership code have been deleted alongside those symbols. Eight state_hash staleness tests (namespace_group_op_zero_state_hash_bypasses_check, namespace_group_op_with_current_state_hash_applies, namespace_group_op_with_stale_state_hash_applies_with_warning, group_op_with_stale_hash_and_absent_meta_applies, namespace_root_op_with_current_state_hash_applies, namespace_root_op_with_stale_state_hash_applies_with_warning, state_hash_allows_sequential_ops, state_hash_zero_skips_validation) and their shared setup_state_hash_test_fixture helper have been deleted — they pinned behaviour that no longer exists. Three new flag-day regression tests have been added: pre_flag_day_group_op_version_is_rejected, pre_flag_day_namespace_op_version_is_rejected, and v7_borsh_layout_group_op_is_rejected_not_misparsed, which assert that ops carrying older schema versions are rejected by verify_signature with GovernanceError::SchemaVersion.
-
governance-store/src/namespace/tests.rs, governance-store/src/tests.rs
New regression and unit tests for namespace genesis: replica_genesis_founder_survives_non_owner_seed_and_applies_owner_ops (#2474 regression guard); bare-store genesis and anti-hijack no-op; placeholder-seed convergence; signer-not-founder rejection; parented NamespaceCreated on bare namespace returns Err and does not advance head; parented on established namespace is Ok no-op; genesis upgrades seeded Member founder to Admin; same-founder idempotent re-arrival ensures Admin row and caps; no downgrade of existing Admin row; owner_identity repair; head not advanced on failed genesis apply; placeholder_admin_identity_never_equals_a_real_key. Existing tee_replica_seed_bootstrap_admits_tee_with_open_join_cap and seed_bootstrap_admin_repairs_missing_member_row tests updated to reflect Member-not-Admin seed behavior.
-
context/tests/local_group_governance_convergence.rs
Two-node convergence, op log, idempotency, DAG ordering, state_hash conflict prevention, cascade removal, head capping.
-
node/src/local_governance_node_e2e.rs
E2E: gossip message → node manager → context apply → membership verified.
- -
- -

Last-Admin At-Cut Gate: Trait, Shadow Logging & Policy Wiring

- -

Overview

-

is_last_admin_at_cut has been added to the AtCutAuthorizer trait, completing the set of at-cut gates used by permission-checking paths. MembershipPolicy now optionally carries the op's parent cut and an authorizer reference, activated via with_apply_auth. GroupApplyCtx::new wires both through automatically, so every group-op apply path participates in last-admin shadow logging.

- -

AtCutAuthorizer Trait

-

The trait now exposes three at-cut predicates:

-
    -
  • is_admin_at_cut(&self, identity) → Option<bool>
    • -
  • is_admin_or_capability_at_cut(&self, identity, cap) → Option<bool>
    • -
  • is_last_admin_at_cut(&self, identity) → Option<bool>
    • -
  • membership_path_at_cut(&self, identity) → Option<AtCutMembershipPath> (new) — returns the kind of membership path the identity holds at the op's causal cut. Option::None means "defer to the live path"; AtCutMembershipPath::None means "not a member at the cut".
    • -
-

All four share the same empty-parents contract: every implementation must return None when the op's parent_op_hashes slice is empty. Violating this contract would cause genesis ops to be falsely rejected — the live-store fallback is always authoritative for the genesis case.

- -

Implementation per Authorizer

-
    -
  • EphemeralProjectionAuthorizer — delegates to ScopeProjections::is_last_admin_at_cut for the last-admin gate. For membership_path_at_cut, delegates to ScopeProjections::membership_path_at_cut (which calls the projection's member_path_at_cut walk using the auth-cut context) and maps the result to AtCutMembershipPath, projecting away the role/anchor detail not needed by the MemberJoinedOpen gate. Guards against empty parents before delegating; returns None if the projection fold is incomplete.
    • -
  • LiveFallbackAuthorizer — implements all methods as unconditional None, including membership_path_at_cut. This keeps all shadow paths inert for any MembershipPolicy constructed outside an apply-auth context (e.g. read-only membership queries).
    • -
- -

MembershipPolicy Changes

-

Two new optional fields are added to MembershipPolicy:

-
    -
  • apply_auth_parents: Option<&[DeltaId]> — the parent cut hashes of the op being applied.
    • -
  • apply_auth: Option<&dyn AtCutAuthorizer> — reference to the current authorizer.
    • -
-

Set both together via the builder method:

-
policy.with_apply_auth(parents: &[DeltaId], authorizer: &dyn AtCutAuthorizer) → Self
-

When these fields are absent (the default), every shadow call is a no-op, preserving the behaviour of non-apply constructions.

- -

Last-Admin Enforcement: would_orphan_admins

-

The two last-admin enforcement methods on MembershipPolicy now delegate to a shared would_orphan_admins helper that resolves the blocking decision from the projection at the op's parent cut via AtCutAuthorizer::is_last_admin_at_cut:

-
    -
  • ensure_not_last_admin_removal — called when a MemberRemoved op targets an admin identity.
    • -
  • ensure_not_last_admin_demotion — called when a MemberRoleSet would drop the last admin to a non-admin role.
    • -
-

would_orphan_admins calls authorizer.is_last_admin_at_cut(identity) and uses that verdict directly. It only falls back to the live membership computation (is_admin && !has_another_admin) when no apply-auth context is present (local pre-checks, cascades, tests) or when the projection fold is incomplete. This live fallback is a temporary measure and is noted as retiring in a follow-up.

-

The shadow_last_admin helper, which previously compared the projection verdict against the live verdict and emitted unified_projection_divergence / last-admin plane warnings on mismatch, has been deleted. Its purpose was to validate parity before the enforcement flip; now that the projection is authoritative it is no longer needed. Similarly, the unified_projection_divergence / membership-path warning path (previously emitted by shadow_membership_path) has also been deleted now that MemberJoinedOpen authorizes directly via the projection.

- -

Wiring through GroupApplyCtx

-

GroupApplyCtx::new now calls .with_apply_auth(parents, authorizer) on the MembershipPolicy it constructs, using the same parents and authorizer already threaded into PermissionChecker. The call site is unchanged for callers; the wiring is automatic. This means:

-
    -
  1. Every op applied via apply_group_op_mutations activates the last-admin shadow.
    2. -
  2. Ops replayed during retry_encrypted_ops_for_group each carry their own parent_op_hashes, so the shadow is evaluated at the correct individual causal cut — not at the enclosing KeyDelivery op's cut.
    3. -
- -

Empty-Parents Contract (Genesis Safety)

-

The contract applies uniformly to all three trait methods. A genesis op has no parents (parent_op_hashes = []). Returning anything other than None in that case would mean the projection is evaluated against an empty fold, which is structurally unsound. The EphemeralProjectionAuthorizer guard and the LiveFallbackAuthorizer unconditional-None path both satisfy this contract. Future implementations must do the same.

- -

Relationship to Existing At-Cut Pattern

-

This addition is intentionally symmetric with the existing is_admin_at_cut and is_admin_or_capability_at_cut gates on PermissionChecker. The last-admin gate now follows the same projection-authoritative model as those gates: the projection result is used directly, and the live store is only consulted as a fallback when no apply-auth context is present or the fold is incomplete. The shadow_last_admin divergence-logging path has been removed now that the flip is complete. Future changes to last-admin enforcement are localized to MembershipPolicy and would_orphan_admins — no trait changes are required.

-

The new membership_path_at_cut method is now used as the primary authorization source for the MemberJoinedOpen gate. member_joined_open::apply first calls NamespaceApplyCtx::projection_membership_path, which queries the authorizer for the projection's at-cut path and returns Option<AtCutMembershipPath>. If the projection returns a path (i.e. Some(...)), the gate's match branches operate on AtCutMembershipPath variants directly — the live check_path call is skipped entirely. Only when projection_membership_path returns None (no apply-auth context, incomplete fold, or the default live-only authorizer) does the gate fall back to the live MembershipRepository::check_path read. The previous NamespaceApplyCtx::shadow_membership_path helper, which accepted an eagerly-computed live_path argument and logged a tracing::warn! on divergence, has been replaced by projection_membership_path, which takes no live_path argument and returns the resolved projection path instead of logging. The divergence-comparison logic and the unified_projection_divergence / membership-path warning have been removed entirely. The helper membership_path_kind, which collapsed the live MembershipPath to AtCutMembershipPath for that comparison, is no longer needed by this path.

-

Note on AtCutMembershipPath: This is a simplified three-variant enum (None / Direct / Inherited) distinct from the richer live MemberPathAtCut. It collapses role and anchor detail to only the kind the MemberJoinedOpen gate needs. Option::None returned from membership_path_at_cut (and from projection_membership_path) means "defer to live"; AtCutMembershipPath::None means "not a member at the cut".

-
-

CI Observability: Governance-Divergence Pull Reporting

-

A non-failing GitHub Actions step named "Report scope_root governance-divergence pulls" has been added to the e2e CI workflow. It provides visibility into how often the corrective governance-divergence pull is triggered during a test run without blocking CI on the count.

- -

What the Step Does

-
    -
  • Greps Docker container logs for the structured log marker gov_divergence_pull_triggered.
    • -
  • Reports the total trigger count across the entire run.
    • -
  • Breaks down counts per context_id to surface any pathological per-tick storm where a single context is responsible for a disproportionate share of pulls.
    • -
- -

Why It Is Non-Failing

-

The step is intentionally designed to observe rather than enforce. A non-zero trigger count is expected and acceptable in normal operation; the step only fails if the log grep itself errors. This matches the intent of the underlying corrective pull: it is a best-effort convergence aid, not a hard invariant.

- -

Interpreting the Output

-
    -
  • Total count of zero — the divergence path was never hit; governance state was already consistent at every heartbeat tick.
    • -
  • Small total, spread across contexts — normal catch-up after transient lag; no action needed.
    • -
  • Large total concentrated on one context_id — indicates a potential pathological storm for that context. Investigate whether the governance epoch is oscillating or whether a peer is repeatedly publishing conflicting heads.
    • -
- -

Finding the Logs

-
# Example grep pattern used by the CI step
-docker logs <container> 2>&1 | grep gov_divergence_pull_triggered
-
-# Per-context breakdown (approximate shell equivalent)
-docker logs <container> 2>&1 \
-  | grep gov_divergence_pull_triggered \
-  | grep -o 'context_id=[^ ]*' \
-  | sort | uniq -c | sort -rn
- -

Relationship to the Corrective Pull

-

The gov_divergence_pull_triggered marker is emitted by the scope_root governance reconciliation path when it detects that the local governance epoch lags behind the epoch carried by an incoming state delta. The corrective pull requests a namespace backfill to close the gap. The CI step makes this otherwise-invisible activity visible in test run logs without introducing a flaky hard gate on trigger frequency.

-
-
-
- - - diff --git a/architecture/membership-and-leave.html b/architecture/membership-and-leave.html deleted file mode 100644 index b0cf15ffa0..0000000000 --- a/architecture/membership-and-leave.html +++ /dev/null @@ -1,1941 +0,0 @@ - - - - -Membership & Leave Operations — Calimero Core Architecture - - - -
-
-
Home/Membership & Leave
- - -

Membership & Leave Operations

-

Voluntary leave, owner-initiated removal, and role-scoped TEE self-purge across namespaces, groups, and contexts

- -
-
3 leaves
context · group · namespace
-
Key rotation
on removal (publisher pipeline)
-
Direct
leave deletes a row
-
TEE purge
role-scoped hard delete
-
- -
-

-As-built page. This was originally a proposal; it now documents shipped behavior. For the -fleet-specific eviction story (owner kicks a ReadOnlyTee node on HA-disable, the -self-purge listener, and the marker-gated startup reconcile) see -Fleet HA. Term definitions live in the -Glossary. The governing design record is -docs/adr/0002-fleet-tee-leave-protocol.md. -

-
- - -
-

Problem this solved

-

-Originally, the only way out of a Calimero group or namespace was to be removed by an admin — -MemberRemoved was admin-initiated and there was no self-leave op. Users could not -voluntarily exit a workspace they joined; muting a context they were auto-followed into required -client-side filtering rather than a real opt-out, because auto-follow re-adds them on the next -ContextRegistered event. -

-

-Separately, every group records a single "creator / fallback admin" identity -(GroupMetaValue.owner_identity) but the role enum did not distinguish that creator -from any other admin. There was no clean way to express "this person owns the group, others -administrate it." -

-

-The shipped design adds three leave operations and an owner identity stored on group metadata that -gates self-leave and involuntary removal. The pieces share infrastructure: the stored owner blocks -leave_group/leave_namespace and involuntary MemberRemoved; -owner-initiated MemberRemoved reuses the group key-rotation pipeline; -leave_namespace is the namespace-root MemberLeft plus a cascade through -descendant groups. -

-

-A separate, role-scoped mechanism layers on top for fleet TEE nodes: when a -ReadOnlyTee member is removed, the evicted node hard-deletes its local key -material (see TEE self-purge), because a TEE node has no rejoin path -under the same identity. Non-TEE removals keep the soft-leave invariant. -

-
- - -
-

Membership Model Recap

-

-Background on the data shapes the leave operations operate over. Names match the code as it -exists today. -

- -

Hierarchy

-
    -
  • Namespace. A ContextGroupId with no parent. The trust anchor for everything - beneath it. There is no separate Namespace type — a namespace is a root group.
    • -
  • Group / subgroup. A ContextGroup with a parent_group_id. - Forms a strict tree under a single namespace.
    • -
  • Context. A unit of replicated state. Belongs to exactly one group via - ContextGroupRef. Members of the group can join, sync, and write to it.
    • -
- -

Direct vs inherited membership

-

-Group membership is computed by check_group_membership_path -(crates/context/src/group_store/membership.rs:217-291): -

-
    -
  • Direct membership = a stored (member, group) row exists. Created by - add_group_members (explicit invitation). Required for Restricted groups; possible - but redundant for Open subgroups.
    • -
  • Inherited membership = no row exists; the walker concludes membership by walking the - parent chain through Open subgroups when the member holds the - CAN_JOIN_OPEN_SUBGROUPS capability in the parent.
    • -
-

-This distinction is critical for leave operations. Leave deletes a direct row. If the only -path to membership is inheritance, there is no row to delete — the member must leave whichever -ancestor provides the inheritance. -

-

-In the governance-DAG projection layer, this distinction is reflected by the fold rule for -RootOp::MemberJoinedOpen: it folds to OpPayload::Noop rather than -OpPayload::MemberAdded. No persistent GroupMember node is written for -an open-subgroup inheritance join; instead, AclView::is_member_at_cut re-derives -membership at query time via the anchor-root membership + subgroup tree + visibility + -CAN_JOIN_OPEN_SUBGROUPS capability walk — exactly mirroring the live store. This -means inherited access is automatically revoked when the member is removed from the root group -and restored if they rejoin, with no over-grant surviving root-anchor removal. -

-

-In the governance-DAG projection layer, this distinction is reflected by the fold rule for -RootOp::MemberJoinedOpen: it folds to OpPayload::Noop rather than -OpPayload::MemberAdded. No persistent GroupMember node is written for -an open-subgroup inheritance join; instead, AclView::is_member_at_cut re-derives -membership at query time via the anchor-root membership + subgroup tree + visibility + -CAN_JOIN_OPEN_SUBGROUPS capability walk — exactly mirroring the live store. This -means inherited access is automatically revoked when the member is removed from the root group -and restored if they rejoin, with no over-grant surviving root-anchor removal. -

-

-This behavior is pinned by three assertions in -projection_membership_equivalence.rs: (1) -projection_matches_live_across_inherited_join_and_root_removal asserts that after -an inherited join, member_at_cut_authoritative returns Some(true); -(2) the same test asserts that after a root removal that the live store rejects, -member_at_cut_authoritative must not return Some(true) — the -over-grant guard; (3) -projection_matches_live_across_leave_and_rejoin_inheritance asserts that after a -full leave-and-rejoin cycle, member_at_cut_authoritative correctly restores access. -Together these tests ensure the authoritative grant resolver agrees with the live store across -all three scenarios and never over-grants on root-anchor removal. -

- -

Visibility (Open vs Restricted)

-

-A per-group flag (VisibilityMode::{Open, Restricted} in -crates/context/config/src/lib.rs:134-137) that controls whether the parent-walk -inherits into this subgroup or stops cold: -

-
    -
  • Open: parent members with CAN_JOIN_OPEN_SUBGROUPS inherit membership.
    • -
  • Restricted: walks halt; only direct invitations admit members.
    • -
-

-At leave-time, visibility doesn't change behavior — it only affects whether a row exists in the -first place. Most members of Open subgroups have no row (inherited); most members of Restricted -subgroups have one (direct invite). -

- -

Roles & capabilities

-

-The GroupMemberRole enum (crates/primitives/src/context.rs) has exactly -four variants: -

-
    -
  • Admin, Member, ReadOnly, ReadOnlyTee.
    • -
  • There is no Owner role variant. "Owner" is not a member role — it is a - single identity stored on group metadata (GroupMetaValue.owner_identity). The owner - is whichever member's public key equals that field; they are otherwise an ordinary - Admin. Owner status is enforced by identity comparison in the apply handlers - (OwnerImmuneFromRemoval, OwnerCannotSelfLeave), not by a role check. See - the Owner section.
    • -
-

-Per-member capability bits (crates/context/config/src/lib.rs, MemberCapabilities): -

-
    -
  • CAN_CREATE_CONTEXT, CAN_INVITE_MEMBERS, - CAN_JOIN_OPEN_SUBGROUPS, MANAGE_MEMBERS, - MANAGE_APPLICATION, CAN_CREATE_SUBGROUP, - CAN_DELETE_SUBGROUP, CAN_MANAGE_VISIBILITY, CAN_MANAGE_METADATA.
    • -
- -

Existing protections

-
    -
  • Last-admin protection (membership_policy.rs:39-46): - MemberRemoved rejects with LastAdmin if it would leave a group with - zero admins. Also enforced on demotion.
    • -
  • Key rotation on removal (group_governance_publisher.rs, - sign_apply_and_publish_inner): publishing MemberRemoved automatically - generates a fresh group key via GroupKeyring, wraps it (wrap_for_member) - for every remaining member — excluding the removed one — and delivers it. The removed - member never receives the new key, so it cannot decrypt anything written after eviction. This is - the forward-secrecy mechanism for the namespace's future writes. Owner-initiated removal is the - cryptographically-complete leave path.
    • -
  • Namespace genesis op (create_group.rs, parent_group_id == None - branch): when a namespace root is created, the handler signs, applies, and publishes a - RootOp::NamespaceCreated { founder: admin_identity } genesis op via - sign_apply_and_publish_namespace_op. A NoPeersSubscribed error from - publish is treated as success (apply succeeded locally; publish is best-effort). Any other - genesis-apply error triggers a full rollback of all locally-written root rows — meta, founder - Admin member row, default caps, group key, and signing key (plus optional name metadata) — via - individual idempotent deletes (GroupKeyring::delete_key_by_id, - SigningKeysRepository::delete_key, MetaRepository::delete, - MembershipRepository::remove_member, CapabilitiesRepository::delete_default, - MetadataRepository::delete_group) before returning Err. The namespace - identity row is deliberately not rolled back so a retry obtains the same - identity/founder. The DAG head does not require rollback because apply_signed_op is - head-atomic (the head advances only after a successful apply).
    • -
-
- - -
-

Forward-only invariant — load-bearing

-

-The apply-time membership check is a function of two things only: the signer's identity -and the governance-DAG position the signer signed against. It is not a function -of the receiver's current local state. This is the forward-only property: a write authored -at governance position P stays valid forever, even if the receiver has since applied -a MemberRemoved for the signer at a later position P' in its local DAG. -

- -

Why this matters

-

-Without forward-only, two peers receiving the same set of ops in different orders would end up -with different state: -

-
    -
  • Peer A receives MemberRemoved first, then the pre-removal delta. If "is the - signer currently a member?" gates the apply, the delta is rejected.
    • -
  • Peer B receives the delta first, then MemberRemoved. The delta applies; the - later removal doesn't retroactively undo it.
    • -
-

-That divergence cascades: any subsequent delta that reads the (different) state diverges further, -context root hashes drift apart, and the network has no honest convergence point. Forward-only -removes the receive-order dependence: pre-removal writes always apply on every peer, regardless -of when the receiver learns about the removal. -

- -

How it's implemented

-

-Every state delta carries a governance_position field that names the governance-DAG -heads the author observed at sign time. On receive, the apply path evaluates membership at the -signed heads via the unified projection — which resolves the ancestry of those heads and derives -the membership state from the folded projection. The resolution visits -only the prefix that was already final when the delta was signed; anything causally after -the signed cut (including a later MemberRemoved for this signer) is invisible to -the resolution and cannot retroactively invalidate the delta. -

-

-The namespace-level applier (NamespaceGovernanceApplier in -crates/context/src/governance_dag.rs) threads this cut into the apply path directly: -DeltaApplier<SignedNamespaceOp>::apply calls -apply_signed_namespace_op_at_cut(&delta.parents, &LIVE_FALLBACK_AUTHORIZER) -rather than the cut-free apply_signed_namespace_op. Passing delta.parents -as the causal cut ensures membership is evaluated at the position the author signed against. -LIVE_FALLBACK_AUTHORIZER is a placeholder authorizer that delegates to the live store; -membership authorization for all other paths is now handled entirely by the unified projection -(EphemeralProjectionAuthorizer / ScopeProjections) — the -acl_view_at, MembershipStatus, prefix_walk_membership, -MembershipTransition, heads_equal, and MAX_PREFIX_WALK_NODES -helpers that previously backed the live DAG-walk resolver have been deleted. -

- -

Resolving the current governance cut — ScopeProjections::namespace_current_heads

-

-Current-state membership reads (as opposed to historical signed-position checks) need a "now" -frontier: the set of parent hashes that represents the namespace's current DAG head. The static -method ScopeProjections::namespace_current_heads provides this: -

-
    -
  1. Resolves the supplied ContextGroupId to its owning namespace by walking the - parent chain.
    2. -
  2. Reads that namespace's DAG head record from the governance store.
    3. -
  3. Returns the parent hashes of that head — the cut argument passed to - membership_status_at (or AclView::is_member_at_cut) for a - current-state query.
    4. -
-

-None return cases. The method returns None when the group cannot -be resolved to a namespace (e.g. the group does not exist in the local store) or when the head -record itself is unreadable. Callers must fall back to the live membership store -(check_group_membership_path) in both cases rather than treating None as -an empty cut — an empty cut would falsely deny all access. -

-

-Projection-derived member counts and migration cohorts. ScopeProjections::member_identities_with -extends this pattern to identity-set reads: given an already-built projection fold it calls -cut_ancestry_complete on the fold before reading the ACL view, and returns -None if the ancestry is only partially folded (a governance-backfill race). This -None propagates to get_group_info (which falls back to -MembershipRepository::count + enumerate_inherited) and to -collect_migration_cohort (which falls back to the live list ∪ enumerate_inherited -union). The private ephemeral_view helper has been removed; the subtree ephemeral path -now calls ephemeral_fold directly and performs the cut_ancestry_complete -check itself before reading the view. -

- -

Ephemeral per-query projection — ScopeProjections::member_now_ephemeral

-

-Context-layer handlers that do not hold a cached ScopeProjections instance use -ScopeProjections::member_now_ephemeral to obtain a one-shot membership verdict from -the governance DAG without requiring a pre-built projection. On each call it: -

-
    -
  1. Resolves the supplied group to its owning namespace.
    2. -
  2. Folds the full governance DAG for that namespace via collect_namespace_ops.
    3. -
  3. Reads the DAG heads after the fold (not before), so a concurrent governance-op - advance cannot leave the cut naming ops the fold hasn't processed.
    4. -
  4. Calls member_at_cut at those heads and returns the boolean verdict.
    5. -
-

-None return cases. The method returns None — never deciding -against the member — when: (a) the namespace is unresolvable, (b) -collect_namespace_ops reports a store fault (surfaced with a tracing::warn), -or (c) the cut-completeness guard determines the recorded heads don't cover the full fold. Callers -must treat None as undecidable and fall back to the live store rather than -denying access. -

- -

Unified membership gate — ScopeProjections::member_now_checked

-

-ScopeProjections::member_now_checked is the single decision point used by all -context-layer membership gates (get_cascade_status, -get_group_upgrade_status, list_group_contexts, -list_all_groups). Its semantics: -

-
    -
  • Projection wins when decidable. It calls member_now_ephemeral first. When - the projection returns a verdict it acts on that verdict, then cross-checks - MembershipRepository::is_member best-effort. A transient live error does not - override a projection verdict.
    • -
  • Live wins when projection is undecidable. When member_now_ephemeral - returns None, member_now_checked falls back to - MembershipRepository::is_member as authoritative and propagates any live error - normally.
    • -
  • Divergence telemetry. When the projection and live store both return definite but - conflicting answers, member_now_checked emits a - tracing::warn with marker=unified_projection_divergence and - plane=membership-query. The projection verdict is still used as the final answer.
    • -
-

-Direct use of MembershipRepository in -get_cascade_status.rs, get_group_upgrade_status.rs, -list_all_groups.rs, and list_group_contexts.rs is eliminated; those -handlers now call member_now_checked exclusively and no longer import -MembershipRepository. -

-

-Behavior change in list_all_groups. Previously, a membership-check error in -the per-group loop would propagate and abort the entire response. Now the loop catches errors from -member_now_checked with a tracing::warn and continues to the next group, -silently omitting groups whose membership cannot be determined. This matches the existing skip -behavior for a missing node identity and avoids a single inaccessible group blocking the whole -listing. -

- -

Op-store reconstruction root — ScopeProjections::scope_root_from_op_store and shadow_compare_op_store

-

-ScopeProjections::scope_root_from_op_store(scope_id, store) provides an independent -view of a scope's governance root derived entirely from the persisted op log rather than the -maintained in-memory projection: -

-
    -
  1. Loads all ops for the given ScopeId via load_scope_ops.
    2. -
  2. Replays them into a fresh ScopeProjections instance (causal ordering recovered - through DagStore<Op> + UnifiedApplier).
    3. -
  3. Returns the resulting scope_root computed with a fixed zero - entities_root.
    4. -
-

-Returns Ok(None) when the scope has no persisted ops; Err on store -faults. This is intended for completeness verification before C2.2b — not for hot-path membership -decisions. -

-

-ScopeProjections::shadow_compare_op_store(namespace_id, store) is an observe-only -companion that compares the maintained projection's scope_root against the -op-store reconstruction. On mismatch it emits a tracing::warn with -marker=unified_op_store_divergence and truncated hex of both roots. -This log marker is the e2e gate signal for C2.2b readiness. The method silently skips -(no divergence reported) when the scope is unfolded, the op-store is empty for it, or the store -read fails — none of those conditions count as divergence. calimero_op / -ScopeId types stay inside the context crate and are not leaked into the node crate. -

- -

Convergence root — ScopeProjections::scope_root_for and group_scope_root_ephemeral

-

-ScopeProjections::scope_root_for(scope) computes a single -SHA-256 digest over the folded state of a governance scope: -SHA-256(entities_root ‖ acl_hash ‖ governance_hash). This -gives callers a cheap way to detect whether two nodes have converged on -the same membership/governance state without enumerating members. -

-
    -
  • Returns None when the scope has not been folded - yet (i.e. it has no entry in the projection). Callers must treat - None as "cannot compare" — never as a divergence signal. - False divergences from mid-backfill nodes are the main hazard here.
    • -
  • ACL component is currently an empty placeholder because - SetWriters ops are not yet routed through the governance - DAG. The hash over ACL state will become meaningful at C2 when that - routing lands.
    • -
-

-The companion free function -ScopeProjections::group_scope_root_ephemeral(group_id, store) -builds an ephemeral projection from the raw store (no -NodeState required), adds an ancestry-completeness guard via -cut_ancestry_complete, and delegates to -scope_root_for. It returns None when the -governance DAG cut is not fully folded, preventing false divergence -reports from nodes that are still mid-backfill. Both symbols are public -API on ScopeProjections -(crates/context/src/scope_projections.rs). -

-

-Behavior is pinned by -scope_root_for_moves_on_hash_neutral_membership_change_and_is_none_when_unfolded: -the test holds entities_root fixed, ingests an -AdminChanged op then a causally-chained MemberAdded -op, and asserts that scope_root_for returns different values -before and after, and returns None for an unfolded scope. -

- -

Batch role resolution — ScopeProjections::member_roles_for

-

-Where namespace_current_heads answers "is this member present?", -member_roles_for answers "what role does the projection assign to each of these -members?". It folds the ACL view exactly once for the supplied batch and returns a -BTreeMap<PublicKey, GroupMemberRole>: -

-
    -
  • Direct members keep their folded role from the projection.
    • -
  • Inherited{via_admin:true} → Admin.
    • -
  • Inherited{via_admin:false} → the member's stored role at the anchor - group (mirroring live's list ∪ enumerate_inherited semantics).
    • -
  • Abstain (empty map) when the ancestry is not fully folded, or when the group has no - entry in view.groups (materialized-fallback parity — Restricted subgroup or - undecryptable member ops). An empty return means "projection has no opinion"; callers must not - treat it as "all roles are wrong".
    • -
-

-list_group_members calls member_roles_for once for all live members -after the existing identity-set shadow. Each live member's role is compared to its projected -role; mismatches are emitted as unified_projection_divergence log events with -plane=membership-role. Members absent from the returned map (projection abstains) -are skipped — presence divergence is already owned by the identity shadow. The handler still -returns the live list unchanged. -

-

-Operator note. membership-role divergence warnings in logs indicate a -pre-flip validation failure. They will block the eventual flip of list_group_members -from the live store to the projection as the authoritative source. Note that -acl_view_at and MembershipStatus (the former live DAG-walk resolver -types) are no longer available; divergence detection now operates entirely through -ScopeProjections projection methods. -

- -

Full effective-member enumeration — ScopeProjections::member_entries_with

-

-member_entries_with returns a Vec<(PublicKey, GroupMemberRole)> for -every effective member of a group at the current cut, derived entirely from the already-folded -auth_cut_context view. It reuses member_identities_in_view_with_ctx for -the identity set and resolves each member's role via member_path_at_cut. -

-

-The method returns None (signalling: defer to the live store) in three cases: -

-
    -
  • The cited ancestry is not yet fully folded — a partial fold could produce an incomplete member - list, so the caller falls back rather than silently truncating.
    • -
  • The target group has no folded direct-membership entry in the view — the projection does not - cover this group yet.
    • -
  • A validated identity resolves to MemberPathAtCut::None or has a missing inherited - anchor row — this indicates a fold inconsistency. The method logs a - unified_projection_divergence warning and abandons the entire enumeration - rather than returning a partial list.
    • -
-

-The all-or-nothing fallback contract means callers never observe a silently truncated member list: -either the projection supplies the complete set or the live store is used in its entirety. -

- -

At-cut effective role — ScopeProjections::role_at_cut_for_group

-

-ScopeProjections::role_at_cut_for_group(member, group_id, cut) resolves the effective -GroupMemberRole of a member in a group at a specific governance cut. It gates on -cut_ancestry_complete via auth_cut_context (returning None -immediately when ancestry is incomplete) and then delegates to member_path_at_cut to -handle three cases: -

-
    -
  • Direct membership — returns the folded role from the projection (the same derivation - already validated by the membership-role divergence plane).
    • -
  • Inherited{via_admin:true} — always resolves to Admin, - mirroring the live semantics for anchor-via-admin inheritance.
    • -
  • Inherited{via_admin:false} — returns the member's stored role at the - anchor group. Returns None rather than guessing when the anchor row is missing, - preventing spurious divergence reports.
    • -
-

-The method returns None — signalling "projection has no opinion, defer to live store" -— in four cases: ancestry is not yet fully folded, the member is absent from the group at the cut, -the auth_cut_context call cannot resolve the namespace, and the anchor row is missing -for an inherited-via-anchor path. Callers must not treat None as a role denial. -

- -

Shared-context identity resolution — ScopeProjections::member_identities_in_view_with_ctx

-

-member_identities_in_view_with_ctx accepts caller-supplied root and -default_cap_base instead of re-reading MetaRepository / -CapabilitiesRepository internally. member_identities_in_view delegates to -it unchanged (existing callers are unaffected). member_entries_with passes the values -it already obtained from auth_cut_context, ensuring the candidate-identity filter and -the per-member role resolution use a single consistent store snapshot with no window for a concurrent -governance write to make them disagree. Prefer member_identities_in_view_with_ctx over -member_identities_in_view whenever the caller already holds the root and capability -values — it avoids a redundant store read and eliminates the TOCTOU window. -

- -

Projection-first list_group_members with stable pagination

-

-The list_group_members handler now reads from the projection authoritatively when a -folded proj_ctx is available. It falls back to the live -MembershipRepository::list ∪ enumerate_inherited union only when -member_entries_with returns None. Previously the handler always read from -the live store and compared the projection in shadow mode; the shadow scaffolding methods -shadow_member_enum_with and member_roles_for have been deleted as they had -no remaining callers. -

-

-Both the projection path and the live-store fallback path now apply a -members.sort_by_key(|(identity, _)| *identity) sort by PublicKey -before the skip(offset).take(limit) slice. The projection returns a -BTreeSet-ordered (PublicKey-sorted) result while the live store returns -insertion/store order; without the explicit sort, a mid-backfill request served by the projection -followed by a next-page request served by the live store could skip or repeat members. The sort -makes pagination order stable and path-independent across a mid-backfill transition. -

- -

member_now_checked / member_now_checked_with

-

-ScopeProjections::member_now_checked and its shared-fold variant -member_now_checked_with are the two query-gate helpers that call -namespace_current_heads and then delegate to AclView::is_member_at_cut. -Their behavior follows a strict two-branch contract: -

-
    -
  • Projection returns Some — the result is acted on directly. There is no - secondary MembershipRepository::is_member live read and no - unified_projection_divergence cross-check. The projection verdict is authoritative - when it is available.
    • -
  • Projection returns None (unresolvable group or unreadable head) — fall - back to the live membership store (check_group_membership_path) as before.
    • -
-

-The earlier best-effort live cross-check (which compared the projection verdict against the live -resolver and emitted a warning/error on divergence) has been removed from both functions. The -None→live fallback path is preserved unchanged. -

- -

At-cut admin & capability accessors — is_admin_at_cut, is_admin_or_capability_at_cut, is_last_admin_at_cut

-

-Five additional methods on ScopeProjections expose admin/capability checks against a -historic DAG cut, mirroring what the live authorization gate does at apply time: -

-
    -
  • auth_cut_context(group_id, cut) — shared setup used by the public - accessors. Resolves the namespace for the supplied group, gates on - cut_ancestry_complete (returns None immediately if the cited ancestry - is not yet fully folded), builds the AclView folded to the cut, and returns the - genesis root tuple plus default-capability base.
    • -
  • is_admin_at_cut(signer, group_id, cut) — returns - Some(true) if the signer holds an admin role at the cut, - Some(false) if they do not, and None when ancestry is incomplete. - Callers must defer to the live gate on None — incomplete ancestry is not - equivalent to unauthorized.
    • -
  • is_admin_or_capability_at_cut(signer, group_id, cut, cap_bits) — - admin OR capability-bit check, mirroring the live - is_authorized_with_capability. Same None semantics as above. This - method is a required method on the AtCutAuthorizer trait; all implementors - must provide it.
    • -
  • is_last_admin_at_cut(member, group_id, cut) — answers, at the - op's parent cut (pre-mutation), whether removing or demoting the given member would - orphan the group's admin set. Mirrors live logic exactly: a member counts as admin if they - hold a direct Admin-role row or are the genesis admin; "another admin" requires a - distinct Admin-role row (the genesis admin alone does not satisfy it). Returns - None when ancestry is incomplete, deferring to the live last-admin guard. -

    - Causal-position semantics. Because this method reads the DAG state the signer observed - at sign time (the parent cut), not the receiver's current local state, it can legitimately - disagree with a live query made after concurrent ops have advanced the DAG — this is the - causal-honor property and is expected behavior, not a divergence. -
    • -
  • membership_path_at_cut(member, group_id, cut) — returns - Some(AtCutMembershipPath) indicating whether the member has a direct row, an - inherited path, or no membership at the cut, and None when ancestry is incomplete. - Delegates to ScopeProjections::membership_path_at_cut, which calls - member_path_at_cut on the folded projection via auth_cut_context and - maps the result (projecting away the role/anchor detail). Callers must defer to the live - gate on None.
    • -
-

-All public accessors return None (not Some(false)) when cited ancestry -is not fully folded, ensuring callers do not treat an incomplete fold as a denial verdict. -

- -

AtCutAuthorizer trait and implementations

-

-The AtCutAuthorizer trait abstracts over how an at-cut admin/capability check is -resolved. It declares four required methods: is_admin_at_cut, -is_admin_or_capability_at_cut, is_last_admin_at_cut, and -membership_path_at_cut. All return Option<…> with the same -None = undecidable semantics. -

-

-Empty-cut contract. Every method on this trait must return None when -the supplied parents slice is empty. Violating this contract would cause a -genesis-state fold (no MemberAdded or role ops processed) to produce -Some(false), falsely rejecting genesis ops that the live store correctly permits. -

-
    -
  • LiveFallbackAuthorizer — the placeholder implementation used by the - current namespace-envelope applier stage. Returns None for all four methods, - delegating all decisions to the live store.
    • -
  • EphemeralProjectionAuthorizer — the projection-backed implementation. - Holds a Mutex<Option<(ContextGroupId, Arc<FoldedProjection>)>> - cache. The private folded(group_id) method checks the cache first; on a cache - miss it calls ScopeProjections::ephemeral_projection and stores the result. If the - fold fails (store or DAG-head fault) it emits a tracing::warn and returns - None. All four at-cut methods go through folded, so a single apply - call reuses the same folded DAG rather than re-folding per check. - is_last_admin_at_cut delegates to - ScopeProjections::is_last_admin_at_cut and guards against an empty - parents slice with an immediate None return, consistent with the - trait-level empty-cut contract. - membership_path_at_cut delegates to - ScopeProjections::membership_path_at_cut, which calls the projection's - member_path_at_cut walk via auth_cut_context and maps the result to - AtCutMembershipPath (projecting away the role/anchor detail). An empty - parents slice returns None immediately, consistent with the - empty-cut contract.
    • -
- -

GroupGovernanceApplier — live gates only, no authorizer injection

-

-The standalone GroupGovernanceApplier::apply continues to call -apply_local_signed_group_op (live-fallback) and does not accept an injected -AtCutAuthorizer. A code comment documents why: a SignedGroupOp's -parent_op_hashes belong to the per-group op log, not the namespace governance log -that EphemeralProjectionAuthorizer is keyed by. Passing those hashes to the -projection authorizer would resolve against the wrong id-space and silently no-op, producing a -false "undecidable" verdict on every group op. The real group-auth projection shadow therefore -fires only on the namespace-envelope group-op path, not through the standalone applier. -

- -

UnifiedApplier — projection-backing DeltaApplier<Op>

-

-UnifiedApplier is the C2.0 struct that bridges DagStore<Op> and -ScopeProjections. It implements DeltaApplier<Op>: its -apply method acquires a write lock on the inner -Arc<std::sync::RwLock<ScopeProjections>>, calls -ScopeProjections::ingest_op with the delta's payload, then releases the lock. The -lock is a std::sync::RwLock (not a Tokio lock) and is never held across an -.await; lock poison is recovered rather than propagated, matching the stance -already documented on NodeState::read_scope_projections. Two constructors are provided: -

-
    -
  • UnifiedApplier::new() — creates a private, standalone projection. Use this for - replay pipelines and tests.
    • -
  • UnifiedApplier::with_projection(Arc<RwLock<ScopeProjections>>) — - attaches to the node's live shared projection. This is the C2.1 dual-write seam: governance ops - processed by the existing live pipeline are simultaneously ingested into the shared projection.
    • -
-

-A projection() accessor exposes the Arc for inspection or cross-checking. -pub mod unified_applier is exported from crates/context/src/lib.rs, -making UnifiedApplier part of the crate's public API. -

-

-Behavior is pinned by -dag_releases_out_of_order_ops_and_the_projection_converges: a four-op causal chain -spanning the admin, membership, ACL, and data planes is fed through DagStore<Op> -+ UnifiedApplier in four delivery orders (in-order, full reverse, and two interleaved). -The test asserts that every op is applied (none stuck pending) and that scope_root -matches the result of direct in-order folding in all cases — confirming causal ordering and -projection convergence regardless of receive order. -

- -

Shadow authorization check in apply_signed_namespace_op

-

-After a governance op is successfully applied by the live gate, the handler runs a non-blocking -shadow authorization check for observability. The check is implemented in two parts: -

-
    -
  • apply_auth_requirement(op) maps the just-applied - SignedNamespaceOp to an (ContextGroupId, ApplyAuthReq) pair. - ApplyAuthReq is either Admin or - AdminOrCap(bits). The mapping covers: -
      -
    • Root-level AdminChanged, PolicyUpdated, - GroupReparented → Admin
      • -
    • GroupCreated → AdminOrCap(CAN_CREATE_SUBGROUP)
      • -
    • Group-level MemberAdded / MemberRemoved → - AdminOrCap(MANAGE_MEMBERS)
      • -
    • SubgroupVisibilitySet → - AdminOrCap(CAN_MANAGE_VISIBILITY)
      • -
    • MemberRoleSet, MemberCapabilitySet, - DefaultCapabilitiesSet → Admin
      • -
    - Explicitly excluded from the mapping (returning None): - MemberJoined (inviter authority), MemberJoinedOpen (inheritance - proof), MemberLeft (self-authored), GroupDeleted and - TransferOwnership (owner-gated), and other metadata/context-registration ops. -
    • -
  • The handler then calls the corresponding at-cut accessor against the op's - delta_parents cut (the DAG state the signer observed, not the post-apply cut). - If the projection returns Some(false) — it would have rejected what the live - gate accepted — a warn! is emitted with - marker=unified_projection_divergence and plane=governance-auth. - None (ancestry incomplete) is silently skipped. The check runs outside the - projection write lock under a brief read lock.
    • -
-

-Live authorization gates are entirely unchanged. The shadow check is log-only and has no -effect on whether an op is accepted or rejected. -

- -

Concurrent-op convergence test — concurrent_independent_member_adds_converge

-

-The integration test concurrent_independent_member_adds_converge pins the CRDT -convergence semantics for concurrent ops authored against the same genesis state. Two admins -independently add different members (D and E) against the same genesis state; the ops are applied -in opposite orders on two nodes. The test asserts: (1) both ops succeed on both nodes, (2) both -nodes converge to the identical four-member set, and (3) replaying a seen op is a no-op — the -per-signer nonce window dedups it and the member count does not change. The only apply gate is the -per-signer nonce window; there is no state-hash staleness check. The old test -(state_hash_prevents_concurrent_op_divergence) asserted that the second concurrent op -must fail; that hard-reject behavior no longer exists and the old test has been removed. -

- -

Where this can break

-

-Two classes of change would silently re-introduce the divergence described above: -

-
    -
  • Calling the projection with the wrong position. Any caller of the at-cut membership - resolver that passes the receiver's current state (or any post-cut heuristic) instead of the - signed delta position breaks forward-only. The three production call sites — gossip-receive, - governance-pending drain, and snapshot-sync replay — each carry inline comments naming this - contract; reviewers should flag any new caller that doesn't pass - delta.governance_position verbatim.
    • -
  • Expanding the cut resolution to visit descendants. A "performance optimization" that - advances the resolved cut beyond the signed heads would observe ops the signer hadn't signed - against, including later MemberRemoved ops. The forward-only contract is - documented on the public projection at-cut methods; the - prefix_walk_forward_only_* regression tests pin the boundary.
    • -
  • Reintroducing a state-hash staleness gate. Governance ops no longer carry a - meaningful root-hash claim (state_hash has been removed from the synthesized - SignedGroupOp in decrypt_and_apply_group_op, and - signed_op_to_delta / signed_namespace_op_to_delta now pass - [0u8; 32] as expected_root_hash). Any new code that gates apply on a - non-zero root-hash match would incorrectly reject concurrent ops and break the CRDT convergence - property pinned by concurrent_independent_member_adds_converge. The per-signer - nonce window is the sole apply gate for replay deduplication.
    • -
- -

Scope

-

-Forward-only applies to MemberRemoved and MemberLeft — operations -that revoke authorship. It does not apply to role demotions -(MemberRoleSet transitioning a Member to ReadOnly): those are enforced against -current state at the receive path, separately from the cross-DAG check, because a member who -was demoted retains a different relationship to their pre-demotion writes than a member who -was fully removed. See is_read_only_for_context for the live-state role gate. -

-
- - -
-

leave_context — local-only opt-out

-

-Stop syncing a context on this node. No DAG op, no broadcast, no key rotation. Other members -never observe the leave; from their perspective the leaver is still in the membership list (which -is implicit anyway, computed from group membership + auto-follow). -

- -

Storage

-

-Two separate stores live side by side: the synced ContextIdentity row -(key at crates/store/src/key/context.rs:120, value at -crates/store/src/types/context.rs:117) and a node-local -ContextLeftMarker tombstone in a new column. -

-

-The marker lives in Column::ContextLocal (a new column -specifically for node-local context-scoped state — never synchronized). -Keying is identical to ContextIdentity so the -(context_id, member_pk) pair is the unit of opt-out: -

-
-pub struct ContextLeftMarker(Key<(ContextId, PublicKeyComponent)>);

-// Borsh value:
-pub struct ContextLeftMarker {
-    pub left_at_ms: u64,
-} -
-

-A separate column instead of a flag on ContextIdentity for two -reasons: (a) ContextIdentity is part of the synced membership -shape, so adding a node-local field to it would conflate replicated and -node-local state; (b) extending the existing Borsh value would break -deserialization of pre-existing on-disk rows, requiring a migration. -The dedicated column makes the node-local-ness explicit at the storage -layer and keeps the synced shape clean. -

- -

Flow

-
    -
  1. Client calls leave_context(context_id).
    2. -
  2. Handler resolves the local ContextIdentity via - find_local_signing_identity — scans the column for the row - where this node holds the private key. Falls back to the namespace - identity if no local row exists.
    3. -
  3. In a single batched store handle (so they land in one commit), the - handler: -
      -
    1. Writes the ContextLeftMarker row in - Column::ContextLocal.
      2. -
    2. Deletes the corresponding ContextIdentity row, - which stops sync (the sync layer iterates these rows).
      3. -
    -
    4. -
  4. Calls node_client.unsubscribe(&context_id) to leave - the gossipsub topic, mirroring join_context's subscribe.
    5. -
  5. Auto-follow handler (auto_follow.rs:255) checks for the - marker via has_left_context on every - ContextRegistered event and skips the auto-rejoin if it - finds one.
    6. -
- -

Properties

- - - - - - - - -
ScopeLocal only
DAG entryNone
Key rotationNone — leaver still holds the group key
CoordinationNone — cannot fail
ReversibilityTrivial — JoinContextRequest deletes the marker as a side-effect of joining and writes a fresh ContextIdentity row
Other members observeNo
Forward secrecyNot provided — leaver retains the key, can decrypt anything they sync later if they rejoin
- -

What this is not

-
    -
  • Not a kick. Leaver still appears in any "members of this context" query.
    • -
  • Not a cryptographic leave. If the user wants forward secrecy on a context, they must - leave the group containing it (which rotates the group key).
    • -
-
- - -
-

leave_group — self-removal, no cascade

-

-Self-removal from a single group via GroupOp::MemberLeft. The leaver publishes the -leave op; every receiver deletes the leaver's (member, group) row. -

-

-As-built caveat — no key rotation on self-leave. Unlike owner-initiated -MemberRemoved, MemberLeft deliberately does not trigger the -key-rotation pipeline. The publisher of a self-leave is the leaver, and the leaver cannot generate -the fresh group key without also retaining it — which would defeat forward secrecy. So -MemberLeft is the governance-level departure (the membership row is removed, peers -observe the leave, the leaver is deny-listed) without a re-key. The path to a -cryptographically-complete leave is an admin/owner-initiated MemberRemoved, which -rotates. A two-phase self-leave rotation (a remaining admin's apply hook publishes the new key) is -a tracked follow-up, not yet shipped. See the inline note in -crates/governance-store/src/ops/group/member_left.rs. -

- -

New op

-
-enum GroupOp {
-    // existing variants...
-    MemberLeft { member: PublicKey },
-} -
-

-Distinct from MemberRemoved for audit clarity — "this member chose to leave" vs -"this member was removed" are semantically different events even when they have similar effects. -

- -

Preconditions

-
    -
  • Signer equals the member being removed (SelfLeaveOnly) and has a direct row - in the group; reject with MemberNotDirect if the signer is only an inherited member — - they must leave the anchor instead.
    • -
  • Signer is not the group's owner_identity (OwnerCannotSelfLeave; must - TransferOwnership first).
    • -
  • Removing the signer does not leave the group with zero admins - (LastAdmin via ensure_not_last_admin_removal).
    • -
- -

Flow

-
    -
  1. Leaver's client calls leave_group(group_id).
    2. -
  2. Apply enforces the preconditions above (self-equality, direct-row, owner, last-admin).
    3. -
  3. Leaver publishes MemberLeft { member: signer } via the governance DAG. Signed - by the leaver. Targets this group.
    4. -
  4. Op broadcasts to all members of the group. Each receiver applies it: - cascade_remove_member_from_group_tree drops the leaver's ContextIdentity - rows, remove_member deletes the (leaver, group_id) membership row, and - the leaver is added to the group deny-list so their state-delta traffic is dropped until they - re-join.
    5. -
  5. The apply emits OpEvent::MemberRemoved; if the leaver's role was - ReadOnlyTee it additionally emits OpEvent::TeeMemberRemoved, which drives - the leaver's own node to self-purge the group's local key material.
    6. -
  6. No automatic re-key. As noted above, MemberLeft does not rotate the group - key — the leaver still holds the last key it received and can decrypt writes made under it. Forward - secrecy on future writes requires the owner-initiated MemberRemoved path (or the - deferred two-phase self-leave rotation).
    7. -
- -

Cascade behavior

- - - -
Direct rows in subgroups under this groupUntouched. Leaver keeps those memberships. (You can be invited to a child while explicitly leaving the parent.)
Inherited memberships in Open subgroupsResolve themselves. The walker stops finding the leaver once their parent row is gone — no extra work needed.
- -

Failure modes

-
    -
  • No forward secrecy on self-leave (by design, not a failure). Because - MemberLeft never rotates, the leaver retains the ability to decrypt writes made under - the key they last held. To revoke that, an admin/owner must issue MemberRemoved - (which rotates) or the deferred two-phase self-leave rotation must land.
    • -
  • Inherited-only membership. A signer whose membership is purely inherited (no direct row) - is rejected with MemberNotDirect; they must leave the anchoring ancestor instead.
    • -
- -

Rejoin

-
    -
  • Restricted group: requires fresh add_group_members from a remaining - admin. New keys delivered via the existing membership-add path.
    • -
  • Open group (if leaver had a redundant direct row): same. If leaver's path to the - group was via inheritance from a parent, that inheritance is automatically restored as soon as - they rejoin the parent.
    • -
-
- - -
-

leave_namespace — cascading, heaviest

-

-A namespace is the root group. Leaving it is a single MemberLeft at the namespace root -whose apply cascades through every descendant group where the leaver has a direct row. This is the -only leave operation that fans out across multiple scopes. As with leave_group, the -self-leave op does not rotate keys (same publisher-holds-the-key constraint); row removal + -deny-listing only. -

- -

Preconditions

-
    -
  • Signer has a direct row at the namespace root (MemberNotDirect otherwise).
    • -
  • Signer is not the owner_identity of the namespace root - (OwnerCannotSelfLeave) nor of any subgroup beneath it where they have a direct row - (OwnerOwnsSubgroup, reporting the offending scope).
    • -
  • Removing the signer doesn't leave any scope (namespace or descendant) admin-less. If it - would → LastAdmin reports the offending scope, forcing a successor promotion before - retry. All these checks run upfront, before any mutation.
    • -
- -

Flow

-
    -
  1. Client calls leave_namespace(namespace_id).
    2. -
  2. Apply detects the no-parent (namespace-root) case and walks the subtree: -
      -
    • Compute the descendant groups where the signer has a direct row.
      • -
    • Run owner check + last-admin check across all of them upfront. If any fails - (OwnerOwnsSubgroup, OwnerCannotSelfLeave, LastAdmin) it - bails before mutating anything — no half-applied cleanup.
      • -
    -
    3. -
  3. Signer publishes a single MemberLeft { member: signer } at the namespace root.
    4. -
  4. The apply cascades: for each direct-row descendant it calls - cascade_remove_member_from_group_tree, removes the membership row, deny-lists the - leaver, and emits OpEvent::MemberRemoved. For each descendant where the leaver's role - was ReadOnlyTee it also emits a per-subgroup OpEvent::TeeMemberRemoved. - The same removal then runs for the namespace root, emitting a root MemberRemoved (and - a root TeeMemberRemoved if the root role was ReadOnlyTee).
    5. -
  5. No per-scope key rotation. The cascade is row-removal only — there is no - KeyDelivery fan-out. Forward secrecy on each scope's future writes is provided by the - owner-initiated MemberRemoved rotation pipeline, not by self-leave.
    6. -
  6. Local cleanup on the leaver's node is role-scoped, not a soft/hard toggle: -
      -
    • Non-TEE roles: soft leave. Local rows (identity, signing keys, contexts) are kept so - rejoin / keyshare / inheritance-rejoin flows can reuse them.
      • -
    • ReadOnlyTee: the emitted TeeMemberRemoved events drive the - self-purge listener to hard-delete local key material across the - whole subtree and unsubscribe from the namespace gossipsub topic.
      • -
    -
    7. -
- -

Edge case — last member of the namespace

-

-If the leaver is the last remaining member, MemberLeft writes locally but has no -peers to broadcast to. The namespace effectively dies on the leaver's node. -

-

-If the leaver is also the owner AND last member, they're stuck in the cycle — they must transfer -ownership but there's no one to transfer to. Use a dedicated DeleteNamespace op -(analogous to DeleteGroup but at root) that bypasses the owner-cannot-leave rule. -

- -

Failure modes

-
    -
  • No forward secrecy on self-leave across any scope (same constraint as - leave_group): the cascade removes rows but does not re-key. Use owner-initiated - MemberRemoved for a cryptographically-complete eviction.
    • -
  • Interrupted TEE self-purge. If the leaver was a ReadOnlyTee node and its - local cascade-purge is interrupted (crash, or a signing-key delete error), a durable - pending-self-purge marker survives and the startup reconcile_sweep completes the purge - on next restart. See TEE self-purge.
    • -
-
- - -
-

Role-scoped TEE self-purge

-

-Removal alone (the MemberRemoved rotation) already provides forward secrecy on a -namespace's future writes — the evicted node never receives the rotated key. But a -ReadOnlyTee fleet node still holds, on its own disk, the old key material from before -the rotation: signing keys, the group encryption keys it last received, its -NamespaceIdentity, and the governance op-log. A regular member's local rows are -deliberately kept (the soft-leave invariant lets kick-and-readd / -rejoin-via-keyshare / inheritance-rejoin reuse them). A ReadOnlyTee node has -no rejoin path — re-admission via MemberJoinedViaTeeAttestation derives a fresh -attestation pubkey — so that material buys nothing and is pure hygiene debt. The self-purge handler -hard-deletes it. -

-

-Design record: docs/adr/0002-fleet-tee-leave-protocol.md. Implementation: -crates/context/src/self_purge.rs. Fleet-side framing: -Fleet HA. -

- -

Role-scoped trigger

-

-The apply path emits a generic OpEvent::MemberRemoved for every removal, and — only when -the removed member's stored role was ReadOnlyTee — an additional -OpEvent::TeeMemberRemoved (see ops/group/member_removed.rs and -member_left.rs). The self-purge listener subscribes to the op-apply event channel and -reacts only to TeeMemberRemoved: -

-
    -
  • Non-TEE removals (Admin/Member/ReadOnly) keep - soft-leave semantics — no purge. Hardening to hard-purge every removal would regress the - apps/scaffolding-e2e/workflows/group-{kick,leave}-* workflows that depend on the - retained rows.
    • -
  • ReadOnlyTee removals trigger a hard purge. The self-detection ("did this op - evict me?") is a handler concern, not part of the node-agnostic apply contract — it reads - this node's stored namespace identity (per-node state). This mirrors the - auto-follow architectural split.
    • -
- -

Subgroup vs namespace-root

-

-decide_purge_action (pure store reads) resolves the namespace owning the evicted group, -confirms the evicted member is this node's own identity, and then picks one of two actions: -

- - - - -
PurgeAction::SubgroupRemoved from one subgroup while still a - member of others under the same namespace. Purge only that group's local rows - (delete_group_local_rows) plus its context-index and tree-edge rows. Do not - unsubscribe from the namespace gossipsub topic — other memberships still need it.
PurgeAction::NamespaceRemoved from the namespace root. Cascade - the whole subtree, drop namespace-level state, then unsubscribe from the namespace topic.
PurgeAction::NoneEvent is for another member, or for a - namespace this node has no identity in. No action.
- -

What gets hard-deleted

-

-The cascade (cascade_namespace_state) iterates the root plus every descendant group and -calls delete_group_local_rows per group. That helper -(crates/governance-store/src/local_state.rs) deletes the membership rows, capabilities, -metadata, upgrade records, the group op-log + head, the deny-list, and — load-bearing — -the private key material: -

-
    -
  • Signing keys. SigningKeysRepository::delete_all_for_group — the 32-byte - private signing-key material. Leaking these is the actual forward-secrecy hazard, so this is the - step the failure-class gating treats as security-critical.
    • -
  • Group encryption keys. GroupKeyring::delete_all_for_group — the AES group - keys the node received while admitted (added in PR #2776). Without this the evicted node would - retain its copy of past group keys on disk even though the rotation already orphaned them for - future writes.
    • -
-

-Forward secrecy is therefore split cleanly: future writes are protected by key rotation -(re-key excluding the removed member, done by the publisher pipeline at removal); the evicted TEE -node's own copies of past keys are deleted here by self-purge. The purge does not — and need -not — trigger any rotation itself. -

-

-Namespace-level teardown (delete_namespace_local_state) then drops the -NamespaceIdentity, the namespace governance head, and the namespace op-log. Dropping the -identity and unsubscribing is gated on the signing-key purge succeeding (a best-effort -dead-pointer cleanup failure does not block the security-critical finalize). -

- -

Durable marker + startup reconcile

-

-TeeMemberRemoved fires once per eviction and an already-evicted identity receives no -further removal events, so a missed or partially-failed purge has no event-driven retry. Recovery is -a startup sweep, made role-safe by a durable marker (ADR 0002 / #2721): -

-
    -
  • Before the namespace cascade, handle_member_removed writes a durable - pending-self-purge marker for the namespace. This is the one site that knows — node-aware and - role-aware — that this is a confirmed TEE self-eviction of our identity.
    • -
  • On startup, reconcile_sweep enumerates markers only and completes a purge - iff (marker present) AND (still no surviving namespace-root membership). If the - identity is already gone, or we have been re-admitted, it clears the stale marker without purging; a - read error skips (never purge on uncertainty).
    • -
  • The marker is essential because, post-eviction, the role row is erased: a role-blind scan of - NamespaceIdentity rows could not distinguish evicted-TEE residue from a pending join or - non-TEE soft-leave residue, and would false-purge both. The marker is the role/intent gate; the - still-evicted check is the safety gate; both must hold.
    • -
-
- - -
-

Owner (stored identity, not a role)

-

-Owner is not a GroupMemberRole variant. It is a single identity stored on group -metadata: GroupMetaValue.owner_identity (crates/store/src/key/group/mod.rs). -Every group has exactly one Owner, and that member is otherwise an ordinary Admin — the -"Owner" status is whatever the apply handlers grant by comparing the signer's public key against -owner_identity. The distinction is a small set of exclusive privileges plus immunity -from involuntary removal; it is enforced by identity comparison, not by a role bit. -

- -

Storage

-

-The struct carries both fields: the legacy admin_identity (now a -fallback creator-admin marker for pre-existing groups) and the shipped owner_identity. -Newly-created groups initialize owner_identity == admin_identity. Field shape and -serialization are stable. -

- -

New op

-
-enum GroupOp {
-    // existing variants + MemberLeft...
-    TransferOwnership { new_owner: PublicKey },
-} -
-
    -
  • Signer must equal current owner_identity.
    • -
  • new_owner must already be a member of the group.
    • -
  • Apply: update owner_identity to new_owner. Old owner becomes a - regular admin (retains admin powers; doesn't lose them as a side-effect of transfer).
    • -
  • No key rotation (membership unchanged).
    • -
- -

Privilege matrix

- - - - - - - - - - - -
ActionMemberAdminOwner
Invite members (with CAN_INVITE_MEMBERS)✓✓✓
Create context (with CAN_CREATE_CONTEXT)✓✓✓
Remove non-owner members (MANAGE_MEMBERS)—✓✓
Promote member → admin—✓✓
Demote admin → member—✓1✓
TransferOwnership——✓
DeleteGroup / DeleteContext / DeleteNamespace——✓
Be involuntarily removed (MemberRemoved)✓✓✗ — OwnerImmuneFromRemoval
Self-leave via leave_group / leave_namespace✓✓ (last-admin permitting)✗ — OwnerCannotSelfLeave / OwnerOwnsSubgroup
-

-1 Open question. The existing code allows admins to demote each other -(mod.rs:823); restricting to Owner-only would tighten centralization. This proposal -preserves the current behavior (admins can demote each other), but the doc author may want to -revisit. -

- -

Owner per context

-

-Same model. CreateContext records its creator as owner_identity on -the context's metadata. The context owner can TransferContextOwnership and -DeleteContext. Other than that, regular context membership is governed by the -parent group as before. -

- -

Interactions with leave operations

-
    -
  • leave_context: no owner constraint. Local opt-out doesn't touch - governance — even an owner can mute a context locally.
    • -
  • leave_group: rejected for owner. Error fast-paths to - TransferOwnership.
    • -
  • leave_namespace: rejected if signer owns the namespace OR any subgroup - beneath it. Error lists every owned scope. UI may bundle these into a single multi-transfer - flow.
    • -
-
- - -
-

Single-Page Reference

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
OpScopeDAG?Key rotation?Cascade?Reversible?
leave_contextLocal onlyNoneNonen/aTrivial — flip flag
leave_groupSingle groupMemberLeftNo (self-leave; deferred)NoneRe-invite (Restricted) / re-inherit (Open)
leave_namespaceWhole subtreeMemberLeft at rootNo (self-leave; deferred)All direct-row scopesRe-invite at root + re-inherit
MemberRemoved (owner/admin)Single group (cascades contexts)MemberRemovedYes — re-key excludes removedContext identitiesRe-invite (re-key delivered on add)
ReadOnlyTee removal + self-purgeSubgroup or whole namespaceMemberRemoved + TeeMemberRemoved eventYes (via removal) + local key deleteSubtree on namespace-rootNone under same identity (fresh attestation)
TransferOwnershipSingle group/contextOpNone (membership unchanged)NoneTransfer back
DeleteGroup (Owner)Single group + descendantsOpn/a (everyone gets removed)YesNone — destructive
DeleteNamespace (Owner)Whole subtreeOpn/aYesNone — destructive
-
- - -
-

Implementation Status

-

-This page documents shipped behavior. Each row points at the code that implements it. -

- - - - - - - - - - - - - - - - - - - - - - - -
SectionStatusCode
Membership Model RecapDocuments existing code—
leave_contextImplementedcrates/context/src/handlers/leave_context.rs, auto_follow.rs gate, POST /admin-api/contexts/:id/leave
leave_groupImplementedGroupOp::MemberLeft, crates/governance-store/src/ops/group/member_left.rs, POST /admin-api/groups/:id/leave — apply enforces self-leave + direct-row + owner + last-admin. No key rotation on self-leave (deferred two-phase rotation).
leave_namespaceImplementedSame member_left.rs apply detects the no-parent case and cascades through descendant groups where the leaver has direct rows; multi-scope owner + last-admin checked upfront. No per-scope key rotation.
Key rotation on MemberRemovedImplementedcrates/governance-store/src/group_governance_publisher.rs (sign_apply_and_publish_inner) + GroupKeyring in group_keys.rs — fresh key wrapped for remaining members, excluding the removed one.
Role-scoped TEE self-purgeImplemented (ADR 0002; #2680/#2724/#2725)crates/context/src/self_purge.rs listener + cascade + marker-gated reconcile_sweep; delete_group_local_rows in governance-store/src/local_state.rs deletes signing keys (and, as of #2776, GroupKeyring encryption keys); OpEvent::TeeMemberRemoved emitted from member_removed.rs / member_left.rs.
Startup curative sweep (redrive_stranded_ops_sweep)Implemented (#2848 Part C)crates/context/src/self_purge.rs — one-shot sweep on startup after op_events subscribe; iterates namespaces via NamespaceRepository::iter_identities, finds decryptable-but-unapplied groups via NamespaceRetryService::groups_with_held_key_buffered_ops, re-drives via redrive_encrypted_ops_for_group_counted; converges up to MAX_PASSES=64 with pass-narrowing. auto_follow::spawn called before self_purge::spawn in ContextManager::started (load-bearing ordering — see note above).
Born-Open atomic subgroup create (RootOp::GroupCreated restricted field)Implemented (#2771) — WIRE BREAKcrates/primitives/src/context.rs (RootOp::GroupCreated), crates/governance-store/src/ops/group/group_created.rs (group_created::apply) — restricted: bool field added; apply writes subgroup visibility via CapabilitiesRepository::set_subgroup_visibility before queuing OpEvent::SubgroupCreated; gated on has_subgroup_visibility absence. CreateGroupRequest and REST visibility field updated. Op-adapter projection reads restricted from live op (no longer hardcodes true).
TEE subgroup admission diagnostics (handle_new_subgroup)Implementedcrates/context/src/tee_subgroup_admit.rs — handle_new_subgroup now emits debug! on entry (subgroup + namespace IDs) and at each early-return path (Open subgroup skip, non-key-holder skip, no-root-TEE-members skip); warn! when a TEE member has no admission verdict in the root op-log; info! just before admitting a TEE member into a Restricted subgroup. Set RUST_LOG=calimero_context=debug (or debug on the context crate) to surface per-subgroup TEE admission decisions in logs.
ScopeProjections::namespace_current_headsImplementedcrates/context/src/scope_projections.rs — resolves a ContextGroupId to its namespace, reads the namespace DAG head, and returns parent hashes as the current-state cut argument. Returns None on unresolvable group or unreadable head; callers fall back to check_group_membership_path.
ScopeProjections::member_now_ephemeralImplementedcrates/context/src/scope_projections.rs — builds a fresh projection per call: resolves the namespace, folds the DAG via collect_namespace_ops, reads heads after the fold, and returns member_at_cut. Returns None (never decides against the member) on unresolvable namespace, store fault, or incomplete fold.
ScopeProjections::member_now_checkedImplementedcrates/context/src/scope_projections.rs — unified membership gate for all four context-layer handlers. Calls member_now_ephemeral first; falls back to MembershipRepository::is_member when projection is undecidable; emits tracing::warn with marker=unified_projection_divergence / plane=membership-query on definite disagreement. Direct MembershipRepository imports removed from get_cascade_status.rs, get_group_upgrade_status.rs, list_all_groups.rs, and list_group_contexts.rs.
ScopeProjections::scope_root_for + group_scope_root_ephemeralImplementedcrates/context/src/scope_projections.rs — scope_root_for computes SHA-256(entities_root ‖ acl_hash ‖ governance_hash) for a folded scope; returns None when the scope is unfolded (callers must treat as "cannot compare", not divergence). group_scope_root_ephemeral builds a fresh ephemeral projection from the raw store, applies an ancestry-completeness guard, and delegates to scope_root_for; returns None mid-backfill. ACL component is currently empty pending C2 SetWriters routing. Pinned by scope_root_for_moves_on_hash_neutral_membership_change_and_is_none_when_unfolded.
ScopeProjections::role_at_cut_for_groupImplementedcrates/context/src/scope_projections.rs — resolves the effective GroupMemberRole at a governance cut via auth_cut_context + member_path_at_cut. Handles direct, inherited-via-admin (Admin), and inherited-via-anchor (anchor role row) cases. Returns None on incomplete ancestry, absent member, unresolvable namespace, or missing anchor row. Prevents spurious membership-role divergence reports by abstaining rather than guessing on missing anchor rows.
UnifiedApplier + pub mod unified_applierImplementedcrates/context/src/unified_applier.rs (exported via crates/context/src/lib.rs) — UnifiedApplier wraps Arc<std::sync::RwLock<ScopeProjections>> and implements DeltaApplier<Op>; apply holds the write lock only across the synchronous ScopeProjections::ingest_op call, never across an .await; lock poison is recovered. new() for standalone/test use; with_projection(arc) as the C2.1 dual-write seam; projection() accessor exposes the Arc. Convergence pinned by dag_releases_out_of_order_ops_and_the_projection_converges (four-op causal chain, four delivery orders, all converge to the same scope_root).
unified_op_store — persist_op + load_scope_opsImplemented (no production call sites yet)crates/context/src/unified_op_store.rs (exported as pub mod unified_op_store from crates/context/src/lib.rs) — persist_op Borsh-serializes an Op and writes it under ScopeUnifiedOpKey in Column::UnifiedOp; idempotent because the key is content-addressed by op id. load_scope_ops performs a seek-then-entries range scan over the scope prefix and deserializes all rows back into Vec<Op>. Rows are returned in key order, not causal order — callers must replay the loaded ops through DagStore<Op> + UnifiedApplier to recover causal ordering and rebuild projections; heads are deliberately not persisted. No production apply or sync path calls these functions yet. Behavior is pinned by persisted_op_log_reloads_and_replays_to_the_same_projection: persists a three-op causal chain (AdminChanged → MemberAdded → SetWriters), reloads from a fresh store handle, replays through DagStore<Op> + UnifiedApplier, and asserts the reconstructed scope_root matches an independent in-memory fold; also asserts loading an empty scope returns zero rows.
ScopeProjections::check_op_store_completenessImplemented (diagnostic-only observe gate; retires at C3 Stage 4)crates/context/src/scope_projections.rs — accepts an already-computed gov-DAG fold (dag_ops) and calls load_scope_ops to compare op-id sets. Emits op_store_incomplete warn on gaps; emits op_store_gate_unavailable and returns None when the store is unreadable. Returns None (unverifiable), Some(vec![]) (verified complete), or Some(ids) (missing ids). Pinned by completeness_gate_flags_governance_ops_missing_from_the_op_store in crates/context/tests/op_store_reconstruction.rs.
ScopeProjections::scope_root_from_op_store + ScopeProjections::shadow_compare_op_storeImplemented (observe-only; C2.2b gate)crates/context/src/scope_projections.rs — scope_root_from_op_store loads all persisted ops for a given ScopeId via load_scope_ops, folds them into a fresh ScopeProjections instance, and returns the governance scope_root using a fixed zero entities_root. Returns Ok(None) when no ops exist for the scope; Err on store faults. This is the op-store's independent view of a scope's governance root, intended for completeness verification before C2.2b. shadow_compare_op_store is an observe-only method that compares the maintained (governance-DAG) projection's scope_root against the op-store reconstruction for a given namespace. On mismatch it logs a tracing::warn with marker=unified_op_store_divergence and truncated hex of both roots — this log marker is the e2e gate signal for C2.2b readiness. Silently skips (no divergence reported) when the scope is unfolded, the op-store is empty for it, or the store read fails. calimero_op/ScopeId types remain entirely inside the context crate and are not leaked into the node crate. Behavior is pinned by two additional assertions in the existing persisted_op_log_reloads_and_replays_to_the_same_projection test: one verifying scope_root_from_op_store equals the expected in-memory fold result, and one verifying it returns None for a scope that has no persisted ops.
AtCutAuthorizer trait + LiveFallbackAuthorizer + EphemeralProjectionAuthorizerImplementedcrates/context/src/governance_dag.rs — AtCutAuthorizer declares four required methods: is_admin_at_cut, is_admin_or_capability_at_cut, is_last_admin_at_cut, and membership_path_at_cut. All must return None when parents is empty (trait-level empty-cut contract). LiveFallbackAuthorizer implements all four returning None. EphemeralProjectionAuthorizer holds a Mutex<Option<(ContextGroupId, Arc<FoldedProjection>)>> fold cache; the private folded helper reuses a cached fold for the same group or calls ScopeProjections::ephemeral_projection on a miss. All four at-cut methods delegate through folded; each opens with an empty-parents guard. is_last_admin_at_cut delegates to ScopeProjections::is_last_admin_at_cut (in crates/context/src/scope_projections.rs), which mirrors live last-admin logic at the parent cut. membership_path_at_cut delegates to ScopeProjections::membership_path_at_cut, which calls member_path_at_cut on the folded projection via auth_cut_context and maps the result to AtCutMembershipPath. The standalone GroupGovernanceApplier::apply is unchanged (live-only); see the GroupGovernanceApplier note in the at-cut accessors section above.
Owner identity + TransferOwnershipImplementedGroupMetaValue.owner_identity, GroupOp::TransferOwnership (ops/group/transfer_ownership.rs), OwnerImmuneFromRemoval / OwnerCannotSelfLeave gates in MemberRemoved / MemberLeft.
Namespace genesis op (RootOp::NamespaceCreated)Implementedcrates/context/src/handlers/create_group.rs (parent_group_id == None branch) — signs, applies, and publishes RootOp::NamespaceCreated { founder: admin_identity } via sign_apply_and_publish_namespace_op; NoPeersSubscribed from publish treated as success; any other error triggers rollback of all locally-written root rows (meta, founder Admin member row, default caps, group key, signing key, optional name metadata) via GroupKeyring::delete_key_by_id, SigningKeysRepository::delete_key, MetaRepository::delete, MembershipRepository::remove_member, CapabilitiesRepository::delete_default, MetadataRepository::delete_group; namespace identity row deliberately excluded from rollback so retries reuse the same identity/founder; DAG head is head-atomic and does not require rollback.
-
- - - -
-

Startup curative sweep — redrive_stranded_ops_sweep

-

-At node startup, a one-shot curative sweep re-drives any encrypted ops that are buffered but -decryptable — that is, the node holds the group key but the op was never applied, typically -because the GroupCreated event that would have triggered delivery arrived late or -was dropped. Without this sweep such ops remain stranded forever. -

- -

Where it runs

-

-redrive_stranded_ops_sweep is called from self_purge::run, after -subscribing to op_events and before entering the main event loop. It blocks the -event loop until the sweep finishes, but errors are logged and swallowed — the sweep never -prevents startup. -

- -

Algorithm

-
    -
  1. Call known_namespace_identities (via - NamespaceRepository::iter_identities) to enumerate every namespace this node has - an identity for.
    2. -
  2. For each namespace, call - NamespaceRetryService::groups_with_held_key_buffered_ops (the inverse of - groups_awaiting_key) to find groups whose key is already held but which still have - buffered ops.
    3. -
  3. Re-drive each group via redrive_encrypted_ops_for_group_counted, which applies - as many buffered ops as possible and returns a count of newly applied ops.
    4. -
  4. Loop until a full pass applies nothing new (cross-signer convergence), bounded by - MAX_PASSES = 64. Pass-narrowing means only groups that made progress in the - previous pass are revisited, avoiding redundant work.
    5. -
- -

Spawn ordering — load-bearing

-

-ContextManager::started calls auto_follow::spawn before -self_purge::spawn. This ordering is intentional and load-bearing: -auto_follow::spawn subscribes to op_events synchronously on the caller -thread before spawning its async task, and passes the receiver into auto_follow::run(). -Because the curative sweep (inside self_purge::run) may emit -OpEvent::ContextRegistered for previously stranded contexts, the auto-follow receiver -must exist before those events are broadcast — otherwise they are silently dropped -by the broadcast channel. Reversing the spawn order would silently break auto-follow for any -context recovered by the startup sweep. -

-
-

Reviewers: both call sites in ContextManager::started -carry inline comments documenting the ordering constraint. Any refactor that reorders -auto_follow::spawn and self_purge::spawn must preserve the -subscribe-before-sweep invariant.

-
- -

Relevant symbols

-
    -
  • redrive_stranded_ops_sweep — orchestrator; lives in - crates/context/src/self_purge.rs
    • -
  • redrive_buffered_ops_for_group — inner per-group apply loop
    • -
  • redrive_encrypted_ops_for_group_counted — wraps the above and returns - a count of newly applied ops for pass-narrowing
    • -
  • namespace_groups_with_held_key_buffered_ops — store query (inverse of - groups_awaiting_key)
    • -
  • NamespaceRetryService::groups_with_held_key_buffered_ops
    • -
  • NamespaceRepository::iter_identities
    • -
-
- - -
-

Born-Open atomic subgroup create — RootOp::GroupCreated wire change

-
-

⚠ WIRE BREAK. RootOp::GroupCreated has a new Borsh -field (restricted: bool). Nodes running this version cannot exchange governance ops -with nodes running the previous wire format. Nodes must reset their governance -op-log before upgrading.

-
- -

Problem

-

-Previously, the GroupCreated op carried no visibility information. The -tee_subgroup_admit subscriber fired on every OpEvent::SubgroupCreated and -wrote a transient direct ReadOnlyTee membership row for Open subgroups — a row that -would need to be retracted once a SubgroupVisibilitySet(Open) op arrived. This -transient row could cause spurious TEE admission flows. -

- -

Fix — restricted field on GroupCreated

-

-RootOp::GroupCreated gains a restricted: bool field: -

-
    -
  • restricted: true (default) — subgroup is born Restricted. Legacy behavior is - preserved; no wire difference for groups created this way.
    • -
  • restricted: false — subgroup is born Open. The apply handler writes the - subgroup's visibility key via CapabilitiesRepository::set_subgroup_visibility - during apply, before OpEvent::SubgroupCreated is queued. - tee_subgroup_admit sees the group as already Open when it reacts, skips it, and - no transient direct row is written.
    • -
-

-The write is gated on CapabilitiesRepository::has_subgroup_visibility absence so -later SubgroupVisibilitySet flips are never clobbered. -

- -

API surface changes

-
    -
  • CreateGroupRequest gains a restricted: bool field (default - true).
    • -
  • The create-group-in-namespace REST endpoint accepts an optional - visibility field ("open" | "restricted", default - "restricted").
    • -
  • The op-adapter projection reads restricted from the live op instead of - hardcoding true.
    • -
- -

Relevant symbols

-
    -
  • RootOp::GroupCreated — crates/primitives/src/context.rs
    • -
  • group_created::apply — crates/governance-store/src/ops/group/group_created.rs
    • -
  • CapabilitiesRepository::has_subgroup_visibility, - CapabilitiesRepository::set_subgroup_visibility
    • -
  • CreateGroupRequest, payload_from_root_op
    • -
-
- -
-

Open Questions & Follow-ups

-
    -
  1. Two-phase self-leave rotation. MemberLeft currently does not re-key (the - leaver can't generate a key without retaining it). A deferred design has a remaining admin's apply - hook publish the new key after the leave. Until it lands, owner-initiated MemberRemoved - is the only rotating leave.
    2. -
  2. Local cleanup is role-scoped, not a toggle. The earlier soft-vs-hard default question is - resolved by role (ADR 0002): non-TEE removals stay soft; ReadOnlyTee removals - hard-purge. No global default knob.
    3. -
  3. Admin-mutual demote. Admins can demote each other. Whether to restrict this to - Owner-only is unresolved; current behavior is preserved.
    4. -
  4. Old owner after TransferOwnership. Becomes a regular admin (current - behavior) or just a member (alternative)?
    5. -
  5. Subgroup-level self-purge reconcile. The startup reconcile is namespace-level only; a - subgroup-only purge has no reconcile retry surface yet (tracked in #2726).
    6. -
  6. Periodic reconcile. A purely-lagged-drop of TeeMemberRemoved writes no - marker and is not recovered until a future eviction; a periodic sweep is an open follow-up.
    7. -
-
- -

EphemeralProjectionAuthorizer — empty-parents early return

-

-Both is_admin_at_cut and is_admin_or_capability_at_cut on -EphemeralProjectionAuthorizer now return None immediately when the -supplied parents slice is empty, deferring unconditionally to the live fallback -authorizer. Previously, an empty parents slice was passed through to the fold -machinery, which evaluated the query against a genesis-state view. That view contains no granted -roles or capabilities, so any capability holder whose grant was established after genesis would be -falsely rejected by the projection even though the live store correctly recognises them. -

- -

When parents is empty

-

-Two cases produce an empty parents slice at the authorizer call site: -

-
    -
  • Genesis op. The very first op in a namespace's governance log has no - predecessor hashes. The causal cut is genuinely empty because no prior ops exist.
    • -
  • PermissionChecker constructed without apply-auth context. Some code paths - build a PermissionChecker outside of a normal apply flow and pass an empty parents - field as a zero value. These callers have no meaningful causal position to resolve against.
    • -
-

-In both cases, judging against an empty fold is semantically wrong: the fold has not processed any -MemberAdded, MemberRoleSet, or DefaultCapabilitiesSet ops, -so the resulting AclView is empty. A query against it would return -Some(false) for every capability check, silently blocking operations that the live -store would correctly permit. -

- -

New behavior

-
    -
  • If parents.is_empty(), both methods return None immediately, - before touching the fold cache or calling ScopeProjections::ephemeral_projection.
    • -
  • The None return propagates to the caller as "projection is undecidable," which - causes the standard live-fallback path to take effect. No warning is emitted — empty parents is - a legitimate undecidable case, not a divergence.
    • -
  • When parents is non-empty, behavior is unchanged: the fold cache is consulted - (or populated on a miss) and the at-cut check proceeds normally.
    • -
- -

Contract clarification

-

-The AtCutAuthorizer contract already specifies that None means -"undecidable — defer to the live gate." This change makes the empty-parents case an explicit -instance of that contract rather than a silent fall-through to a semantically incorrect verdict. -Callers must not treat None as a denial; the live authorizer is always consulted when -the projection abstains. -

-

-The None-on-incomplete-ancestry guard (from cut_ancestry_complete) is -distinct and remains in place: it fires when parents is non-empty but the fold has -not yet processed all ops reachable from those parents. The empty-parents guard fires earlier, -before the fold is attempted at all. -

- -

Affected code

-
    -
  • crates/context/src/governance_dag.rs — - EphemeralProjectionAuthorizer::is_admin_at_cut and - EphemeralProjectionAuthorizer::is_admin_or_capability_at_cut: both now open with - if parents.is_empty() { return None; } before the folded(group_id) - call.
    • -
  • No changes to ScopeProjections or any call sites — the early return is - entirely internal to the authorizer methods. LiveFallbackAuthorizer is unaffected - (it already returns None unconditionally for all methods, including the new - is_last_admin_at_cut).
    • -
-
-

Deferred op-store read-flip — ops_for_namespace indirection layer

- -

-Status: C2.2b deferred. The unified op-store (load_scope_ops) is -not yet the source of truth for projection rebuilds. All projection-rebuild call sites -now route through a new stable indirection method, ScopeProjections::ops_for_namespace, -which currently delegates to collect_namespace_ops (the governance-DAG walk). The -flip to load_scope_ops is explicitly deferred until the late-decrypted membership -bug described below is resolved. -

- -

ops_for_namespace — the stable indirection point

-

-ScopeProjections::ops_for_namespace(namespace_id, store) is the single dispatch -point used by every projection-rebuild site: -

-
    -
  • backfill_namespace in scope_projections.rs
    • -
  • member_now_ephemeral / ephemeral_fold in - scope_projections.rs
    • -
  • The scope_root reconstruction path in scope_projections.rs
    • -
  • refresh_projection_for_cut in the node crate
    • -
-

-All four previously called collect_namespace_ops directly. They now call -ops_for_namespace instead. collect_namespace_ops remains as the -underlying implementation and is unchanged; ops_for_namespace is a thin wrapper -that will be flipped to call load_scope_ops at C2.2b without requiring changes at -any call site. -

- -

Why the flip was blocked — late-decrypted membership (resolved by C2.2c)

-

-An encrypted MemberAdded op can arrive at a node before that node holds the group -key. At apply time the op is processed as Noop and written to the op-store with a -Noop payload. When the group key later arrives — via a keyshare or a -KeyDelivery event — the governance-DAG walk (collect_namespace_ops) -re-decrypts the op at read time and correctly surfaces it as MemberAdded. -Previously, the op-store entry was not updated; it still recorded Noop. -

-

-Consequence: if ops_for_namespace were flipped to load_scope_ops before -this was fixed, any member whose MemberAdded op was frozen as Noop at -write time would silently disappear from the projection after a rebuild. This was confirmed by e2e -timeouts on the joiner's post-write sync in the group-3node, -scaffolding-e2e, and shared-storage rotation workflows. -

-

-Fix: ScopeProjections::repersist_namespace_ops. After any group key -is delivered, the key-delivery path calls -ScopeProjections::repersist_namespace_ops(namespace_id, store), which re-walks the -namespace through collect_namespace_ops (the canonical read-time decryption path) and -calls persist_op for every op it finds. Because ops are content-addressed by the same -op id, a now-decryptable op (e.g. MemberAdded) overwrites the stale Noop -that was frozen at apply time when the key was absent. The method is best-effort (errors are logged -but not propagated), idempotent, and bounded by the same MAX_BACKFILL_OPS walk cap -used elsewhere. -

- -

ScopeProjections::check_op_store_completeness — diagnostic completeness gate

-

-ScopeProjections::check_op_store_completeness(dag_ops, store) is a diagnostic-only -observe gate that compares the op-ids from an already-computed gov-DAG fold (dag_ops) -against the op-ids loaded from the unified op-store via load_scope_ops. For every op -present in the gov-DAG fold but absent from the op-store it emits an op_store_incomplete -tracing::warn event carrying missing_count, dag_count, -op_store_count, and a hex-encoded sample_missing id. If the op-store -cannot be loaded at all it emits a distinct op_store_gate_unavailable marker and -returns None so a store fault cannot masquerade as a verified-clean result. -

-

Return semantics:

-
    -
  • None — op-store was unreadable; completeness is unverified. Callers must not - treat this as a clean result.
    • -
  • Some(vec![]) — verified complete; every gov-DAG op-id is present in the - op-store.
    • -
  • Some(ids) — those op-ids are present in the gov-DAG fold but missing from the - op-store.
    • -
-

-This method is diagnostic-only and has no effect on whether ops are accepted or rejected. -It retires at C3 Stage 4 when the authoritative read path flips onto the op-store. The companion -regression test -completeness_gate_flags_governance_ops_missing_from_the_op_store in -crates/context/tests/op_store_reconstruction.rs builds three governance ops, persists -only two to the op-store, calls check_op_store_completeness with all three as the -dag-fold, and asserts the return is Some(vec![op3.id]); after persisting the third it -asserts the return becomes Some(vec![]), confirming the None vs -empty-vec distinction works correctly. -

- -

Removed: shadow_compare_op_store and scope_root_from_governance_dag

-

-The observe-only shadow_compare_op_store method on ScopeProjections -and its companion scope_root_from_governance_dag have been removed, along with the -call site in refresh_projection_for_cut that triggered a full governance-DAG fold -on every backfill. -

-

-The comparison these methods performed was structurally flawed: the maintained projection holds -live ingest_op-fed ops that may not yet be written to the op-store. A gap in the -op-store could therefore be masked by the live apply path, making the divergence invisible to the -cross-check. The marker=unified_op_store_divergence log signal described in earlier -sections of this page is no longer emitted; the C2.2b readiness gate is now the -op_store_reconstruction_matches_governance_dag_for_late_decrypted_membership -regression test described below. -

- -

Regression test — op_store_reconstruction.rs

-

-The integration test file -crates/context/tests/op_store_reconstruction.rs is the CI regression gate for this -fix. The test is named -op_store_reconstruction_recovers_late_decrypted_membership_after_key_delivery and is -structured as an explicit before/after: -

-
    -
  1. Injects an encrypted MemberAdded op into the governance DAG with the group key - absent at write time, persisting the op-store entry as Noop.
    2. -
  2. Before re-persist: asserts that load_scope_ops reconstruction - sees Some(false) for the member (the frozen Noop state).
    3. -
  3. Calls repersist_namespace_ops to simulate key delivery.
    4. -
  4. After re-persist: asserts that the op-store reconstruction matches the - governance-DAG fold — the member is now Some(true).
    5. -
-

-The test runs in CI without #[ignore] and serves as the C2.2c readiness gate. The -module-level doc comment in the test file explains the full bug chain and the fix. -

- -

Summary of changes

-
    -
  • ScopeProjections::ops_for_namespace introduced; all four projection-rebuild - call sites updated from collect_namespace_ops to ops_for_namespace.
    • -
  • collect_namespace_ops unchanged — still the implementation behind - ops_for_namespace.
    • -
  • shadow_compare_op_store and scope_root_from_governance_dag - removed from ScopeProjections; marker=unified_op_store_divergence is - no longer emitted.
    • -
  • refresh_projection_for_cut call site that triggered a shadow fold on every - backfill deleted.
    • -
  • ScopeProjections::repersist_namespace_ops added; called by the key-delivery - path after a group key arrives to overwrite stale Noop op-store entries with their - now-decryptable payloads. Best-effort, idempotent, bounded by MAX_BACKFILL_OPS.
    • -
  • crates/context/tests/op_store_reconstruction.rs updated: test renamed to - op_store_reconstruction_recovers_late_decrypted_membership_after_key_delivery, - #[ignore] removed, body restructured into an explicit before/after assertion around - a repersist_namespace_ops call.
    • -
-
-

ScopeProjections::persist_namespace_head_ops — mirroring authored ops into the op-store

- -

-After a local authoring step publishes a governance op (GroupCreated, -GroupDeleted, MemberJoinedOpen), the unified op-store must be updated so -that any subsequent load_scope_ops-based projection rebuild sees the authored op -without relying on a peer retransmission. The method -ScopeProjections::persist_namespace_head_ops(store, namespace_id) is the single -call site for this update. -

- -

Why not a heads-only walk

-

-A naive implementation would read the namespace's current DAG heads immediately after publish and -persist only those ops. This races: if a concurrent peer op is applied between the local -publish call and the heads read, the head record advances past the just-authored op. -The authored op is then unreachable from a heads-only walk and is silently omitted from the -op-store. The member or group it describes disappears from any rebuild that sources ops from -load_scope_ops. -

- -

Implementation — full gov-DAG fold via repersist_namespace_ops

-

-persist_namespace_head_ops delegates unconditionally to -repersist_namespace_ops, which performs a full -collect_namespace_ops-backed fold bounded by MAX_BACKFILL_OPS. This -walk is read-time decryption-aware, so an authored op that was encrypted at publish time is -decoded here and written to the op-store with its actual payload (not as Noop). The -walk captures the authored op regardless of where it landed in the DAG relative to concurrent -peer ops, because collect_namespace_ops traverses the full ancestry from the current -head rather than just the head itself. -

-

-The call is: -

-
    -
  • Idempotent. persist_op is content-addressed by op id; calling - it twice for the same op is a no-op.
    • -
  • Best-effort. Errors are logged by repersist_namespace_ops - (using the same tracing::warn paths as the key-delivery re-persist) and are - never propagated. A failure to update the op-store never causes the authoring step to return - an error to the caller.
    • -
  • Bounded. The walk is capped at MAX_BACKFILL_OPS, the same - limit used by every other collect_namespace_ops-backed path. An unusually large - namespace cannot cause an unbounded fold at authoring time.
    • -
- -

Unified authoring wrapper — sign_apply_and_publish_group_op

-

-Rather than calling persist_namespace_head_ops individually in each handler, -all group-op authoring is routed through a single crate-internal async wrapper: -crate::sign_apply_and_publish_group_op. It calls -calimero_governance_store::sign_apply_and_publish and, on success, resolves the -namespace from the group ID via NamespaceRepository::resolve and calls -ScopeProjections::persist_namespace_head_ops. A namespace-resolve failure logs a -tracing::warn identifying the group ID but does not fail the authoring operation. -

-

-Callers must not call calimero_governance_store::sign_apply_and_publish -directly for group ops. Using the wrapper ensures the op-store is always updated at the -single chokepoint and that namespace-resolve failures are observable via tracing warnings. -

- -

Wired call sites

-

-The method is called after the governance op is successfully published or applied, never before. -A failure at the authoring step is not affected: -

-
    -
  • create_group — called after GroupCreated is - successfully reported to the governance pipeline.
    • -
  • delete_group — called after GroupDeleted is - applied.
    • -
  • join_context — called after the MemberJoinedOpen - op is published. The call is skipped (effectively a no-op path) if the publish or apply step - errored.
    • -
  • join_subgroup_inheritance — called after - MemberJoinedOpen is published, following the same skip-on-error pattern.
    • -
  • All other group-op handlers — add_group_members, - admit_tee_node, create_context (two call sites), - delete_context, detach_context_from_group, leave_group, - leave_namespace, set_member_auto_follow, - set_member_capabilities, set_subgroup_visibility, - set_tee_admission_policy, update_member_role, upgrade_group - (four call sites), dispatch_cascade, and the generic governance handler in - lib.rs all call crate::sign_apply_and_publish_group_op instead of the - underlying store function.
    • -
-

-Exception — remove_group_members. The key-rotation removal path uses -sign_apply_and_publish_removal (not the standard publish function) and cannot use the -wrapper. After each removal it inlines the same namespace-resolve + -persist_namespace_head_ops logic directly. A namespace-resolve failure logs a -tracing::warn identifying the group ID. -

- -

Regression test

-

-The test -persist_namespace_head_ops_lands_locally_authored_op_in_the_op_store in -crates/context/tests/op_store_reconstruction.rs verifies the full round-trip: -

-
    -
  1. An encrypted MemberAdded op is injected into the governance DAG (op-log + - DAG head) but not into the op-store, simulating the local-author gap.
    2. -
  2. The test asserts that load_scope_ops returns no entry for the op before the - call — confirming the gap exists.
    3. -
  3. persist_namespace_head_ops is called.
    4. -
  4. The test asserts the op now appears in the op-store.
    5. -
  5. A fresh projection is folded from the op-store contents and - member_at_cut is called. The test asserts it returns Some(true) — - proving the encrypted payload was decoded correctly, not merely that the op id was recorded.
    6. -
-

-The test runs without #[ignore] in CI and is the readiness gate for any future flip -of ops_for_namespace to source from load_scope_ops at authoring call -sites. -

- -

Relationship to repersist_namespace_ops

-

-repersist_namespace_ops was introduced to recover late-decrypted -MemberAdded ops after key delivery (see the deferred op-store read-flip section -above). persist_namespace_head_ops reuses that same mechanism for a different trigger -— local authoring — ensuring both the key-delivery path and the authoring path produce a -consistent op-store state. The two call sites share the same underlying fold, the same error -handling, and the same MAX_BACKFILL_OPS bound. -

-
-

Atomic op-store write — removal of per-site dual-write scaffolding

-

-Status: shipped. store_signed_operation now persists every -governance op atomically at apply time, making all per-site op-store write calls redundant. This -section documents what was removed and why, and supersedes the -Deferred op-store read-flip and -persist_namespace_head_ops sections above where they describe the old -per-site scaffolding. -

- -

What was removed

-
    -
  • Receive-handler dual-write. The persist_op call inside - apply_signed_namespace_op that was labelled C2.1b dual-write has been - deleted. The ingest_op fold into the live projection is deliberately kept; only - the now-redundant durable-store write is gone.
    • -
  • sign_apply_and_publish_group_op wrapper. The crate-private - function in lib.rs that wrapped - calimero_governance_store::sign_apply_and_publish and appended a - persist_namespace_head_ops call is deleted entirely. All ~15 call sites - (add_group_members, admit_tee_node, create_context, - delete_context, detach_context_from_group, join_context, - leave_group, leave_namespace, set_member_auto_follow, - set_member_capabilities, set_subgroup_visibility, - set_tee_admission_policy, update_member_role, - upgrade_group, dispatch_cascade, and others) now call - calimero_governance_store::sign_apply_and_publish directly.
    • -
  • ScopeProjections::persist_namespace_head_ops. The thin alias - for repersist_namespace_ops that documented the C3 Stage 1/2/3 per-site - persist pattern is deleted from scope_projections.rs. All call sites are also - removed: create_group, delete_group, join_context, - join_subgroup_inheritance, remove_group_members, - join_namespace (node crate), reparent_group (admin API), and - create_group_in_namespace (admin API).
    • -
- -

Why the per-site scaffolding was safe to delete

-

-store_signed_operation is called on every apply path — both local authoring and -peer-receive — and now writes the op into the unified op-store as part of the same atomic -commit. Because the persist is guaranteed at the apply chokepoint, no handler needs to issue a -separate post-publish persist call. The two problems the scaffolding addressed are resolved at -source: -

-
    -
  • Local-author gap. A locally authored op is written by - store_signed_operation at apply time, before the handler returns. There is no - window in which the op could be absent from the op-store after a successful publish.
    • -
  • Late-decrypted membership. repersist_namespace_ops (called by - the key-delivery path) still overwrites stale Noop entries with their - now-decryptable payloads. The atomic write path records the best payload available at apply - time; the key-delivery re-persist corrects any entry frozen as Noop before the - group key arrived. Both paths remain in place; only the redundant per-site calls are removed.
    • -
- -

Race that the old heads-only walk would have introduced

-

-The deleted persist_namespace_head_ops performed a full -collect_namespace_ops-backed fold after publish to avoid a heads-only-walk race -(a concurrent peer op advancing the head past the just-authored op before the walk ran). The -atomic write in store_signed_operation sidesteps this entirely: the authored op is -written at apply time under its content-addressed op id, so its presence in the op-store is -independent of where the DAG head points afterward. -

- -

Surviving use of repersist_namespace_ops

-

-repersist_namespace_ops itself is not deleted. It continues to -serve the key-delivery path: when a group key arrives late, the key-delivery handler calls -repersist_namespace_ops to re-walk the namespace via -collect_namespace_ops and overwrite any op-store entries that were frozen as -Noop at apply time because the key was absent. This is the only remaining call site. -

- -

Updated test — locally_authored_op_lands_in_the_op_store_atomically

-

-The test previously named -persist_namespace_head_ops_lands_locally_authored_op_in_the_op_store is renamed -locally_authored_op_lands_in_the_op_store_atomically. Its structure is updated to -reflect the atomic-write path: -

-
    -
  1. An encrypted MemberAdded op is passed through store_signed_operation - (the atomic path).
    2. -
  2. The test asserts the op appears in the op-store immediately after that call, with - no intervening persist_namespace_head_ops call required — authoring handlers need - only call calimero_governance_store::sign_apply_and_publish directly; no post-author - mirror step is needed.
    3. -
  3. The test then calls ScopeProjections::repersist_namespace_ops directly - (replacing the now-deleted persist_namespace_head_ops alias) and asserts the - op-store state is unchanged — confirming idempotency over an already-present entry.
    4. -
-

-The test runs without #[ignore] in CI and is the readiness gate confirming that the -atomic-write path is sufficient and that the per-site scaffolding carried no load that -store_signed_operation does not now cover. -

-
-
-
- - - diff --git a/architecture/metrics-reference.html b/architecture/metrics-reference.html deleted file mode 100644 index cfbac887b8..0000000000 --- a/architecture/metrics-reference.html +++ /dev/null @@ -1,510 +0,0 @@ - - - - -Metrics Reference — Calimero Core Architecture - - - - -
-
-
Home/Metrics Reference
- -

Metrics Reference

-

All Prometheus metrics organized by subsystem

- -
-
16
sync metrics
-
5
channel metrics
-
2
context metrics
-
3
self-purge metrics
-
- - -
-

Sync Metrics

-

Metrics emitted by the sync engine covering protocol selection, data transfer, phase timing, and invariant monitoring.

-crates/node/src/sync/prometheus_metrics.rs - -

Transfer & Volume

- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MetricTypeLabelsDescription
sync_messages_sentFamily<Counter>protocolTotal sync protocol messages sent
sync_bytes_sentFamily<Counter>protocolTotal sync protocol bytes sent
sync_round_tripsFamily<Counter>protocolTotal sync round trips
sync_entities_transferredCounter—Total entities transferred during sync
- -
- Transfer metric usage notes -
-
sync_messages_sent — High values per protocol indicate frequent sync activity. Compare across protocols to understand which sync strategy dominates. Alert if any single protocol accounts for >90% of messages (may indicate stuck fallback).
-
sync_bytes_sent — Use with sync_messages_sent to compute average message size. Snapshot protocol will naturally produce larger values. A sudden spike may indicate a node falling behind and triggering full snapshots.
-
sync_round_trips — Higher round trips for hash comparison is normal (binary search). For snapshot sync, this should be very low (1-2). High round trips on level-wise sync suggests deep DAG divergence.
-
sync_entities_transferred — Track the rate of this counter. Sustained high rates indicate active state divergence across the cluster. Should be near-zero during steady state.
-
-
- -

CRDT Operations

- - - - - - - - - - - - - - - - -
MetricTypeLabelsDescription
sync_mergesFamily<Counter>protocolTotal CRDT merge operations
sync_comparisonsCounter—Total entity hash comparisons
- -
- CRDT metric usage notes -
-
sync_merges — Each merge represents a conflict resolution. High merge rates relative to entity transfers indicate heavy concurrent writes. Useful for tuning write concurrency.
-
sync_comparisons — Hash comparisons are cheap but high volume indicates the hash-compare protocol is doing a lot of work. The ratio of comparisons to entities transferred measures sync efficiency (lower ratio = more divergence).
-
-
- -

Phase Timing

- - - - - - - - - - - - - - - - -
MetricTypeLabelsDescription
sync_phase_duration_secondsFamily<Histogram>phaseDuration of sync phases (buckets: 1ms, 2ms, 4ms, 8ms, 16ms, 32ms, 64ms, 128ms, 256ms, 512ms, 1s, 2s, 4s, 8s, 16s)
sync_duration_secondsFamily<Histogram>protocolFull sync session duration (buckets: 10ms, 20ms, 40ms, 80ms, 160ms, 320ms, 640ms, 1.28s, 2.56s, 5.12s, 10.24s, 20.48s, 40.96s, 81.92s, 160s)
- -
- Timing metric usage notes -
-
sync_phase_duration_seconds — Break down which phase is the bottleneck. Phases include: discovery, comparison, transfer, and apply. If the "apply" phase is slow, storage writes may be the bottleneck. If "transfer" is slow, check network bandwidth.
-
sync_duration_seconds — The end-to-end duration of a complete sync session. Hash comparison should complete in <100ms; level-wise in <1s; snapshot may take 10s+. Alert on p99 > 30s for any protocol.
-
-
- -

Invariant Monitoring

- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MetricTypeLabelsDescription
sync_snapshot_blockedCounter—Snapshot blocked on initialized nodes (invariant I5)
sync_verification_failuresCounter—Snapshot verification failures (invariant I7)
sync_lww_fallbackCounter—LWW fallback events
sync_buffer_dropsCounter—Delta buffer drop events (invariant I6 risk)
- -
- Invariant metric usage notes -
-
sync_snapshot_blocked — Should always be 0 in normal operation. Any increment means a snapshot sync was attempted on an already-initialized node, which is blocked by safety invariant I5. Investigate immediately if non-zero.
-
sync_verification_failures — Indicates post-snapshot state hash mismatch (invariant I7). Non-zero values suggest data corruption or non-deterministic execution. Requires urgent investigation.
-
sync_lww_fallback — LWW (last-writer-wins) fallback is a safety net when normal merge fails. Occasional occurrences are acceptable; sustained increases may indicate a bug in CRDT merge logic.
-
sync_buffer_drops — Delta buffer is bounded; drops mean incoming deltas are arriving faster than they can be processed (invariant I6 risk). May lead to data loss requiring catch-up sync. Alert on any increment.
-
-
- -

Protocol Selection & Outcomes

- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MetricTypeLabelsDescription
sync_attemptsFamily<Counter>protocolTotal sync attempts by protocol
sync_successesFamily<Counter>protocolSuccessful syncs by protocol
sync_failuresFamily<Counter>protocolFailed syncs by protocol
sync_protocol_selectionsFamily<Counter>protocolProtocol selection decisions
- -
- Protocol outcome usage notes -
-
sync_attempts / sync_successes / sync_failures — Compute success rate per protocol: sync_successes / sync_attempts. Hash comparison should have >95% success rate. If snapshot failures are non-zero, check network stability and storage health.
-
sync_protocol_selections — Shows which protocol the adaptive selector chose. Healthy clusters should see mostly hash comparison. Increasing snapshot selections indicates growing state divergence across nodes.
-
-
-
- - -
-

Network Event Channel Metrics

-

Metrics from the bounded event channel between the network layer and node manager. All metrics are registered under the network_event_channel sub-registry.

-crates/node/src/network_event_channel.rs - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MetricTypeDescription
depthGaugeCurrent events waiting in channel
received_totalCounterTotal events sent to channel
processed_totalCounterTotal events received from channel
dropped_totalCounterEvents dropped (channel full)
processing_latency_secondsHistogramEvent send-to-processing latency
- -
- Channel metric usage notes -
-
depth — Instantaneous queue depth. Sustained high values (>80% of capacity) indicate the NodeManager can't keep up with incoming network events. May cause drops.
-
received_total / processed_total — The difference received_total - processed_total should equal depth + dropped_total at any point. Growing divergence indicates processing lag.
-
dropped_total — Any increment means the channel was full and events were lost. Dropped events may cause missed gossip messages or delayed sync triggers. Alert on any non-zero rate.
-
processing_latency_seconds — Measures end-to-end event latency from when NetworkManager enqueues to when NodeManager dequeues. p99 > 500ms indicates backpressure. p50 should be < 10ms in healthy systems.
-
-
-
- - -
-

Context Metrics

-

Per-context execution metrics for monitoring WASM runtime performance.

-crates/context/src/metrics.rs - - - - - - - - - - - - - - - - - -
MetricTypeLabelsDescription
execution_countFamily<Gauge>context_idContext runtime execution counter
execution_duration_secondsFamily<Histogram>context_idExecution duration (buckets: 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s, 1024s)
- -
- Context metric usage notes -
-
execution_count — A gauge tracking cumulative executions per context. Use rate() to get executions per second. Contexts with unusually high execution rates may be under heavy client load or in a retry loop.
-
execution_duration_seconds — Measures WASM method execution time including host function calls and storage I/O. Bucket boundaries start at 1s — anything in the first bucket is healthy. Executions > 64s may indicate infinite loops or expensive storage operations. Consider setting WASM gas limits.
-
-
-
- - -
-

Governance metrics (planned)

-

These Prometheus-style names are reserved for future governance observability. They are not necessarily registered in the codebase yet; treat this as a specification placeholder.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MetricTypeLabelsDescription
calimero_governance_ops_applied_totalCounter—Total governance operations applied successfully
calimero_governance_ops_rejected_totalCounterreason (suggested)Total governance operations rejected (authorization failures, stale state, validation, etc.)
calimero_governance_dag_headsGaugegroup_id (suggested)Current number of DAG heads per group
calimero_governance_heartbeat_mismatches_totalCounter—Heartbeat head comparison failures
- -
Planned — Implementations may use different label sets or histograms; align exported names with this table when adding instrumentation.
-
- - -
-

TEE Self-Purge Metrics

-

Observability for the self-purge cleanup that runs when a hardware-attested fleet node is evicted from its ReadOnlyTee role. Self-purge hard-deletes a node's own local rows and keys (signing-key material and, as of #2776, the per-group encryption keys) and is recovered across restarts by a marker plus a startup reconcile sweep. See Membership & Leave and TEE Fleet HA.

-crates/governance-store/src/metrics.rs - -
Naming — These families register under the context → group_store sub-registries, so the exported Prometheus names carry the context_group_store_ prefix (e.g. context_group_store_self_purge_failures_total). The registered names are shown below.
- - - - - - - - - - - - - - - - - - - - - - - -
MetricTypeLabelsDescription
self_purge_failures_totalFamily<Counter>branch, classSelf-purge local-state cleanup failures. branch ∈ {namespace, subgroup}; class ∈ {signing_key, context_cleanup}. The class="signing_key" series is the security-relevant one (forward-secrecy residue on disk); class="context_cleanup" is a best-effort dead-pointer leak and is informational.
self_purge_reconcile_totalFamily<Counter>outcomeStartup reconcile-sweep outcomes, one increment per marked namespace processed. outcome ∈ {reconciled, retained, cleared_stale, stale_clear_failed, skipped}. retained stuck across restarts means a signing-key purge keeps failing; read-uncertainty lands in skipped (not in self_purge_failures_total).
self_purge_events_dropped_totalCounter—Self-purge op-events dropped by the broadcast Lagged arm (subscriber fell behind). An upper bound on dropped eviction events, each of which writes no reconcile marker — un-recoverable on-disk residue (bounded; not a forward-secrecy hole, which is held by key rotation).
- -
- Self-purge metric usage notes -
-
self_purge_failures_total — Alert on any non-zero rate of class="signing_key": it means a node's own private signing-key material may linger on disk after eviction, pending the reconcile sweep. class="context_cleanup" is informational (orphaned dead-pointer rows); namespace deletion and unsubscribe still proceed.
-
self_purge_reconcile_total — In steady state the startup sweep should be quiet. A namespace persistently in outcome="retained" across restarts indicates a signing-key purge that keeps failing — investigate. stale_clear_failed is benign (re-evaluated next restart); skipped means the sweep declined to purge under read uncertainty (never purge on uncertainty).
-
self_purge_events_dropped_total — Any non-zero rate means the self-purge subscriber fell more than the broadcast capacity behind and some evictions may have left un-reconcilable on-disk residue. Bounded and not a forward-secrecy hole, but worth alerting as a correctness signal.
-
-
-
- - -
-

Dashboard Queries

-

Example PromQL queries for common monitoring scenarios. Adapt the label selectors to match your deployment.

- -

Sync Health Overview

-
-# Sync success rate by protocol (5m window)
-sum(rate(sync_successes_total{protocol=~".*"}[5m])) by (protocol)
-/ sum(rate(sync_attempts_total{protocol=~".*"}[5m])) by (protocol) -
- -
-# Sync failure rate — alert when > 0.1/s
-sum(rate(sync_failures_total[5m])) by (protocol) > 0.1 -
- -
-# P99 sync duration by protocol
-histogram_quantile(0.99, sum(rate(sync_duration_seconds_bucket[5m])) by (le, protocol)) -
- -

Invariant Alerts

-
-# CRITICAL: Any snapshot verification failure
-increase(sync_verification_failures_total[5m]) > 0 -
- -
-# WARNING: Delta buffer drops (I6 risk)
-rate(sync_buffer_drops_total[5m]) > 0 -
- -
-# WARNING: Snapshot blocked on initialized node (I5)
-increase(sync_snapshot_blocked_total[5m]) > 0 -
- -

Network Channel Backpressure

-
-# Channel utilization (assuming capacity of 1000)
-network_event_channel_depth / 1000 -
- -
-# Event drop rate — alert on any drops
-rate(network_event_channel_dropped_total[1m]) > 0 -
- -
-# P99 event processing latency
-histogram_quantile(0.99, sum(rate(network_event_channel_processing_latency_seconds_bucket[5m])) by (le)) -
- -

Context Execution Performance

-
-# Top 5 contexts by execution rate
-topk(5, rate(execution_count[5m])) -
- -
-# P95 execution duration per context
-histogram_quantile(0.95, sum(rate(execution_duration_seconds_bucket[5m])) by (le, context_id)) -
- -
-# Slow executions (> 30s) per context
-sum(rate(execution_duration_seconds_bucket{le="32"}[5m])) by (context_id)
-- sum(rate(execution_duration_seconds_bucket{le="1024"}[5m])) by (context_id) -
- -

Sync Bandwidth & Efficiency

-
-# Bytes per message (average message size)
-rate(sync_bytes_sent_total[5m]) / rate(sync_messages_sent_total[5m]) -
- -
-# Sync efficiency: entities per round trip
-rate(sync_entities_transferred_total[5m]) / sum(rate(sync_round_trips_total[5m])) -
- -
-# Protocol selection distribution
-sum(rate(sync_protocol_selections_total[5m])) by (protocol)
-/ ignoring(protocol) group_left sum(rate(sync_protocol_selections_total[5m])) -
- -

TEE Self-Purge Health

-
-# CRITICAL: signing-key residue left on disk after eviction
-increase(context_group_store_self_purge_failures_total{class="signing_key"}[15m]) > 0 -
- -
-# WARNING: a namespace stuck "retained" — repeated purge failure
-increase(context_group_store_self_purge_reconcile_total{outcome="retained"}[1h]) > 0 -
- -
-# WARNING: dropped self-purge events — un-recoverable residue risk
-rate(context_group_store_self_purge_events_dropped_total[5m]) > 0 -
-
- -
-
- - - diff --git a/architecture/migrations.html b/architecture/migrations.html deleted file mode 100644 index 528818437f..0000000000 --- a/architecture/migrations.html +++ /dev/null @@ -1,968 +0,0 @@ - - - - -Writing App State Migrations — Calimero Core Architecture - - - - -
-
-
Home/Migrations
- - -

Writing App State Migrations

-

Reshape a context's existing data when an app's state shape changes — convergently, lazily, and recoverably

- -
-
Lazy
on next access
-
Converge
pure fn of v1 state
-
Ladder
multi-hop catch-up
-
Resync
stranded recovery
-
- -

-When you ship a new version of a Calimero app whose state shape changes, you -write a migration that reshapes each context's existing data into the new layout. This -guide covers when you need one, how to write it (the easy way and the hand-written way), the one -rule that actually matters (convergence), how the SDK keeps you from getting it wrong, and how to -test and ship it. -

-
Audience: app developers using calimero-sdk. For -the internals, see crates/sdk/AGENTS.md.
- - -
-

1. Do you even need a migration?

-

-A migration is needed only when old serialized state can no longer be read as the new state. Borsh -is positional, so: -

- - - - - - - - - -
ChangeMigration?
Add a method, fix logic (no field change)No
Append a variant to an enum (kept indices)No
Add a fieldYes — old bytes have no value for it
Remove / rename a fieldYes
Change a field's typeYes
-

-If you're unsure, the calimero-abi diff CI lint compares the old and new -state-schema.json and tells you whether the change is additive, breaking, or an unsafe -identity downgrade (see §6). -

-

-A migration runs once per node, the first time each context is accessed after the -upgrade, under the LazyOnAccess upgrade policy. -

-
- - -
-

2. Quick start — #[derive(Migrate)]

-

-For the common cases — add / remove / rename a field — you don't write the -migration body at all. Declare the new state, derive Migrate, point it at the old -layout, and annotate only what changed: -

-
use calimero_sdk::app;
-use calimero_sdk::borsh::BorshDeserialize;
-use calimero_storage::collections::{LwwRegister, UnorderedMap};
-
-#[app::state(version = 2)]
-#[derive(app::Migrate)]
-#[migrate(from = DocV1Data)]
-pub struct DocV2 {
-    entries: UnorderedMap<String, LwwRegister<String>>,  // carried automatically
-    title:   LwwRegister<String>,                         // carried automatically
-    #[migrate(new = LwwRegister::new("".to_owned()))]
-    notes:   LwwRegister<String>,                         // added → you give the seed
-}
-
-// The old layout, as a borsh-only shadow. Field order MUST match v1's
-// `#[app::state]` struct (borsh is positional). Don't import the v1 crate's
-// `#[app::state]` — it would pull in v1's full SDK surface.
-#[derive(BorshDeserialize)]
-#[borsh(crate = "calimero_sdk::borsh")]
-struct DocV1Data {
-    entries: UnorderedMap<String, LwwRegister<String>>,
-    title:   LwwRegister<String>,
-}
- -

The derive generates migrate_v1_to_v2() for you. The rules:

- - - - - - - - - - -
AnnotationWhereResult
(none)fieldcarried from the old state by name (old.field)
#[migrate(new = EXPR)]fieldadditive — you provide the seed value
#[migrate(from = old_name)]fieldrenamed — carry old.old_name
#[migrate(with = EXPR)]fieldtransform — EXPR(old.field) (combine with from to convert a renamed field). Handles type changes, struct→enum, etc.
field omitted from the new struct—dropped (the remove case)
#[migrate(emit = EXPR)]structemit an app event from the migration (e.g. Migrated { from, to })
-

-with and emit cover the two most common reasons to drop to a hand-written -body — a type change and an event — so much of what used to need -#[app::migrate] is now a one-line annotation. Example: -

-
#[app::state(emits = for<'a> MigrateEvent<'a>)]
-#[derive(app::Migrate)]
-#[migrate(from = V1, method = migrate_v1_to_v2,
-          emit = MigrateEvent::Migrated { from: "1.0.0", to: "2.0.0" })]
-pub struct V2 {
-    items: UnorderedMap<String, LwwRegister<String>>,   // carried
-    #[migrate(from = count, with = u64_reg_to_string)]
-    count: LwwRegister<String>,                          // u64 -> String via `with`
-}
-fn u64_reg_to_string(c: LwwRegister<u64>) -> LwwRegister<String> {
-    LwwRegister::new(c.get().to_string())
-}
-

-A new field you forget to annotate is a compile error ("no field notes -on the old type") — it can't silently misbuild. A dropped field, by contrast, is silent -(that's the remove case), so review your new field list against the old one deliberately. -

-

-method defaults to the versioned name migrate_v{N-1}_to_v{N} derived from -#[app::state(version = N)] (plain migrate only for unversioned states), so -omitting it is the norm — names stay unique across releases and across multiple derives in one -module. Explicit method = … remains supported and resolves identically. -

-
- - -
-

3. Writing a migration by hand

-

-When you need real transformation — a type change, splitting a field, seeding from -other data — write the #[app::migrate] function yourself. The derive isn't magic; it -just generates this shape: -

-
use calimero_sdk::state::read_raw;
-
-#[app::migrate]
-pub fn migrate_v1_to_v2() -> DocV2 {
-    // 1. Read the old root bytes (None only if no prior state exists).
-    let old_bytes = read_raw().unwrap_or_else(|| panic!("no prior state"));
-
-    // 2. Deserialize into the old-layout shadow.
-    let old: DocV1Data = BorshDeserialize::deserialize(&mut &old_bytes[..])
-        .unwrap_or_else(|e| panic!("v1 deserialize: {e:?}"));
-
-    // 3. Build and return the new state.
-    DocV2 {
-        entries: old.entries,                       // carry a collection — handle survives
-        title:   old.title,
-        notes:   LwwRegister::new("".to_owned()),   // seed a new field
-    }
-}
-

Notes:

-
    -
  • A migration returns the new state by value; there is no Result. - On unrecoverable input (no prior state, undeserializable bytes) you panic!, which - aborts the upgrade and leaves v1 state intact for a retry — a failed migration is - non-destructive.
    • -
  • Carrying a collection (entries: old.entries) reuses its existing storage handle; - no re-population needed.
    • -
- -

When you must hand-write it

-

-The derive (with with / emit) handles a single-field transform, a type -change, a struct→enum, and emitting an event. You still need a hand-written -#[app::migrate] when one source feeds many fields or a new -field is derived from a field you're also keeping — i.e. the transform crosses fields: -

- - - - - - - - -
You need to…Why the derive can'tExample scenario
Split one field into severala with yields one field, not threescenario-field-split
Re-derive a value from a field you also carrythat source would be moved twicescenario-invariant-reshuffle
Seed an ordered collection from another you carrysame double-use, plus ordering logicscenario-crdt-native
Anything genuinely multi-step / imperative——
-

-(A single-field type change, struct→enum, content transform, or archiving a dropped -field is now a #[migrate(with = …)] one-liner — see §2.) -

-

-These are all in apps/migrations/ as scenario-* crate pairs with a matching -merobox workflow — use them as a cookbook. -

- -

Common hand-written patterns

-

Split a field — parse one field into several, with an explicit fallback:

-
let parts: Vec<&str> = old.address.get().split(", ").collect();
-let (street, city, zip) = match parts.as_slice() {
-    [s, c, z] => (s.to_string(), c.to_string(), z.to_string()),
-    _ => (old.address.get().clone(), String::new(), String::new()), // handle malformed input
-};
-

-Seed an ordered structure (determinism!) — building a Vector from a -map you also carry; sort first, because two nodes may iterate an unordered source in -different orders: -

-
let mut keys: Vec<String> = old.items.entries()?.map(|(k, _)| k).collect();
-keys.sort();                                   // ← required, or the roots diverge
-let mut tags = Vector::new();
-for k in keys { tags.push(k.into())?; }
-DocV2 { items: old.items, tags, .. }           // `items` carried + re-read above
-

-Drop entries — remove from the carried collection; don't rebuild a fresh one. -A new same-named UnorderedMap::new() is re-keyed to that field's deterministic id during -migrate, so it shares the carried v1 storage and unions with it — the entries you -"skipped" survive, so nothing is dropped. To actually drop, mutate the carried collection (sort first -for determinism): -

-
let mut items = old.items;                     // carry — same storage id
-let mut keys: Vec<String> = items.entries()?.map(|(k, _)| k).collect();
-keys.sort();
-if let Some(k) = keys.first() { items.remove(k)?; }   // actually deletes one entry
-DocV2 { items, .. }
-
-For a single-field type change, struct→enum, or content transform, prefer -#[migrate(with = …)] (§2) — these hand-written patterns are for the cross-field cases the -derive can't express. All of them run under the convergence rules in §4 and the category rules in §5, -the same as a derived migration. The merge-mode guards (Counter/RGA) still apply. -
-
- - -
-

4. The convergence rule (the important part)

-

-#[app::migrate] runs independently on every node, against that node's -own (already-synced, byte-identical) v1 state. The migrated state is not sent over -sync — each node re-derives it locally. Therefore: -

-
-A migration's output must be a deterministic, pure function of the old state. Two -nodes running the same migration on the same v1 bytes must produce a byte-identical v2 root, or they -fork and CRDT sync breaks. -
- -

What the SDK handles for you

-

-A migration body runs under storage merge mode, and the SDK removes the two -structural sources of per-node entropy automatically: -

-
    -
  • Node-local timestamps. LwwRegister::new(...) / .set(...) - and Element update times are zeroed during a migration (instead of stamping this node's - clock + id).
    • -
  • Random collection ids. New collections and Vector/AuthoredVector - elements are re-keyed deterministically (by field name / append index) instead of the - Id::random() the live path uses.
    • -
-

-So a migration that only carries fields and adds new ones with ::new() -cannot trigger a determinism bug. -

- -

What you must still avoid (app-level)

-
    -
  • Wall-clock / RNG / iteration order. If you materialize an ordered - structure (a Vector) from an unordered one (a map/set), sort first - — two nodes may iterate the source in different orders.
    • -
  • See the data categories below for the CRDT-specific rules.
    • -
-
- - -
-

5. The three data categories

-

Which migration moves are safe depends on the field type:

- - - - - - - -
CategoryTypesIn a migration
ConvergentUnorderedMap, Vector, UnorderedSet, UserStorage, FrozenStoragekey/content-addressed — rebuild freely; auto-converges
ReplayableCounter/GCounter/PNCounter, RGAcarry across, or replay deterministically — see below
Identity-gatedAuthoredMap, AuthoredVector, SharedStoragecarry-through only — re-inserting stamps this node as owner and diverges
- -

Replayable: Counter and RGA are guarded

-

-Counter::increment / decrement stamp the running node's id; -RGA::insert / insert_str stamp a node-local clock. Calling them in a -migration would silently fork the network — so the SDK makes them panic during a -migration: -

-
Counter::increment() is non-deterministic during a state migration: it stamps
-this node's identity… Carry the counter across unchanged (`field: old.field`)
-or replay with `increment_for(executor_id, …)`.
-

-To rebuild one in a migration, carry it (c: old.c) or use the deterministic replay APIs -— increment_for(id, …), decrement_for(id, …), -insert_str_at_timestamp(pos, fixed_ts, s) — which take the identity/clock explicitly so -every node produces the same result. -

- -

Identity-gated: carry, don't re-insert

-

-AuthoredMap/AuthoredVector/SharedStorage record ownership as -the running node's executor_id. Re-inserting their entries during a migration would -stamp each node as the owner and diverge. Carry the whole collection through -(entries: old.entries); the v1 owner stamps are preserved. Inside the -#[app::migrate] body, carry-through is still the only rule — converting an entry to the -new schema happens after the migration, owner-by-owner (below). -

- -

Identity-gated: converting entries to the new schema (owner-driven)

-

-Carry-through preserves the v1 entries, but each keeps its v1 schema_version (a -Merkle-invisible tag) until its owner re-signs it — nobody can re-sign another -identity's entry. Until then the entry is served at its v1 shape via dual-read. Two paths re-stamp an -entry to the binary's target: -

-
    -
  • Organically — the owner's (or a current writer's) next ordinary signed - write of that entry stamps schema_version = target and re-signs on its monotonic - nonce, replicating as a normal Action::Update. A non-owner can never drive it.
    • -
  • One tap — #[app::state] auto-generates a - migrate_my_entries() method (a wasm export) for any state with an - AuthoredMap/AuthoredVector field. One signed call sweeps every entry the - caller owns that is still below target, converting each through the path above; it returns - {converted, remaining} and is idempotent. Call it from the app/frontend ("migrate my - data") after an upgrade; remaining == 0 means the caller's data is fully converted.
    • -
-

-For either path to fire, the new binary must declare its schema target with -#[app::state(version = N)] — the value the convert compares each entry's tag against. It -defaults to 0 (inert), so a v2 binary that omits it never converts its identity-gated -data. WriterSetCell/SharedStorage (group writer-set data) converts only via -the organic writer-write path, never the one-tap batch (it is group data, not single-owner "my data"). -

-

-You do not write migrate_my_entries — #[app::state(version = N)] -generates it. All you do is declare the version; the method appears on your app and is -exported for RPC: -

-
// v2 binary. `version = 2` both sets the convert target AND generates
-// `migrate_my_entries()` because the state has an AuthoredMap field.
-#[app::state(version = 2, emits = for<'a> Event<'a>)]
-#[derive(app::Migrate)]
-#[migrate(from = NotesV1, method = migrate_v1_to_v2)]
-pub struct NotesV2 {
-    notes: AuthoredMap<String, LwwRegister<String>>,  // carried by the migrate
-    #[migrate(new = LwwRegister::new(String::new()))]
-    migration_note: LwwRegister<String>,
-}
-// No migrate_my_entries body anywhere — the macro emits it.
-

-Then the owner triggers it like any other method — one signed call, no args, returning -{converted, remaining}: -

-
# after the upgrade, from the owner's node (frontend "migrate my data" button):
-app_call(context_id, "migrate_my_entries", {})
-  → { "converted": 2, "remaining": 0 }   # this owner's 2 stale notes converted
-

-Loop until remaining == 0 if you want to drain everything in one sitting (a single call -already converts all of the caller's currently-stale entries; a second call returns -{converted: 0, remaining: 0}). It only ever touches entries the caller owns, so each user -converts their own data independently. -

- -

Migration UX surfaces

-

Two node-level events help frontends track migration progress without polling:

-

-AppVersionChanged event — fired once, over the context's SSE/WebSocket -event stream, when the context's application version flips (i.e. a migrate/upgrade was applied). -contextId rides on the outer ContextEvent envelope; the payload carries the -fromVersion / toVersion semver strings (either may be null if the -corresponding ApplicationMeta row was unavailable at emit time): -

-
{
-  "type": "AppVersionChanged",
-  "data": {
-    "fromVersion": "1.0.0",
-    "toVersion": "2.0.0"
-  }
-}
-

-Subscribe to context events via the node's SSE endpoint and react to this event to prompt owners to -run migrate_my_entries() — this avoids bundle-skew where the frontend loads v2 assets but -the context is still running v1. -

-

-authored_remaining in the migration status endpoint — tracks how many -members in the cohort still have unconverted identity-gated entries: -

-
GET /admin-api/groups/{namespace_id}/migration-status
-→ { ..., "authored_remaining": 3 }   # 3 cohort members still have stale entries
-

-authored_remaining == 0 means all members have run migrate_my_entries() or -had no entries to convert. Use this to drive a "migration complete" indicator in admin UIs. -

-
-Authorization. This is an /admin-api/ route gated by the same -admin-credential check as the rest of that surface — the node rejects unauthenticated callers before -any migration state is returned. The authored_remaining count (and the group/namespace -structure it implies) is internal operational data: never expose it on an unauthenticated path, and -scope the admin token to operators of that node/namespace. -
-
- - -
-

6. The no-silent-downgrade rail

-

-Changing an identity-gated type to a plain one — AuthoredMap → UnorderedMap, -SharedStorage → UnorderedMap, AuthoredVector → Vector, or dropping the field -— strips per-entry authorship / the writer ACL across the whole network. This is -refused: -

-
    -
  • in CI, by calimero-abi diff (an UNSAFE_IDENTITY_DOWNGRADE finding), and
    • -
  • at the node, by the upgrade gate, before the upgrade op is even emitted.
    • -
-

-If you hit this, the fix is not to strip the type — it's an owner-driven rewrite -(each owner re-migrates their own signed entries). Stripping authorship is almost never what you want. -

-
- - -
-

7. Guarding a migration: migration_check + abort

-

-A migration that compiles and runs can still be wrong — drop entries, break an invariant, -orphan a reference. To catch that before it commits, declare an optional -#[app::migration_check]. The migrate runs against an in-memory staging -buffer; the check runs against that same buffer before anything is written to the -live store. If it returns false, the runtime logically aborts — -the staging buffer is dropped, so the context stays on v1 with zero residue (root -and every child entry intact; no byte snapshot/restore needed). An app with no check commits -as before (backwards-compatible). -

- -

What the check can read (the contract)

-

-The check receives old (the committed v1 root) and new (the produced v2 -root). Staging makes these asymmetric: -

-
    -
  • new is fully trustworthy — its scalar/inline fields and its - lazy collections (read through the staging buffer) reflect the produced v2 state. Read - new.items.len(), walk new's collections, etc.
    • -
  • old's scalar/inline fields are pristine v1 (decoded from the - committed v1 root bytes) — a safe baseline.
    • -
  • old's lazy collections are NOT pristine: old.items and - new.items resolve to the same deterministic bucket, so in one check execution - they read the same (staged) data. Do not diff old vs new - collections — the comparison is always trivially equal.
    • -
-

-So write the check as an invariant over new (optionally against an -old scalar baseline), not as an old-vs-new collection diff. -

- -

Carrying a v1 baseline: the transient migration witness

-

-When the invariant needs a v1 value the v2 schema doesn't keep (e.g. "every item survived"), the -migrate returns a (State, Witness) tuple. The Witness is a borsh blob -delivered to the check and never persisted — it rides out on the runtime Outcome like -logs/events: -

-
#[derive(BorshSerialize, BorshDeserialize)]
-#[borsh(crate = "calimero_sdk::borsh")]
-struct MigrationWitness { v1_count: u64 }
-
-#[app::migrate]
-fn migrate() -> (DocV2, MigrationWitness) {
-    let mut items = old.items;
-    let v1_count = items.len().unwrap_or(0) as u64;   // captured BEFORE any change
-    // ... transform ...
-    (DocV2 { items, /* .. */ }, MigrationWitness { v1_count })
-}
-
-#[app::migration_check]
-fn check(_old: DocV1, new: DocV2, witness: MigrationWitness) -> bool {
-    // `new.items` is the produced collection; compare it to the v1 baseline.
-    matches!(new.items.len(), Ok(n) if n as u64 == witness.v1_count)
-}
-

-A migrate returning a plain State (no tuple) and a 2-arg check(old, new) stay -valid — the witness is opt-in. Prefer invariants that need no extra field where you -can: a required key present (new.items.get("alpha")?.is_some()), conservation against an -existing field (new.total == new.items.values().sum()), or a monotonic version -(new.version > old.version, an old scalar). -

-

-Built-in helpers (calimero_sdk::migration_check) operate on slices you build from -soundly-readable data (new collections, scalars, a witness): -

-
    -
  • entity_count_parity(a, b, delta) — counts match within delta
    • -
  • no_orphaned_refs(refs, keys) — every reference still resolves
    • -
  • conservation(old_total, new_total) — a total is preserved
    • -
-

-The check and any witness must be a deterministic pure function of -the v1 state, exactly like the migrate: they run independently on every node against byte-identical -input, so all nodes reach the same verdict — either all commit or all abort. (A -non-deterministic check or witness is a split-verdict bug, the same hazard -assert_migrate_converges guards.) A failed check is retryable: no -migration marker is recorded, so the context re-runs migrate+check on its next access -— a transient cause (e.g. not-yet-synced v1) self-heals once the input is complete. -

-
-Diffing old vs new collection cardinality directly (without a -witness/baseline) would need a pristine-snapshot read path for old that is not yet -implemented — tracked as a follow-up. Use the witness pattern above. -
- -

Aborting an in-flight migration (admin)

-

An operator can call off a migration that's rolling out:

-
POST /admin-api/groups/{namespace_id}/migration/abort
-

-It flips the group's pending target back to the pre-migration app id and drops the -pending marker, cascading to every descendant subgroup carrying the same pending -migration. Idempotent — a subtree with nothing pending is a no-op. It's a forward "stop" (un-migrated -contexts stop switching), not a rewind of any context that already migrated. -

-
- - -
-

8. Testing your migration

- -

Fast, in-process — TestHost

-

-Run the migration entirely in memory (cargo test, no Docker), and — most importantly — -assert it converges across nodes: -

-
#[cfg(test)]
-mod tests {
-    use calimero_sdk::testing::{assert_migrate_converges, TestHost};
-
-    #[test]
-    fn migrate_carries_and_seeds() {
-        let mut app = TestHost::new(DocV1::init);
-        app.call(|s| s.set_title("my-doc".to_owned())).unwrap();
-
-        let v2 = app.migrate(migrate_v1_to_v2);          // run it in-process
-
-        assert_eq!(v2.view(|s| s.title().unwrap()), "my-doc");   // title carried
-        assert_eq!(v2.view(|s| s.notes().unwrap()), "");          // notes seeded
-    }
-
-    #[test]
-    fn migration_converges_across_nodes() {
-        // Runs the migration as two different node identities from an identical
-        // v1 and asserts the two merkle roots match — a non-deterministic
-        // migration fails here in milliseconds instead of forking production.
-        assert_migrate_converges::<DocV1, DocV2>(
-            install_v1, migrate_v1_to_v2, [1u8; 32], [2u8; 32],
-        );
-    }
-}
-

-assert_migrate_converges compares the merkle root hash, which folds in -every child-collection entry — so a per-node value baked anywhere in the migrated state (a field, -or a value inside a carried collection) is caught. -

-
-In-process limits. Both "nodes" share one deterministic mock store, so this catches -identity/value divergence but not iteration-order divergence (the mock sorts child -entries by id). Cover ordering with a merobox e2e. -
- -

Full end-to-end — merobox

-

-For real cross-node behaviour (and iteration-order determinism), run a merobox workflow. See -workflows/app-migration/README.md ("Running locally": merobox bootstrap run …) -and the worked scenarios in apps/migrations/. -

-

-#[app::migration_check] has no in-process TestHost entry point — it's exercised -end-to-end by merobox: workflows/app-migration/29-migration-check-pass.yml (the check -passes → the migration commits) and 30-migration-check-fail-abort.yml (the check fails → -zero-residue logical abort, v1 still served). Model new check scenarios on those. -

- -

merobox step types for migrations

-

-A merobox workflow is a YAML list of steps run against a throwaway multi-node cluster -(merobox bootstrap run workflows/app-migration/<name>.yml). The migration-relevant step -types — every one of these is used by a real scenario-* workflow you can copy: -

- - - - - - - - - - - - - -
StepDoesKey fields
install_applicationinstall a bundle/wasm on a nodenode, path, dev, outputs: {app_v1: applicationId}
update_group_settingsset the upgrade policy — migrations need lazy_on_accessnode, group_id, upgrade_policy: lazy_on_access
upgrade_grouptrigger the upgrade; the node resolves the migrate from the target's ABInode, group_id, target_application_id, cascade
callinvoke a method — a read counts, so it triggers the lazy migratenode, context_id, method, args
get_migration_status / assert_migration_completeread / poll the cohort rollup — admin node onlynode, namespace_id [, timeout_seconds, poll_interval]
get_cascade_status / assert_cascade_completeper-subgroup cascade rollupnode, namespace_id [, timeout_seconds]
resync_contextrecover a stranded member (destructive)node, context_id, force: true
abort_migrationadmin stop of a rolling migrationnode, namespace_id
assert_log_present / assert_log_absentassert a node log line did / didn't firenodes: [...], patterns: [...]
-

The canonical shape — install v1, write data, set LazyOnAccess, install v2, upgrade, poke each peer to migrate, assert:

-
steps:
-  - { type: install_application, node: n1, dev: true,
-      path: apps/migrations/migration-suite-v1/res/migration-suite-1.0.0.mpk,
-      outputs: { app_v1: applicationId } }
-  # ... create_namespace / create_context / node-2 joins / write v1 data via `call` ...
-  - { type: update_group_settings, node: n1, group_id: "{{group_id}}",
-      upgrade_policy: lazy_on_access }
-  - { type: install_application, node: n1, dev: true,
-      path: apps/migrations/migration-suite-v2/res/migration-suite-2.0.0.mpk,
-      outputs: { app_v2: applicationId } }
-  - { type: upgrade_group, node: n1, group_id: "{{namespace_id}}",
-      target_application_id: "{{app_v2}}", cascade: true }
-  - { type: call, node: n2, context_id: "{{ctx}}", method: get_something }  # read ⇒ lazy migrate
-  - { type: assert_log_present, nodes: [n2], patterns: ["Migration completed successfully"] }
-  - { type: assert_migration_complete, node: n1, namespace_id: "{{namespace_id}}" }
-

-See workflows/app-migration/README.md for running locally, and any -scenario-*.yml for one complete, copy-pastable workflow per migration shape. -

- -

Worked examples

-
    -
  • apps/migration-harness-example/ — #[derive(Migrate)] + TestHost - unit tests (carry, seed, rename, convergence, divergence-detection).
    • -
  • apps/migrations/scenario-* + workflows/app-migration/*.yml — one crate - pair and one merobox workflow per migration shape (additive, remove, rename, type-change, - CRDT-native, authored-map, …).
    • -
-
- - -
-

9. Shipping a migration

-

-Migrations run only under UpgradePolicy::LazyOnAccess. Trigger the upgrade — note there is -no method name to pass; the node resolves the migration from the version + edge your -app's build embeds in its ABI: -

-
upgrade_group(
-    target_application = <new bundle's application id>,
-    cascade            = true,   // optional: fan out across a namespace subtree
-)
-

-Per service, the node compares the state version of the bytecode the group currently runs against the -target's and decides: equal → code-only swap (nothing runs); one ahead with a declared edge → run that -edge's migrate; version bump with NO edge, more than one hop, or an older target → the upgrade is -rejected with an actionable error instead of silently shipping new code onto -old-layout state. There is no caller-supplied method at all — the declared ABI is the single source of -truth. -

-

-Each node self-migrates on its next context access (logged as Executing migration → -Migrated state written successfully; the extra performing lazy upgrade before -execution line precedes them only on the non-cascade lazy-on-read path — under -cascade: true the cascade propagator drives the migrate instead). Picking -Automatic/Coordinated for a migration is rejected at emit time — only -LazyOnAccess runs the migrate function. -

-

-If the app has identity-gated state (AuthoredMap/AuthoredVector), -declare the new binary's schema target with #[app::state(version = N)] so the post-migrate -owner-driven convert (§5) has a target to compare against. -

- -

9.1 The policy prerequisite: create groups as LazyOnAccess

-

-Migrations run only under UpgradePolicy::LazyOnAccess; an -upgrade_group whose target declares a migration is rejected at emit time -under Automatic or Coordinated. The trap: if your app creates its namespace -with upgradePolicy: 'Automatic' (a tempting-sounding default), every migration you ever -ship to that workspace fails at the upgrade call. Either create groups as LazyOnAccess from -day one, or heal before upgrading: -

-
PATCH /admin-api/groups/:group_id   { "upgradePolicy": "LazyOnAccess" }
-
-There is also a runtime guard: an admin can no longer flip a group away from -LazyOnAccess while a migration is pending — that flip is rejected, because un-accessed -contexts would otherwise swap bytecode without running the migrate. -
- -

9.2 Bundle apps: the id never changes — the blob does

-

-A bundle (.mpk) derives ApplicationId = hash(package, signer) — it is -version-stable. v1 and v2 of the same package share one id; installing v2 overwrites -the blob in place under that id. Consequences: -

-
    -
  • Never key "is an update available / applied" off the ApplicationId changing. It - won't. The version discriminator is the bytecode blob id, recorded on the group as appKey - at upgrade time.
    • -
  • Download ≠ apply. Installing the new bundle (registry download) flips the - local node's installed version immediately — but the workspace group still targets the old - bytecode until upgrade_group runs. A UI that only compares "installed version vs registry - latest" reads "up to date" in the downloaded-but-never-migrated state and strands the - admin. The ground truth for "has this workspace been migrated" is: - group.appKey == installed_application.blob.bytecode (note: appKey is hex on - the admin API, blob ids are base58 — decode before comparing).
    • -
  • Single-wasm apps content-address the id per version, so the old id-comparison instinct works there - — bundles are the exception that ships to real users.
    • -
- -

9.2b The node resolves the method — callers carry nothing

-

-Your build embeds state_version and the declared migration edges -(migrations: [{method, fromVersion}]) in each service's ABI, and the node reads them at -upgrade time. Updater UIs/CLIs never carry a method name; the typo/omission class of failure is -structurally gone. The export name is generated (migrate_v{N-1}_to_v{N}) unless you pass -method = … explicitly in the derive — explicit names remain supported and resolve -identically. -

- -

9.2c Multi-service bundles: per-service version arithmetic

-

-A multi-service bundle (e.g. a registry service + a data service) upgrades as one unit, but the node -judges each service by its own declared state version: -

-
    -
  • a service whose version is unchanged is code-only by arithmetic — no wasm - probing, no fake failure, nothing to declare;
    • -
  • a service one version ahead with a declared edge runs its own migrate;
    • -
  • a service that bumps its version without declaring an edge rejects the whole upgrade at emit time - (mis-built bundle).
    • -
-

-So each service simply declares its own truth and the release composes: "only service A changed", "only -B", and "both" all work with zero coordination. One current restriction: if two services declare -different explicit method names for the same release, the upgrade is rejected (the wire carries -one method for pre-v2 receivers) — omit method = … or use the same name; full per-service -actuation lifts this later. -

- -

9.3 Reaching every context: cascade: true

-

-upgrade_group without cascade upgrades only the target group. -If your app puts data in subgroup contexts (per-folder/per-channel child contexts), they stay on the -old schema forever and the UI on top of them keeps reading v1. Pass cascade: true to fan -the upgrade out across the namespace subtree as one atomic op. -

- -

9.4 What each member's node does (and when)

-

After the admin's upgrade_group, peers converge lazily:

-
    -
  1. the upgrade replicates via the group op stream; a peer's sync gate detects the - pending upgrade (id change, or same-id appKey divergence for bundles) and pre-stages the - new blob over BlobShare;
    2. -
  2. the node migrates on its next access to each context — reads count, so merely - opening the app advances it; an idle node stays on v1 until then (write your UI copy accordingly: - "updates next time you use it", not "updates automatically");
    3. -
  3. while a migration is in flight, writes are refused with upgrade in progress … writes refused - until migration completes — handle this error in the frontend as a status ("workspace - updating"), not a raw failure;
    4. -
  4. on completion the node emits the AppVersionChanged SSE event — subscribe to flip - version displays live (mero.events.onAppVersionChanged).
    5. -
- -

9.5 Frontend checklist (the part nobody tests until a real user does)

-
    -
  • Show the installed version to every member, not just admins; pair it with honest - LazyOnAccess copy.
    • -
  • Detect the downloaded-but-not-applied state via the appKey-vs-blob - comparison (§9.2) and keep offering the apply step — it must survive a page reload (the bundle id is - stable, so the group's targetApplicationId still addresses the downloaded bytecode).
    • -
  • Map the upgrade-gate write refusal (§9.4.3) to an "updating…" banner.
    • -
  • Subscribe to AppVersionChanged for the completion signal.
    • -
  • Browser-test the full admin flow once per release: CLI clients skip CORS, so a missing preflight - method (e.g. PATCH for updateGroupSettings) surfaces only in browsers, as an opaque - HTTP 0 network error.
    • -
- -

9.6 Verifying propagation by hand

-
# the group's target bytecode (hex appKey) — same on every member after sync
-meroctl --output-format json --node <n> namespace ls
-
-# what's actually installed under the (stable) application id
-meroctl --node <n> app ls          # check the Source/Blob columns' version
-
-# poke a context to trigger the lazy migrate (reads count)
-meroctl --node <n> call <any_read_method> --context <ctx_id>
-

-appKey equal everywhere + each node's installed blob matching it + the call returning -post-migration data = the migration has fully landed. -

- -

9.7 Multiple versions on one node

-

-A context executes the bytecode its own group points at — not whatever was downloaded -last. Installing a newer bundle never changes what existing workspaces run: each migrates when its own -admin upgrades it. Consequences you can rely on: -

-
    -
  • two workspaces on one node can run different versions of the same app indefinitely;
    • -
  • createNamespace accepts an optional appKey to pin a new workspace to any - installed version (default: latest);
    • -
  • the admin API lists every retained version of a package - (GET /admin-api/applications/:id/versions) and namespace DTOs carry a per-workspace - appVersion — display that, never the shared installed version.
    • -
- -

9.8 Catching up several versions behind

-

-A context behind its group by more than one version catches up by replaying the -group's recorded upgrade ladder hop by hop — each hop runs in that release's -own bytecode, not the latest. The node fetches each rung's blob from peers as needed and runs -that version's own migrate. Because every member runs the same frozen bytes for a given hop, the result -is identical across the group by construction (no cross-node divergence). A single access on a stale -context walks all pending hops in order and lands on the current version; nothing is run eagerly. -

-

-An admin can also move a group several versions in one action: the upgrade plans a rung per intermediate -state version (a release that doesn't change state adds no rung) from the versions -installed on the node, and the group advances rung by rung. -

- -

9.9 Support window and recovering a stranded member

-

-The versions whose blobs remain obtainable (from peers or the registry) define which upgrades chain -directly. A member below that window — an intermediate blob is gone everywhere, or it was offline across -several releases and never fetched one — is detected, not silently wedged: it stays on -its current real version and reports failed with no_migration_path in the -migration rollup. It never runs a later hop's migrate against older state. -

-

Two recoveries, both operator-initiated:

-
    -
  • Stepwise reinstall — install an intermediate version from the registry. Any - installed version can be an upgrade target (§9.7), so the gap becomes a single hop and the next access - finishes the chain.
    • -
  • Resync — POST /admin-api/contexts/{id}/resync adopts an up-to-date - peer's state wholesale (a full-state snapshot), bypassing replay. This is destructive: - any local edits the context hasn't broadcast are discarded, so it refuses with the local-head count - unless you pass {"force": true}. After it completes the member is back at the current - version. (The 37-stranded-resync merobox workflow exercises exactly this path end to - end.)
    • -
-

-A self-heal needs no action: if the missing blob simply arrives later, the next access resolves the hop, -migrates, and clears the failed marker. -

-
- - -
-

10. Wiring it into your app — mero-js & mero-react

-

-The node exposes the migration surface over its admin + RPC + SSE APIs. The JS SDK -(@calimero-network/mero-js) and React bindings (mero-react) — both on the -migrations-v2 3.0.0 line — wrap those so a frontend can drive and observe an upgrade -without hand-rolling HTTP. -

- -

mero-js

- - - - - - - - - - - -
CallWhat it does
admin.upgradeGroup(groupId, { targetApplicationId, cascade })trigger the upgrade (§9); cascade fans out the subtree (§9.3).
admin.getMigrationStatus(namespaceId)cohort rollup — migrated / in-progress / unknown / failed, allMigrated, authoredRemaining (§5).
admin.getCascadeStatus(namespaceId)per-subgroup cascade rollup.
admin.listApplicationVersions(applicationId)every retained version of a package — feeds a version picker (§9.7).
admin.resyncContext(contextId, { force })recover a stranded member (§9.9) — destructive, needs force.
rpc.migrateMyEntries(contextId)owner one-tap convert of identity-gated data → { converted, remaining } (§5).
events.onAppVersionChanged(handler)SSE subscription; fires { contextId, fromVersion, toVersion } when a context's version flips (§9.4).
-
// trigger + observe an upgrade
-await mero.admin.upgradeGroup(groupId, { targetApplicationId: v2Id, cascade: true });
-const unsub = mero.events.onAppVersionChanged(({ toVersion }) => setVersion(toVersion));
-
-// owner converts their own identity-gated entries after the upgrade
-let { remaining } = await mero.rpc.migrateMyEntries(contextId);
-// (call again until remaining === 0 to drain everything in one sitting)
-
-// recover a member stuck with no_migration_path
-await mero.admin.resyncContext(contextId, { force: true });
-
-Bundle skew. Never infer "already updated" from installed-version-vs-registry-latest: -for a bundle the application id is version-stable (§9.2). Drive the UI off -getMigrationStatus / AppVersionChanged, never the id. -
- -

mero-react hooks

- - - - - - - - - - - - -
HookFor
useUpgradeGroup()an admin "apply update" action → upgradeGroup with loading/error.
useGroupUpgradeStatus(groupId)live cohort upgrade rollup (upgradeStatus) for a progress UI.
useMigrationStatus(namespaceId)the full migration-status rollup (per-member states + authoredRemaining).
useAppVersion(contextId, expected)context's installed version + isStale vs the app's build constant; auto-updates on AppVersionChanged ("reload to update").
useMyAuthoredMigration(contextId)pending + authorize() — the per-user "migrate my data" tap (wraps migrate_my_entries).
useResyncContext()resyncContext (force), for a "recover this member" action.
useRetryGroupUpgrade()re-drive a stalled cohort upgrade.
useLatestVersion(applicationId)registry latest + retained versions for a picker.
-
function WorkspaceBanner({ contextId }) {
-  const { appVersion, isStale } = useAppVersion(contextId, BUILD_VERSION);
-  const { pending, authorize } = useMyAuthoredMigration(contextId);
-
-  if (isStale) return <Banner>Update available — reload to use v{appVersion}</Banner>;
-  if (pending) return <Button onClick={authorize}>Migrate my data</Button>;
-  return null;   // up to date, nothing of mine left to convert
-}
-
-Exact response/return field names track each repo's typings -(mero-js/src/admin-api/admin-types.ts, mero-react/src/hooks/index.ts) — the -canonical source if a field is renamed in a later release. -
-
- - -
-

11. Quick reference

- - - - - - - - - - - -
DoDon't
Carry fields: field: old.fieldRe-insert an AuthoredMap/UserStorage/SharedStorage in a migration
Seed new fields with ::new() / #[migrate(new = …)]Call Counter::increment/decrement or RGA::insert in a migration (they panic)
increment_for / insert_str_at_timestamp to replay a CRDT deterministicallyUse wall-clock, RNG, or unsorted iteration order
sort() before building a Vector from a map/setChange an identity-gated type to a plain one (refused)
Prove convergence with assert_migrate_convergesAssume single-node tests prove cross-node determinism
panic! on bad input (non-destructive abort)Expect a Result from the migrate fn
Keep old version blobs obtainable so behind members can chainDrop an intermediate version and expect members below it to auto-upgrade (they strand → reinstall or resync)
-
- -
-
- - - diff --git a/architecture/nav.js b/architecture/nav.js deleted file mode 100644 index 3528ef1aa0..0000000000 --- a/architecture/nav.js +++ /dev/null @@ -1,452 +0,0 @@ -/* Calimero Core Architecture — Shared Navigation */ -(function () { - 'use strict'; - - const REPO = 'https://github.com/calimero-network/core'; - const PAGES_BASE = getBase(); - - function getBase() { - const p = location.pathname; - if (p.includes('/crates/') || p.includes('/protocol/')) return '../'; - return './'; - } - - const NAV = [ - { label: 'Home', href: 'index.html', dot: '#f59e0b' }, - { section: 'Protocol Reference' }, - { label: 'Overview', href: 'protocol/index.html', dot: '#a5ff11' }, - { label: '1 · Concepts & Scopes', href: 'protocol/concepts.html', dot: '#a5ff11', sub: true }, - { label: '2 · Identities & Keys', href: 'protocol/identities.html', dot: '#a5ff11', sub: true }, - { label: '3 · Operations & the DAG', href: 'protocol/operations.html', dot: '#a5ff11', sub: true }, - { label: '4 · Projection & Root Hash', href: 'protocol/projection.html', dot: '#a5ff11', sub: true }, - { label: '5 · Application Execution', href: 'protocol/execution.html', dot: '#a5ff11', sub: true }, - { label: '6 · Governance', href: 'protocol/governance.html', dot: '#a5ff11', sub: true }, - { label: '7 · Networking & Wire', href: 'protocol/networking.html', dot: '#a5ff11', sub: true }, - { label: '8 · The Write Path', href: 'protocol/write-path.html', dot: '#a5ff11', sub: true }, - { label: '9 · The Receive Path', href: 'protocol/receive-path.html', dot: '#a5ff11', sub: true }, - { label: '10 · Sync & Convergence', href: 'protocol/sync.html', dot: '#a5ff11', sub: true }, - { label: 'Appendix · Storage Schema', href: 'protocol/storage.html', dot: '#06b6d4', sub: true }, - { section: 'For Builders' }, - { label: 'Getting Started', href: 'getting-started.html', dot: '#10b981' }, - { label: 'Core Concepts', href: 'concepts.html', dot: '#10b981' }, - { label: 'SDK Reference', href: 'crates/sdk.html', dot: '#f97316' }, - { label: 'App Lifecycle', href: 'app-lifecycle.html', dot: '#06b6d4' }, - { label: 'Example: Chat', href: 'example-chat.html', dot: '#10b981' }, - { label: 'Example: Docs', href: 'example-docs.html', dot: '#3b82f6' }, - { section: 'For Operators' }, - { label: 'merod & meroctl', href: 'crates/tools.html', dot: '#f59e0b' }, - { label: 'Config Reference', href: 'config-reference.html', dot: '#f97316' }, - { label: 'TEE Mode', href: 'tee-mode.html', dot: '#ec4899' }, - { label: 'Fleet HA', href: 'tee-fleet-ha.html', dot: '#a855f7' }, - { label: 'Release Process', href: 'release.html', dot: '#10b981' }, - { label: 'Auth Service', href: 'crates/auth.html', dot: '#84cc16' }, - { section: 'Architecture Deep-Dive' }, - { label: 'Unified Causal Log Cutover', href: 'unified-causal-log-cutover-plan.html', dot: '#10b981' }, - { label: 'ABI Conformance', href: 'abi-conformance.html', dot: '#10b981' }, - { label: 'System Overview', href: 'system-overview.html', dot: '#3b82f6' }, - { label: 'Local Governance', href: 'local-governance.html', dot: '#10b981' }, - { label: 'Auto-Follow', href: 'auto-follow.html', dot: '#10b981' }, - { label: 'Sequence Diagrams', href: 'sequence-diagrams.html', dot: '#ec4899' }, - { label: 'Wire Protocol', href: 'wire-protocol.html', dot: '#8b5cf6' }, - { label: 'Storage Schema', href: 'storage-schema.html', dot: '#06b6d4' }, - { label: 'Migrations', href: 'migrations.html', dot: '#f59e0b' }, - { label: 'Error Flows', href: 'error-flows.html', dot: '#ef4444' }, - { label: 'Metrics Reference', href: 'metrics-reference.html', dot: '#10b981' }, - { label: 'Dependency Explorer', href: 'dependency-explorer.html', dot: '#f59e0b' }, - { label: 'Glossary', href: 'glossary.html', dot: '#d4d4dc' }, - { section: 'Crate Internals' }, - { label: 'Node', href: 'crates/node.html', dot: '#3b82f6', sub: true }, - { label: 'Context & Groups', href: 'crates/context.html', dot: '#10b981', sub: true }, - { label: 'Network & P2P', href: 'crates/network.html', dot: '#8b5cf6', sub: true }, - { label: 'Storage', href: 'crates/store.html', dot: '#06b6d4', sub: true }, - { label: 'Sync Engine', href: 'crates/sync.html', dot: '#f59e0b', sub: true }, - { label: 'WASM Runtime', href: 'crates/runtime.html', dot: '#ec4899', sub: true }, - { label: 'Server & API', href: 'crates/server.html', dot: '#f97316', sub: true }, - { label: 'Causal DAG', href: 'crates/dag.html', dot: '#f59e0b', sub: true }, - ]; - - /* ── Full-text search index ── */ - - let searchIndex = null; - let searchReady = false; - let selectedIdx = -1; - - function buildIndex() { - const pages = NAV.filter(n => n.href); - return Promise.all(pages.map(item => { - const url = PAGES_BASE + item.href; - return fetch(url).then(r => r.ok ? r.text() : '').then(html => { - const doc = new DOMParser().parseFromString(html, 'text/html'); - doc.querySelectorAll('script, style, nav, .sidebar, .breadcrumb').forEach(el => el.remove()); - const headings = Array.from(doc.querySelectorAll('h1, h2, h3, h4')).map(h => h.textContent.trim()); - const text = (doc.body ? doc.body.textContent : '').replace(/\s+/g, ' ').trim(); - return { title: item.label, href: item.href, headings, text }; - }).catch(() => ({ title: item.label, href: item.href, headings: [], text: '' })); - })); - } - - function searchDocs(query, index) { - const q = query.toLowerCase().trim(); - if (!q) return []; - const tokens = q.split(/\s+/).filter(Boolean); - const scored = []; - for (const page of index) { - let score = 0; - const titleLow = page.title.toLowerCase(); - const headingsLow = page.headings.join(' ').toLowerCase(); - const textLow = page.text.toLowerCase(); - for (const t of tokens) { - if (titleLow.includes(t)) score += 10; - if (headingsLow.includes(t)) score += 5; - if (textLow.includes(t)) score += 1; - } - if (score > 0) scored.push({ ...page, score }); - } - scored.sort((a, b) => b.score - a.score); - return scored.slice(0, 12); - } - - function getExcerpt(text, query) { - const tokens = query.toLowerCase().trim().split(/\s+/).filter(Boolean); - const lower = text.toLowerCase(); - let best = -1; - for (const t of tokens) { - const idx = lower.indexOf(t); - if (idx !== -1) { best = idx; break; } - } - if (best === -1) return ''; - const start = Math.max(0, best - 60); - const end = Math.min(text.length, best + 140); - let slice = (start > 0 ? '...' : '') + text.slice(start, end).trim() + (end < text.length ? '...' : ''); - for (const t of tokens) { - const re = new RegExp('(' + t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + ')', 'gi'); - slice = slice.replace(re, '$1'); - } - return slice; - } - - /* ── Search overlay DOM ── */ - - function createSearchOverlay() { - const overlay = document.createElement('div'); - overlay.className = 'search-overlay'; - overlay.id = 'search-overlay'; - overlay.innerHTML = ` -
-
- - - ESC -
-
-
Type to search across all pages.
Tip: Use ⌘K to open search anytime.
-
-
- ↑↓ navigate - ↵ open - esc close -
-
- `; - document.body.appendChild(overlay); - - overlay.addEventListener('click', (e) => { - if (e.target === overlay) closeSearch(); - }); - - const input = overlay.querySelector('#search-input'); - let debounce = null; - input.addEventListener('input', () => { - clearTimeout(debounce); - debounce = setTimeout(() => runSearch(input.value), 150); - }); - - input.addEventListener('keydown', (e) => { - if (e.key === 'ArrowDown') { e.preventDefault(); moveSelection(1); } - else if (e.key === 'ArrowUp') { e.preventDefault(); moveSelection(-1); } - else if (e.key === 'Enter') { - e.preventDefault(); - const items = document.querySelectorAll('.search-result-item'); - if (items[selectedIdx]) items[selectedIdx].click(); - } - else if (e.key === 'Escape') { closeSearch(); } - }); - } - - function runSearch(query) { - const results = document.getElementById('search-results'); - if (!results) return; - selectedIdx = -1; - if (!query.trim() || query.trim().length < 2) { - results.innerHTML = '
Type to search across all pages.
Tip: Use ⌘K to open search anytime.
'; - return; - } - if (!searchReady) { - results.innerHTML = '
Building search index...
'; - return; - } - const matches = searchDocs(query, searchIndex); - if (matches.length === 0) { - results.innerHTML = '
No results for ' + escHtml(query) + '
'; - return; - } - results.innerHTML = matches.map((m, i) => { - const titleHtml = highlightTokens(m.title, query); - const excerpt = getExcerpt(m.text, query); - return '' + - '
' + titleHtml + '
' + - (excerpt ? '

' + excerpt + '

' : '') + - ''; - }).join(''); - selectedIdx = 0; - } - - function highlightTokens(text, query) { - const tokens = query.toLowerCase().trim().split(/\s+/).filter(Boolean); - let out = escHtml(text); - for (const t of tokens) { - const re = new RegExp('(' + t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + ')', 'gi'); - out = out.replace(re, '$1'); - } - return out; - } - - function escHtml(s) { - return s.replace(/&/g, '&').replace(//g, '>'); - } - - function moveSelection(dir) { - const items = document.querySelectorAll('.search-result-item'); - if (!items.length) return; - items.forEach(i => i.classList.remove('selected')); - selectedIdx = (selectedIdx + dir + items.length) % items.length; - items[selectedIdx].classList.add('selected'); - items[selectedIdx].scrollIntoView({ block: 'nearest' }); - } - - function openSearch() { - const overlay = document.getElementById('search-overlay'); - if (!overlay) return; - overlay.classList.add('open'); - const input = overlay.querySelector('#search-input'); - input.value = ''; - input.focus(); - runSearch(''); - if (!searchReady && !searchIndex) { - buildIndex().then(idx => { searchIndex = idx; searchReady = true; }); - } - } - - function closeSearch() { - const overlay = document.getElementById('search-overlay'); - if (overlay) overlay.classList.remove('open'); - } - - /* ── Sidebar builder ── */ - - function currentPage() { - const p = location.pathname; - for (const item of NAV) { - if (!item.href) continue; - if (p.endsWith(item.href) || p.endsWith('/' + item.href)) return item.href; - } - if (p.endsWith('/') || p.endsWith('/architecture/') || p.endsWith('/architecture')) return 'index.html'; - return ''; - } - - function buildSidebar() { - const sb = document.createElement('nav'); - sb.className = 'sidebar'; - sb.id = 'sidebar'; - - const cur = currentPage(); - - sb.innerHTML = ` - -
- -
- -
-
© Calimero Network
-
- - - GitHub - - - - Website - -
-
- `; - - const themeBtn = document.createElement('button'); - themeBtn.id = 'theme-toggle'; - themeBtn.className = 'theme-toggle'; - themeBtn.textContent = '\u263e'; - themeBtn.title = 'Toggle light/dark mode'; - themeBtn.onclick = () => { - const isLight = document.documentElement.classList.toggle('light'); - themeBtn.textContent = isLight ? '\u2600' : '\u263e'; - try { localStorage.setItem('arch-theme', isLight ? 'light' : 'dark'); } catch(e) {} - }; - sb.querySelector('.sidebar-logo').appendChild(themeBtn); - - try { - if (localStorage.getItem('arch-theme') === 'light') { - document.documentElement.classList.add('light'); - themeBtn.textContent = '\u2600'; - } - } catch(e) {} - - const linksEl = sb.querySelector('#nav-links'); - for (const item of NAV) { - if (item.section) { - const s = document.createElement('div'); - s.className = 'nav-section'; - s.textContent = item.section; - linksEl.appendChild(s); - continue; - } - const a = document.createElement('a'); - a.className = 'nav-link' + (item.sub ? ' sub' : '') + (item.href === cur ? ' active' : ''); - a.href = PAGES_BASE + item.href; - a.innerHTML = `${item.label}`; - a.dataset.label = item.label.toLowerCase(); - linksEl.appendChild(a); - } - - document.body.prepend(sb); - - const btn = document.createElement('button'); - btn.className = 'menu-toggle'; - btn.textContent = '\u2630'; - btn.onclick = () => sb.classList.toggle('open'); - document.body.prepend(btn); - - sb.querySelector('#open-search').addEventListener('click', openSearch); - } - - /* ── Breadcrumb, tabs, ghLink ── */ - - function buildBreadcrumb(items) { - const bc = document.querySelector('.breadcrumb'); - if (!bc) return; - bc.innerHTML = items.map((item, i) => { - if (i === items.length - 1) return `${item.label}`; - return `${item.label}/`; - }).join(''); - } - - function tabSystem() { - document.querySelectorAll('[data-tabs]').forEach(container => { - const tabs = container.querySelectorAll('.tab'); - const panels = container.parentElement.querySelectorAll('.panel'); - tabs.forEach(tab => { - tab.addEventListener('click', () => { - tabs.forEach(t => t.classList.remove('on')); - panels.forEach(p => p.classList.remove('on')); - tab.classList.add('on'); - const target = document.getElementById(tab.dataset.target); - if (target) target.classList.add('on'); - }); - }); - }); - } - - function ghLink(path, line) { - const base = REPO + '/blob/master/'; - const url = line ? base + path + '#L' + line : base + path; - return `${path}`; - } - - /* ── Keyboard shortcuts ── */ - - document.addEventListener('keydown', (e) => { - if ((e.metaKey || e.ctrlKey) && e.key === 'k') { - e.preventDefault(); - openSearch(); - } - if (e.key === 'Escape') { - closeSearch(); - } - }); - - /* ── TABLE OF CONTENTS (right sidebar) ── */ - - function buildToc() { - const content = document.querySelector('.content'); - if (!content) return; - - const headings = content.querySelectorAll('h2, h3'); - if (headings.length < 3) return; - - headings.forEach((h, i) => { - if (!h.id) h.id = 'toc-' + i + '-' + h.textContent.trim().toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/-+$/, ''); - }); - - const toc = document.createElement('aside'); - toc.className = 'toc-sidebar'; - toc.innerHTML = '
On this page
'; - - headings.forEach(h => { - const a = document.createElement('a'); - a.className = 'toc-link' + (h.tagName === 'H3' ? ' toc-h3' : ''); - a.href = '#' + h.id; - a.textContent = h.textContent.trim(); - a.addEventListener('click', (e) => { - e.preventDefault(); - h.scrollIntoView({ behavior: 'smooth', block: 'start' }); - history.replaceState(null, '', '#' + h.id); - }); - toc.appendChild(a); - }); - - const main = document.querySelector('.main'); - if (main) main.appendChild(toc); - - const links = toc.querySelectorAll('.toc-link'); - let ticking = false; - - function updateActive() { - let current = null; - headings.forEach(h => { - if (h.getBoundingClientRect().top <= 80) current = h; - }); - links.forEach(a => { - a.classList.toggle('active', current && a.getAttribute('href') === '#' + current.id); - }); - ticking = false; - } - - window.addEventListener('scroll', () => { - if (!ticking) { requestAnimationFrame(updateActive); ticking = true; } - }, { passive: true }); - updateActive(); - } - - /* ── Init ── */ - - document.addEventListener('DOMContentLoaded', () => { - buildSidebar(); - createSearchOverlay(); - tabSystem(); - buildToc(); - buildIndex().then(idx => { searchIndex = idx; searchReady = true; }); - }); - - window.arch = { ghLink, buildBreadcrumb, openSearch, REPO, PAGES_BASE }; -})(); diff --git a/architecture/protocol/concepts.html b/architecture/protocol/concepts.html deleted file mode 100644 index 3838aa4d61..0000000000 --- a/architecture/protocol/concepts.html +++ /dev/null @@ -1,128 +0,0 @@ - - - - -Concepts & Scopes — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/Concepts & Scopes
-

Concepts & Scopes

-

Chapter 1 — the vocabulary every later chapter depends on.

- -
-

The nouns

-

Five words carry the whole model. Get these straight and the rest of the reference reads easily.

-
-

Node

A single peer process. It holds keys, runs applications, stores state, and talks to other nodes over a peer-to-peer network. One machine can run one node.

-

Application

A WASM module that defines the rules of a shared state machine — its data types and the methods that mutate them. Apps are sandboxed and deterministic.

-

Identity

An Ed25519 key. Two layers exist (covered in Chapter 2): a transport identity that names a node on the network, and a member identity that authors and authorizes operations.

-

Operation

The atomic, signed unit of change. Every mutation — to data, access, membership, or admin state — is an operation in a causal DAG. Chapter 3 dissects it.

-

Scope

The unit of "stuff that replicates and converges together." This is the load-bearing abstraction, and the rest of this page is about it.

-
-
- -
-

What a scope is

-

A scope is a self-contained replication, encryption, and convergence domain. It is one node in a tree of scopes. Every scope owns exactly four things:

- - - - - -
An op-log (its own DAG)The causally-ordered set of signed operations that belong to this scope, and only this scope.
A keyOperations in the scope are encrypted under the scope's key. Only members hold the key, so only members can read the operations.
A membership setWho may author and read operations here. Membership is itself maintained by operations (Chapter 6).
A scope rootOne 32-byte hash summarising the scope's entire state. Convergence is always checked per scope by comparing scope roots.
-

A scope is named by a stable 32-byte ScopeId. Everything in the protocol — operations, encryption, sync, convergence — is parameterised by scope.

-
- -
-

What is actually a scope

-

The things you already think of as nouns are each a scope:

-
    -
  • The root governance scope of a namespace — where membership, admin, and policy live — is one scope.
    • -
  • Each subgroup is its own scope, with its own key, its own members, and its own operations.
    • -
  • Each context — a running application's replicated state — is its own scope.
    • -
-

So what you informally call "a context" is really a small tree of scopes: the governance scope(s) above it, plus the context's own data scope.

-
-
- - - - - - - - - - - - - - Root governance scope - members · admin · policy - - - - - Subgroup scope - own key · own members - - Context scope - app state operations - - - - - Context scope - app state operations - - -
-

Each box is an independent scope: its own op-log, key, members, and scope root. The edges form the scope tree.

-
- -
-

Why the concept exists

-

This is the part worth internalising. Earlier, Calimero had three different kinds of log with three different convergence signals: a data-delta log per context, a governance DAG per namespace, and a separate access-control rotation log. Three mechanisms, three hashes, three sync paths — and bugs hid in the seams between them.

-

The scope abstraction collapses all of that into one idea: a scope is anything that has its own op-log and converges on its own. Data writes, access-control changes, membership changes, and admin actions are all just operations living in some scope. One DAG structure, one fold, one scope root, one sync engine — applied uniformly to contexts and governance. That generalisation is what makes the operation log "unified," and scope is the noun that made it possible.

-
- -
-

The one subtlety: scopes form a tree, and operations can cross scope edges

-

An operation's parents are normally other operations in the same scope. But an operation may also reference the current head of its parent governance scope.

-

This is safe because membership is nested: a subgroup's members are always a subset of its ancestor's members, so pointing "up" the tree never reveals anything a reader couldn't already see. That cross-scope parent link is what causally ties a child scope to the governance that authorises it — for example, a context write can pin the exact membership state it was authored under.

-

That single mechanism is the foundation of authorization at the causal cut ("forward-only" authorization): a write authored before a member was revoked stays valid regardless of the order in which a peer receives the two operations. Chapter 6 builds the full rule on top of this; for now, just hold the shape: one parent set, one causal model, spanning data and governance.

-
- -
-

Where this leads

-

With scope in hand, the rest of the reference is short:

-
    -
  • Chapter 3 defines the operation that lives in a scope, and how its DAG is ordered.
    • -
  • Chapter 4 defines the fold from a scope's DAG to its state and its scope root.
    • -
  • Chapters 6–10 are all variations on "operations move between scopes, and scopes converge."
    • -
-
- -
-← Protocol Reference -Identities & Keys → -
- -
-
- - - - diff --git a/architecture/protocol/execution.html b/architecture/protocol/execution.html deleted file mode 100644 index 2c0d27d65e..0000000000 --- a/architecture/protocol/execution.html +++ /dev/null @@ -1,226 +0,0 @@ - - - - -Application Execution — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/Application Execution
-

Application Execution

-

Chapter 5 — where data operations come from: running a WASM application.

- -
-

What an execution is

-

An application is a WASM module that defines a state machine: data types and the methods that mutate them. An execution is one method call. It reads the context's current state, runs deterministic guest code, and produces a set of state changes plus a fresh root hash. Those state changes are the data-plane operations from Chapters 3–4.

-

Crucially, execution is sandboxed and self-contained: the guest cannot touch the network, the clock, or the disk directly. It can only call back into the host through a fixed set of functions — the ABI. Everything the guest does is therefore observable, bounded, and reproducible.

-
-
-
- -
-

The host ABI boundary

-

The guest and the host share nothing but a block of linear memory and a set of numbered registers. Arguments cross the boundary as (pointer, length) descriptors into guest memory; the host reads or writes those bytes, and hands larger results back through registers the guest then reads. The host bounds-checks every access.

-
-
- - - - - - - WASM guest (sandbox) - - linear memory - (ephemeral, per execution) - - export: method() -> () - no args, no return - cannot reach network / - clock / disk directly - - - - - ABI boundary - - host call (ptr,len) - - result via register - - - - - Host runtime - registers · buffers - bounds checks · limits - crypto · time · random - - - - - - replicated state - - private state - - ordered index - node storage - - -
-
- -
-

What the host offers

-

The ABI is a fixed catalog of host functions. The guest never does anything outside this list — which is exactly why execution stays deterministic and bounded. The families:

- - - - - - - - - - -
FamilyExamplesPurpose
Context / inputcontext_id, executor_id, input, read_registerRead who is calling, in which context, with what arguments.
Statestorage_read, storage_write, storage_removeRead and mutate replicated context state. Writes are buffered (below).
Outputvalue_return, log, emitReturn a result, log a line, emit an application event.
Commitcommit / root-state functionsFinalize buffered writes into a new state root and an artifact.
Cryptoed25519_verifyVerify signatures inside guest logic (deterministic).
Host-providedtime_now, random_bytesThe only sources of non-determinism — injected by the host, never read by the guest itself.
Cross-contextxcallQueue a fire-and-forget call into another context on the same node.
Blobsblob_*Stream large binary artifacts outside the state tree.
-
- -
-

Three storage layers

-

Not everything a guest writes should replicate. The ABI exposes three distinct layers, and the guest chooses per write:

-
-

Replicated

The shared context state. Writes here become operations that sync to every member and fold into the scope root. This is "the app's state."

-

Private

Node-local state that never leaves the machine and never enters the scope root. For per-node caches and secrets.

-

Ordered index

A node-local, byte-ordered secondary index for range and prefix queries — a local accelerator, also never replicated.

-
-

Only the replicated layer produces operations. The other two are conveniences that stay on one node and have no effect on convergence.

-
- -
-

Determinism is the contract

-

Every node that applies the same operation must reach the same state, so execution must be a pure function of (prior state, input, injected host values). The runtime enforces this:

-
    -
  • Writes buffer, then commit. A storage_write updates an in-execution buffer that later reads can see, but nothing is persisted until commit. If the guest traps or never commits, every change is discarded — execution is transactional.
    • -
  • Non-determinism is injected, not reached. A guest can't read the clock or an RNG on its own; it asks the host via time_now / random_bytes, so those values are controlled and recorded rather than free.
    • -
  • Resources are bounded. Memory pages, stack depth, and the counts and sizes of logs, events, cross-calls, and storage keys/values are all capped. Exceeding a limit traps the execution — which, being uncommitted, changes nothing.
    • -
  • The method shape is fixed. Exported methods take no arguments and return nothing; input arrives through input() and results leave through value_return, so a method signature can't smuggle in hidden state.
    • -
-
- -
-

From execution to operations

-

A successful commit yields an outcome: the return value, any logs and events, the cross-context calls to dispatch, the new state root hash, and an artifact describing the committed state changes.

-

That artifact is the bridge back to the core: its committed writes are packaged as data-plane operations (Put / Delete) in the context scope — each stamped with the executor's identity, the scope's current heads as parents, and a hybrid timestamp. From there they are exactly the operations of Chapters 3–4: appended to the DAG, folded into the root, and ready to broadcast.

-
handoff

Execution ends where the write path begins.

-

This chapter stops at "an outcome exists." Chapter 8 picks it up: how the outcome is signed, persisted atomically, and broadcast to the context's members.

-
- -
-

Specification

-

The exact host ABI. All host functions live in the WASM env namespace. Pointers and lengths are u64 offsets into the guest's linear memory; all integers are little-endian.

- -

Calling convention

-
    -
  • A buffer is passed as a 16-byte descriptor { ptr: u64, len: u64 } at a guest memory offset — the host reads len bytes from ptr.
    • -
  • Larger results are staged in host registers addressed by a u64 id. register_len(id) returns the length or u64::MAX if absent; read_register(id, dest_ptr) copies it into guest memory and returns 1 on success, 0 on a size mismatch.
    • -
  • A return value is written by the guest as a ValueReturn tagged union — a 1-byte tag (0=Ok, 1=Err) then, at offset 8, a 16-byte buffer descriptor.
    • -
- -

Host functions (the env namespace)

- - - - - - - - - - - - - - -
FamilySignature
Inputcontext_id(reg: u64), executor_id(reg: u64), input(reg: u64), xcall_origin(reg: u64) -> u32
Registersregister_len(id: u64) -> u64, read_register(id: u64, dest: u64) -> u32
Replicated statestorage_read(key: u64, reg: u64) -> u32, storage_write(key: u64, val: u64, reg: u64) -> u32, storage_remove(key: u64, reg: u64) -> u32
Private stateprivate_storage_read / write / remove(…) — same shapes, node-local
Ordered indexstorage_index_set / remove / remove_prefix / scan / last(…) — node-local
Outputvalue_return(ptr: u64), log_utf8(ptr: u64), emit(ptr: u64), emit_with_handler(ptr, handler)
Commitcommit(root_hash: u64, artifact: u64)
Cryptoed25519_verify(sig: u64, pk: u64, msg: u64) -> u32
Host-providedtime_now(dest: u64), random_bytes(dest: u64)
Cross-contextxcall(ptr: u64)
Blobsblob_create() -> u64, blob_write / read / open / close(…), blob_announce_to_context(…)
Panicpanic(loc: u64), panic_utf8(msg: u64, loc: u64)
-

A storage read/write/remove returns 1 on hit and 0 on miss. The index scan result is encoded as u32 count, then for each pair u32 key-len ‖ key ‖ u32 value-len ‖ value.

- -

Resource limits (defaults)

- - - - - - - - - -
LimitDefaultLimitDefault
memory pages1024 (64 MiB)stack size200 KiB
registers100register capacity1 GiB total
max register100 MiBlogs1024 × 16 KiB
events100 × 16 KiBxcalls100 × 16 KiB
storage key1 MiBstorage value10 MiB
blob handles100blob chunk10 MiB
method name256 Bmodule size20 MiB
-

Exceeding any limit traps the execution; because nothing has committed, the trap leaves state unchanged.

- -

Outcome

-
-struct Outcome {
-  returns: Result<Option<Vec<u8>>, FunctionCallError>,
-  logs: Vec<String>,
-  events: Vec<Event>, // { kind: String, data: Vec<u8>, handler: Option<String> }
-  xcalls: Vec<XCall>, // { context_id: [u8;32], function: String, params: Vec<u8> }
-  root_hash: Option<[u8; 32]>,
-  artifact: Vec<u8>,
-  migration_witness: Option<Vec<u8>>,
-} -
-
- -
-

Where this leads

-

We now know where data operations originate. The other planes — access control, membership, admin — don't come from WASM; they come from governance, which is Chapter 6.

-
- -
-← State, Projection & the Root Hash -Governance → -
- -
-
- - - - - diff --git a/architecture/protocol/governance.html b/architecture/protocol/governance.html deleted file mode 100644 index df02f6d858..0000000000 --- a/architecture/protocol/governance.html +++ /dev/null @@ -1,204 +0,0 @@ - - - - -Governance — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/Governance
-

Governance

-

Chapter 6 — membership and authority, encoded as operations like everything else.

- -
-

Governance is just operations

-

There is no separate governance engine, no consensus round, and no quorum. The access-control, membership, and admin planes from Chapter 3 are governance — they are operations in a governance scope, folded by the same last-writer-wins projection, summarized by the same scope root. A membership change converges exactly the way a data write does.

-

What this chapter adds is meaning: what those governance operations say, the structure they maintain, and how the system decides whether an author was allowed to write one.

-
- -
-

The hierarchy

-

Governance scopes form a tree (the scope tree from Chapter 1):

-
    -
  • A namespace is the root governance scope — a group with no parent. It anchors identity, admin, and policy for everything beneath it.
    • -
  • Subgroups nest under it, each its own scope with its own key and members.
    • -
  • Contexts (the running applications) attach to a group; a context's members are the group's members.
    • -
-

A subgroup is either Open or Restricted. In an Open subgroup, a parent member who holds the "join open subgroups" capability is a member by inheritance — no explicit operation needed. A Restricted subgroup is closed: membership requires a direct, deliberate admission. Because membership only ever narrows as you descend, pointing an operation "up" the tree never leaks visibility — the property Chapter 1 relied on.

-
- -
-

The founder

-

A scope tree has to start somewhere, and that start can't appeal to a prior admin — there isn't one yet. So a namespace begins with a single genesis operation that names its founder. It is self-authorizing: its only checks are that it is signed by the founder it names and that it has no parents (it is genuinely the first operation). From it, the founder is recorded as admin and owner, and every node that later replays the DAG derives the same founder — authority is established by a replayable operation, not by out-of-band trust.

-
- -
-

Membership: joining

-

Joining is a two-part dance, because a new member needs two things: a row in the membership state, and the scope key to actually read the scope's encrypted operations. The key is delivered privately, wrapped to the joiner's public key (ECDH), by existing members who already hold it.

-
-
-

The membership operation is public on the governance scope; the key delivery is a private, wrapped envelope. Once the joiner unwraps the key, every prior and future operation in the scope becomes readable, and they fold the scope just like anyone else.

-
- -
-

Membership: leaving and removal

-

There are two ways a member stops being a member, and the difference is entirely about the scope key.

-
-

Leave (self-initiated)

A member removes themselves. No key rotation happens — the leaver already holds the key and could have copied it, so rotating would be theater. Leaving is just a membership operation that drops their own row.

-

Removal (admin-initiated)

An admin removes someone else. Here the key is rotated: a fresh scope key is generated and delivered, wrapped, to every remaining member — so the removed member can read nothing authored after their removal. This is forward secrecy.

-
-
-
-

Two invariants guard both paths: the scope's owner cannot be removed involuntarily and cannot self-leave without first transferring ownership, and a removal that would leave a group with no admin is rejected — the tree can never be orphaned.

-
- -
-

Roles, capabilities, and the owner

-

Authority is layered. A member has a role for coarse rights, and fine-grained capabilities for specific actions; the owner is a distinguished identity sitting above roles.

- - - - - - -
RoleCan do
AdminAll governance operations: manage members, set policy and capabilities, manage subgroups.
MemberRead and write context state. Governance actions only with the matching capability.
ReadOnlyRead state; writes are rejected, locally and on peers.
ReadOnlyTeeA read-only fleet node admitted by hardware attestation; evicted with a self-purge signal.
-

Capabilities are individual grants a non-admin can hold — for example: create contexts, invite members, join open subgroups, manage members, manage the application, or create/delete/configure subgroups. Admins pass every capability check implicitly; defaults can be set per group and overridden per member.

-

The owner is not a role but a stored identity with exclusive rights — transfer ownership, delete the group or namespace — and the removal immunity noted above. Every other admin is a regular admin.

-
- -
-

Authorization: judged at the author's cut

-

The decision "was this author allowed to write this operation?" reuses the primitive from Chapter 4: the projection resolves membership and capabilities at the operation's causal cut — the governance state the author had observed — not at the receiver's present.

-
the rule

Forward-only authority.

-

An operation is authorized against the membership and access-control state that existed at its parents. So a write authored while you were still a member stays valid even if a peer applies your removal first; and a governance action you took as admin stands even if you were later demoted. Every node reaches the same verdict because every node folds the same cut — authority converges for free, with no coordination.

-

The same mechanism covers data writes: an entity's writer set (the access-control plane) is resolved at the cut, and a write is accepted only if its author was an authorized writer at that point.

-
- -
-

Specification

- -

Capability bits

-

Capabilities are a u32 bitmask. Each grant is a fixed bit position:

- - - - - - - - - - - -
BitValueCapability
00x0001CAN_CREATE_CONTEXT
10x0002CAN_INVITE_MEMBERS
20x0004CAN_JOIN_OPEN_SUBGROUPS
30x0008MANAGE_MEMBERS
40x0010MANAGE_APPLICATION
50x0020CAN_CREATE_SUBGROUP
60x0040CAN_DELETE_SUBGROUP
70x0080CAN_MANAGE_VISIBILITY
80x0100CAN_MANAGE_METADATA
-

Roles are a distinct enum: Admin, Member, ReadOnly, ReadOnlyTee. Admins pass every capability check implicitly.

- -

The signed governance op

-

Governance operations travel as a signed envelope. The group-scoped form:

-
-struct SignedGroupOp {
-  version: u8, // current schema version = 8
-  group_id: [u8; 32],
-  parent_op_hashes: Vec<[u8; 32]>, // DAG parents
-  signer: PublicKey,
-  nonce: u64, // per-signer monotonic, starts at 1
-  op: GroupOp,
-  signature: [u8; 64],
-} -
-

The content hash (and the bytes the signature covers) are domain-separated:

-
-op_id = SHA-256( DOMAIN ‖ borsh(signable) ) // signable = all fields except signature
-signature = Ed25519_sign( signer_secret, DOMAIN ‖ borsh(signable) )
-
-// group ops     DOMAIN = b"calimero.group.v1"
-// namespace ops DOMAIN = b"calimero.namespace.v1"  (SignedNamespaceOp, version = 2) -
- -

Replay protection

-

Each (group, signer) tracks a nonce window: a contiguous applied floor plus a sparse set of out-of-order applied nonces above it. A nonce is a duplicate if nonce ≤ floor or it is already in the set. This is checked before apply, and tolerates concurrent siblings (same author, different parents) arriving out of order without false-flagging them. Nonces start at 1; 0 is never issued.

- -

Key delivery: the wrapped envelope

-

A scope key is delivered wrapped per recipient:

-
-struct KeyEnvelope {
-  recipient: PublicKey,
-  ephemeral_pk: PublicKey, // sender's ephemeral key for this envelope
-  nonce: [u8; 12], // AES-GCM nonce
-  ciphertext: Vec<u8>, // AES-256-GCM(scope_key)
-} -
-
    -
  • Shared secret — X25519-style ECDH on the Curve25519 form of the two identities (sender ephemeral × recipient): the 32-byte compressed point is the AES-256 key (no separate KDF).
    • -
  • AEAD — AES-256-GCM, 12-byte nonce, empty additional data, GCM tag appended inline.
    • -
  • Rotation — a removal carries a KeyRotation { new_key_id = SHA-256(new_key), envelopes } with one envelope per remaining member; the removed member gets none.
    • -
-

A scope key is identified on the wire by key_id = SHA-256(scope_key), so a receiver can look up which delivered key decrypts a given operation.

- -

Owner vs admin

-

The owner is stored alongside admin in the group metadata (owner_identity vs admin_identity). A group may have several admins but exactly one owner. Only the owner may sign TransferOwnership or delete the group/namespace, and the owner is immune to involuntary removal. Ownership is itself folded into the governance hash, so an op signed before vs after a transfer resolves against different state.

-
- -
-

Where this leads

-

We now have the full meaning of all four planes and the rules that govern who may write them. Everything so far has been local — operations, folds, authority — assuming the operations are simply present. Chapter 7 turns to how they actually move between nodes: the network and the wire protocol.

-
- -
-← Application Execution -Networking & the Wire Protocol → -
- -
-
- - - - - diff --git a/architecture/protocol/identities.html b/architecture/protocol/identities.html deleted file mode 100644 index 6532787fa6..0000000000 --- a/architecture/protocol/identities.html +++ /dev/null @@ -1,79 +0,0 @@ - - - - -Identities & Keys — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/Identities & Keys
-

Identities & Keys

-

Chapter 2 — the three kinds of key, and what each one is for.

- -
-

Three keys, three jobs

-

It is easy to conflate "who a node is on the network" with "who authored an operation" — they are different keys with different lifetimes. Calimero uses three distinct kinds of key, and keeping them separate makes the rest of the protocol clear.

-
-
- - - - - Transport identity - Ed25519 → libp2p PeerId · per node · names the node on the wire - node - - - - Member identity - Ed25519 → namespace keypair · per namespace · authors & signs operations - namespace - - - - Scope key - symmetric · per scope · encrypts operations · delivered ECDH-wrapped - scope - - -
-
- -
-

Transport identity — naming a node

-

Every node has an Ed25519 keypair used only by the network layer. Its public key is the node's peer id: the address other nodes dial, the identity the encrypted transport authenticates. It says nothing about authority — it answers "which box am I talking to," not "who wrote this." A node can serve many namespaces over one transport identity.

-
- -
-

Member identity — authoring operations

-

Authority lives in the member identity: an Ed25519 keypair held per namespace, generated on first use and persisted. It is the author on every operation and the key that signs an operation's id (Chapter 3). Membership, roles, and capabilities (Chapter 6) are all expressed in terms of member public keys.

-

Because identity is per namespace, the same node participating in two namespaces presents two unrelated member keys — identity isolation across application instances, independent of the single transport identity underneath.

-
- -
-

Scope key — reading a scope

-

A scope's operations are encrypted at rest and on the wire under a symmetric scope key (Chapter 1: every scope owns a key). Holding the key is what lets a member read a scope; lacking it, a node sees only opaque ciphertext.

-

Scope keys are never sent in the clear. They are delivered wrapped to a specific member's public key using an ECDH exchange between that member key and the sender's — the KeyDelivery step in Chapter 6's join flow. Removing a member rotates the scope key and re-wraps the fresh one to everyone who remains, which is how forward secrecy is enforced.

-
- -
-

How they compose

-

Put together, a single received operation touches all three: it arrives over a connection authenticated by transport identities, its ciphertext is decrypted with the scope key, and its authorship is verified against the author's member key before anything trusts it. Chapters 7 and 9 show that sequence on the wire.

-
- -
-← Concepts & Scopes -Operations & the Causal DAG → -
- -
-
- - - - diff --git a/architecture/protocol/index.html b/architecture/protocol/index.html deleted file mode 100644 index 6a215309f4..0000000000 --- a/architecture/protocol/index.html +++ /dev/null @@ -1,152 +0,0 @@ - - - - -Protocol Reference — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference
-

Protocol Reference

-

How a Calimero node works, told as the life of one operation — enough to reimplement the protocol from scratch.

- -
-

What this is

-

This is the implementation reference: a design-level description of the Calimero protocol aimed at someone building an independent node in another language. It explains how and why the pieces work — with diagrams, step-by-step flows, and pseudocode — rather than touring the Rust crates. It is an architecture guide, not a normative standard: there is no MUST/SHOULD conformance language here yet.

-

For installing, the SDK, running merod/meroctl, and building apps, see the For Builders and For Operators sections — this reference is the layer underneath them.

-
- -
-

The one idea everything hangs on

-

A Calimero node runs sandboxed WASM applications over replicated, causally-consistent shared state. There is no global consensus and no quorum. Every change — a data write, an access-control change, a membership change, an admin action — is a signed operation in a content-addressed DAG.

-

Each node folds that DAG deterministically into materialized state. Because the fold is order-independent (last-writer-wins per slot, keyed on a hybrid clock), any two nodes that have seen the same set of operations compute the same state and the same scope root hash. Nodes don't prevent divergence — they detect it by comparing scope roots, and heal it by syncing missing operations.

-

Hold onto that sentence — "a signed op DAG that folds to a state plus a root hash, with divergence detected, not prevented" — and every chapter below is a detail of it.

-
- -
-

The life of an operation

-

The whole reference follows one operation from birth to convergence. This is the spine:

-
-
-

Chapters 8–10 walk the right half of this diagram (write, receive, sync) in full detail; chapters 3–4 define the fold; chapters 1–2 define the nouns.

-
- -
-

How this reference is written

-

Follow the data, not the crates. The reference is the story of one operation's life, end to end. A crate-by-crate map is an appendix, not the main thread.

-

Describe the target model. Calimero uses a single unified operation log across data and governance. Where older mechanisms are mid-retirement, this reference documents where the protocol is going, not the transitional scaffolding.

-

Name things by behavior. Mechanisms are named for what they do (e.g. "post-sync governance reconciliation"), never by internal milestone codes.

-

Diagrams and pseudocode first. Every non-trivial flow gets a diagram and, where it clarifies, language-neutral pseudocode — not Rust excerpts.

-
- -
-

Chapters

-

Status: draft being written · planned outlined, not yet authored.

-
- -
1 -

Concepts & Scopes

-

The nouns: node, identity, application, operation, and the central abstraction — the scope. What replicates together and who can see it.

- draft
- -
2 -

Identities & Keys

-

The two identity layers — transport (libp2p peer) vs member (the key that authors and authorizes ops) — and how scope keys are wrapped and delivered.

- draft
- -
3 -

Operations & the Causal DAG

-

The operation envelope, content-addressed ids, parents and causality, the hybrid clock, and the four planes (data, access-control, membership, admin) in one model.

- draft
- -
4 -

State, Projection & the Root Hash

-

How the DAG folds into readable state, the per-slot last-writer-wins rule, and how scope_root = H(entities · acl · governance) becomes the convergence signal.

- draft
- -
5 -

Application Execution

-

How one method call runs in WASM: the host ABI, buffered reads/writes, determinism rules, and how an execution becomes an operation.

- draft
- -
6 -

Governance

-

The scope tree, membership lifecycle (join, leave, remove with key rotation), roles and capabilities, and authorization evaluated at an operation's causal cut.

- draft
- -
7 -

Networking & the Wire Protocol

-

The libp2p stack, scope-to-topic mapping, eager gossip vs pull-based streams, the message catalog, and the encryption layers.

- draft
- -
8 -

The Write Path

-

Request → execute → build operation → persist atomically → broadcast, end to end.

- draft
- -
9 -

The Receive & Apply Path

-

Verify → authorize at the governance cut → decrypt → DAG insert (buffering) → merge → run event handlers.

- draft
- -
10 -

Sync & Convergence

-

Triggers, peer discovery, the handshake, the convergence verdict (converged / governance-diverged / data-diverged), protocol selection, and post-sync reconciliation.

- draft
- -
A -

Appendix · Storage Schema

-

The on-disk format: column families, the exact key byte-layouts (context_id ‖ op_id, the Group prefix scheme), and value encodings — reimplementation-grade.

- draft
- -
-
- -
-
- - - - - diff --git a/architecture/protocol/networking.html b/architecture/protocol/networking.html deleted file mode 100644 index dc86c6ad7c..0000000000 --- a/architecture/protocol/networking.html +++ /dev/null @@ -1,176 +0,0 @@ - - - - -Networking & the Wire Protocol — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/Networking & the Wire Protocol
-

Networking & the Wire Protocol

-

Chapter 7 — how operations actually move between nodes.

- -
-

The transport stack

-

Calimero is built on libp2p. Nodes connect over TCP (with Noise/TLS) or QUIC, multiplex streams with Yamux, and run a set of behaviours for discovery, gossip, and direct exchange. The application logic sits on top and never touches sockets directly — it speaks in terms of topics and streams.

-
-
- - - - - Calimero node logic — contexts · governance · sync - - - - gossipsub - eager broadcast (topics) - deltas · governance · heartbeats - - streams · request-response - pull, point-to-point - sync · snapshots · blobs · key recovery - - - - discovery — rendezvous · mDNS · Kademlia DHT · relay / hole-punch for NAT - - - - transport — TCP + Noise/TLS · QUIC · Yamux multiplexing - authenticated by transport identity (Chapter 2) - - -
-
- -
-

Two channels: gossip and streams

-

Everything on the wire travels one of two ways, and the choice is about eager vs. on-demand:

-
-

Gossip (eager broadcast)

A node publishes to a topic; every subscriber receives it through the gossipsub mesh. This is how fresh operations propagate the instant they're created — fire it once, the mesh fans it out. Best-effort: a node that was offline simply misses it.

-

Streams (pull, point-to-point)

A node opens a direct, length-delimited stream to one peer and asks for something specific — missing operations, a snapshot, a blob, a key. This is how a node catches up on anything gossip didn't deliver. Reliable and targeted.

-
-

Gossip keeps everyone roughly current; streams make them exactly current. Chapter 10 is essentially "the stream channel, driven by the scope-root comparison."

-
- -
-

Topics follow the scope tree

-

Subscriptions map directly onto scopes, so a node only hears about what it belongs to:

-
    -
  • context/<id> — a context's data operations and heartbeats.
    • -
  • group/<id> — a group's governance operations.
    • -
  • ns/<id> — a namespace's root governance operations and heartbeats.
    • -
-

Gossip's mesh is membership-biased: peers known to be members are preferred when forming the mesh, while unknown peers are never penalized — so a healthy cluster stays well-connected without excluding newcomers during cold start.

-
- -
-

The message catalog

-

All messages are Borsh-encoded (compact, deterministic binary). They fall into the two channels:

- - - - - - - - - - -
ChannelMessageCarries
GossipStateDeltaa context's data operation(s): author, parents, hlc, encrypted artifact, new root, signature.
GossipGovernanceDeltaa governance operation (root op in clear, group op encrypted).
GossipHeartbeatcurrent DAG heads + root, published periodically for divergence detection.
StreamDeltaRequest / Responsefetch a specific operation by id.
StreamDagHeadsRequest / Responseexchange heads + roots during a sync handshake.
StreamTreeNode / LevelWisecompare state trees to locate exactly what differs (Chapter 10).
StreamSnapshot*transfer a full materialized scope when incremental sync isn't enough.
StreamKeyRequest / Responserecover a scope key wrapped to the requester.
-
- -
-

Three layers of encryption

-

Confidentiality is defense-in-depth, and the three layers map exactly onto Chapter 2's three keys:

-
    -
  • Transport — Noise/TLS encrypts every connection, authenticated by the peers' transport identities.
    • -
  • Payload — an operation's sensitive content is encrypted under its scope key, so even a subscribed non-member on the topic sees only ciphertext.
    • -
  • Key wrapping — scope keys themselves move ECDH-wrapped to a recipient's member key, never in the clear.
    • -
-

Root operations that must be readable by everyone in a namespace (a join announcement, a key delivery envelope) travel in clear at the payload layer but are still inside the encrypted transport.

-
- -
-

Specification

-

Every message is Borsh. Recall the encoding: an enum is a single-byte variant index then its fields in order; a Vec<u8> is a u32 little-endian length then the bytes; an Option is a 1-byte present/absent flag.

- -

Stream envelope

-

Point-to-point messages are wrapped in a StreamMessage with a rolling nonce:

- - - - - - -
VariantFields
Initcontext_id: ContextId, party_id: PublicKey, payload: InitPayload, next_nonce: Nonce
Messagesequence_id: u64, payload: MessagePayload, next_nonce: Nonce
OpaqueError— (failure without leaking state)
NotMaterialized— (member, but hasn't joined this context yet)
- -

Stream requests (InitPayload) and responses (MessagePayload)

- - - - - - - - -
RequestResponsePurpose
DagHeadsRequestDagHeadsResponse { dag_heads: Vec<[u8;32]>, root_hash: Hash, scope_root: Option<Hash> }sync handshake
DeltaRequest { delta_id }DeltaResponse { delta, author_id, governance_position_blob?, delta_signature?: [u8;64] }fetch one op
TreeNodeRequest { node_id, max_depth?: u8 }TreeNodeResponse { nodes: Vec<TreeNode>, not_found }tree-diff sync
LevelWiseRequest { level: u32, parent_ids? }LevelWiseResponse { level, nodes, has_more_levels, deleted_children }level-wise sync
SnapshotStreamRequest { boundary_root_hash, page_limit: u16, byte_limit: u32, resume_cursor? }SnapshotPage { payload (lz4), uncompressed_len, cursor?, total_records }full transfer
GroupKeyRequest { group_id, requester_public_key }GroupKeyResponse { key_envelope_bytes, responder_identity }key recovery
- -

Gossip broadcasts (BroadcastMessage)

-

The data-plane broadcast carries the operation and everything a peer needs to verify, decrypt, and authorize it:

-
-StateDelta {
-  context_id: ContextId, author_id: PublicKey,
-  delta_id: [u8;32], parent_ids: Vec<[u8;32]>, hlc: HybridTimestamp,
-  root_hash: Hash,
-  artifact: bytes, // ENCRYPTED under the scope key
-  nonce: Nonce, events: Option<bytes>, // events also encrypted
-  governance_position: Option<GovernanceParentEdge>, // the authorization cut
-  key_id: [u8;32], // = SHA-256(scope_key)
-  delta_signature: Option<[u8;64]>, producing_app_key: Option<[u8;32]>,
-} -
- - - - - -
Other broadcastFields
HashHeartbeatcontext_id, root_hash: Hash, dag_heads: Vec<[u8;32]>
NamespaceGovernanceDeltanamespace_id, delta_id, parent_ids, payload: Vec<u8> (signed gov op)
NamespaceStateHeartbeatnamespace_id, dag_heads: Vec<[u8;32]>
- -

Wire limits

- - - - - -
ConstantValueConstantValue
tree request depth16DAG heads / msg100
signed gov op payload64 KiBsnapshot page (default)256 KiB
entities / page1000decompressed page cap8 MiB
-

Signatures are fixed [u8;64] (Ed25519); hashes and ids are [u8;32]; a Nonce is the crypto-layer nonce, distinct from the raw [u8;32] nonces used by node-discovery messages.

-
- -
-

Where this leads

-

We can now move operations between nodes. The next three chapters trace the round trip end to end: Chapter 8 creates and broadcasts an operation, Chapter 9 receives and applies one, and Chapter 10 reconciles whatever the eager channel missed.

-
- -
-← Governance -The Write Path → -
- -
-
- - - - diff --git a/architecture/protocol/operations.html b/architecture/protocol/operations.html deleted file mode 100644 index 77f59568e7..0000000000 --- a/architecture/protocol/operations.html +++ /dev/null @@ -1,264 +0,0 @@ - - - - -Operations & the Causal DAG — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/Operations & the Causal DAG
-

Operations & the Causal DAG

-

Chapter 3 — the atomic unit of change, and how a scope orders it.

- -
-

The operation envelope

-

Every change in a scope — a data write, an access-control change, a membership change, an admin action — is one operation. It is a small, signed, self-describing envelope. There is only one kind of envelope; the payload says which plane the change belongs to.

-
-// the operation envelope (calimero-op)
-struct Op {
-  id: Hash,    // content address — H(everything below except signature)
-  scope: ScopeId, // which replication/convergence domain this belongs to
-  parents: Vec<Hash>, // causal predecessors (the DAG edges)
-  author: PublicKey, // member identity that wrote it
-  hlc: HybridTimestamp, // causally-monotonic clock at author time
-  payload: OpPayload, // the change itself, across four planes
-  expected_scope_root: Hash, // author's predicted root — an assertion, never trusted
-  signature: Sig, // Ed25519 by `author` over `id`
-} -
-

The rest of this chapter is just: how the id is formed, what the payload can say, and how parents turn a pile of operations into an ordered DAG.

-
- -
-

The id is the content

-

An operation's id is not assigned — it is derived by hashing the operation's contents. Same contents, same id, on every node. This is what makes operations safe to gossip, deduplicate, and reference by hash.

-
-// content address — deterministic on every node
-id = SHA-256(
-  scope  ‖
-  u64_le(count(parents)) ‖ sorted(parents) // sorted ascending ⇒ order can't change the id
-  author ‖
-  borsh(hlc) ‖
-  borsh(payload)
-) -
-

Two details carry weight:

-
    -
  • Parents are sorted (ascending, by their raw 32-byte value) before hashing, so the set of parents determines the id — the order an author happened to list them in cannot.
    • -
  • The parent list is length-prefixed with a little-endian u64 count, so the boundary between "parents" and "author" is unambiguous and two different field layouts can never collide into the same preimage.
    • -
-

The signature is excluded from the preimage and is an Ed25519 signature over the resulting id. Exact byte layout and the reimplementer's checklist are in the specification at the foot of this chapter.

-
- -
-

Signature & trust boundary

-

The author signs the id with their member key. The signature is deliberately not folded back into the id (a hash can't include a signature of itself), so the id stays purely a function of content while the signature proves who authored that content.

-
rule

Callers must verify the signature before trusting an operation.

-

The fold and the authorization layer (Chapters 4 and 6) assume already-verified operations — they reason about content alone and perform no signature check. Feeding an unverified operation into the fold bypasses authentication entirely. Verification happens once, at the network edge (Chapter 9).

-

expected_scope_root is the author's prediction of the root after this operation applies. It is an assertion, not an input: it is not signed and never grants authority. Peers recompute their own scope root and compare; a tampered value can at worst flag a divergence the recompute would have caught anyway. Security never depends on this field.

-
- -
-

One envelope, four planes

-

The payload is a single enum spanning four planes of change. Folding them into one envelope — one DAG, one ordering, one convergence signal — is the whole point of the unified model (Chapter 1).

-
-

Data state

-
  • Put { entity, value }
  • Delete { entity }
-

Access control acl

-
  • SetWriters { object, writers } — who may write an entity, with what capability mask
-

Membership who

-
  • MemberAdded { group, member, role }
  • MemberRemoved { group, member }
-

Admin & structure gov

-
  • AdminChanged, PolicyUpdated
  • SubgroupCreated / Reparented / Deleted
  • capability grants, visibility
-
-

A fifth, trivial variant — Noop — carries no change. It exists only so the DAG can have a node that merges several heads without asserting anything (Chapter 10 uses it to consolidate tips).

-
- -
-

The causal DAG

-

A scope's operations form a directed acyclic graph. Each operation's parents are the operations its author had already seen — its causal predecessors. The DAG is what lets independent authors write concurrently and still converge.

-
-
- - - - - - - causal time → - - - - - ∅ - genesis - - - - - - A - Put - - - - - - B - MemberAdded - - - - - - C - Put - - B and C are concurrent (neither is an ancestor of the other) - - - - - - D - merges B + C - - head - - - - - ? - - - P - pending — parent missing - - -
-

The rules that govern this graph are short:

-
    -
  • Genesis is the zero hash (∅), and it is always considered applied — every scope's history roots there.
    • -
  • Heads are the current tips: operations no later operation references yet. A new operation takes the current heads as its parents.
    • -
  • Concurrency is simply two operations where neither is an ancestor of the other (B and C above). The fold resolves them deterministically — Chapter 4.
    • -
  • An operation can only be applied once all its parents are applied. One whose parents haven't arrived (P) is held in a pending buffer until they do.
    • -
-
- -
-

Appending and applying

-

Adding an operation to the local DAG is the same procedure whether it was authored here or received from a peer. It is a buffer-and-cascade loop, never recursion or blocking:

-
-
-

In words:

-
1

Receive the operation and look at its parents.

If every parent is already applied, the operation is ready. If any parent is missing, store it in the pending buffer keyed by the missing parents and stop — it will be revisited.

-
2

Apply a ready operation by folding it into the projection.

This is the deterministic step from Chapter 4. Applying never has side effects on consensus — it only updates materialized state and advances the heads.

-
3

Cascade.

Applying an operation can unblock pending children whose last missing parent just arrived. Scan the buffer, apply any now-ready operations, and repeat until nothing new becomes ready. This is how a backfilled history snaps into place in one pass.

-
- -
-

Specification

-

The exact wire-level details a second implementation must match byte-for-byte to compute identical operation ids. All multi-byte integers are little-endian; composite fields use Borsh encoding.

- -

Id preimage byte layout

- - - - - - - - -
FieldBytesEncoding
scope32raw ScopeId
parent count8u64 little-endian
parents32 × neach a 32-byte hash, sorted ascending
author32raw Ed25519 public key
hlcborshhybrid timestamp (below)
payloadborshenum: 1-byte variant tag, then fields in order
-
-// id = SHA-256 over the concatenation, in this exact order
-sorted = sort_ascending(parents)
-h = Sha256()
-h.update( scope )        // 32 bytes
-h.update( u64_le(len(sorted)) ) // 8 bytes
-for p in sorted: h.update( p ) // 32 bytes each
-h.update( author )       // 32 bytes
-h.update( borsh(hlc) )
-h.update( borsh(payload) )
-id = h.finalize()      // 32 bytes
-signature = Ed25519_sign( member_secret, id ) -
-

Borsh reminder for reimplementers: a Vec<u8> is a u32 little-endian length followed by the bytes; a BTreeMap is a u32 length followed by entries in ascending key order; an enum is a single-byte variant index then its fields.

- -

Payload variants

- - - - - - - - - - - - - - - -
PlaneVariantFields
DataPutentity: Id, value: Vec<u8>
Deleteentity: Id
AccessSetWritersobject: Id, writers: BTreeMap<PublicKey, OpMask> (mask is a 1-byte bitflag)
MembershipMemberAddedgroup: ContextGroupId, member: PublicKey, role: GroupMemberRole
MemberRemovedgroup: ContextGroupId, member: PublicKey
Admin / structureAdminChangednew_admin: PublicKey
PolicyUpdatedpolicy_bytes: Vec<u8>
SubgroupCreatedchild: ScopeId, parent: ScopeId, restricted: bool, admin: PublicKey
SubgroupReparented / SubgroupDeletedchild, new_parent / scope
SubgroupVisibilitySetscope: ScopeId, restricted: bool
CapabilityDefaultCapabilitiesSetgroup: ContextGroupId, capabilities: u32
MemberCapabilitySetgroup, member, capabilities: u32
GraphNoop— (folds to nothing)
- -

The hybrid logical clock

-

The hlc is a hybrid logical clock: a wall-clock-ish value that is also nudged forward on observation, so it stays causally monotonic without depending on synchronized clocks.

-
    -
  • Timestamp — a 64-bit NTP64 value (RFC-5909): high 32 bits seconds, low 32 bits fraction, with the low 16 bits serving as a logical counter that increments when two events share a physical instant.
    • -
  • Node id — a u128 unique per clock instance, making every timestamp globally unique without coordination and giving a final deterministic tiebreak.
    • -
  • Anti-drift — a timestamp more than a few seconds ahead of local time is rejected, bounding the damage a skewed peer can do.
    • -
-

Comparison is lexicographic over (timestamp, node id). Governance operations are authored with a zero hlc, which is why their ordering falls to the generation component of the fold stamp in Chapter 4 rather than to wall-clock time.

-
- -
-

Where this leads

-

We now have a content-addressed, signed operation and a DAG that orders operations causally and buffers what it can't yet apply. The one thing we deferred is the actual apply — how folding the DAG produces readable state and the single hash that tells two nodes whether they agree. That is Chapter 4.

-
- -
-← Identities & Keys -Projection & the Root Hash → -
- -
-
- - - - - diff --git a/architecture/protocol/projection.html b/architecture/protocol/projection.html deleted file mode 100644 index f75a8f5a14..0000000000 --- a/architecture/protocol/projection.html +++ /dev/null @@ -1,223 +0,0 @@ - - - - -State, Projection & the Root Hash — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/State, Projection & the Root Hash
-

State, Projection & the Root Hash

-

Chapter 4 — folding the DAG into readable state and one comparable hash.

- -
-

From DAG to state: the fold

-

A scope's DAG is the source of truth, but you don't query a DAG — you query state. The projection is the deterministic function that replays a scope's operations into materialized state plus a single summary hash, the scope root.

-
-
- - - - - - - Scope DAG - - - - - - signed operations - - - fold - - - - Projection — per-slot LWW - entities → value · stamp - acl → writers · stamp - groups → members · stamp - admin → identity · stamp - policy → bytes · stamp - subgroups → tree · stamp - caps → grants · stamp - - - hash - - - - scope_root - H( entities ‖ - acl ‖ governance ) - - -
-

The fold has one essential property: it is order-independent. Replay the same set of operations in any order — the order they were authored, gossiped, or backfilled — and you get the identical state and the identical scope root. That is what makes "detect divergence, don't prevent it" work: two nodes never have to agree on an order, only on a set.

-
- -
-

Last-writer-wins, per slot

-

Order-independence comes from how each piece of state is resolved. The projection treats every addressable piece of state — an entity, an object's writer set, a member's role — as a slot. Each slot remembers the stamp of the operation that last wrote it, and a write only takes effect if it carries a higher stamp.

-
-// applying one operation — pure, deterministic, order-independent
-fn apply(op):
-  for (slot, value) in op.writes():
-    stamp = (op.hlc, generation(op), op.id)
-    if stamp > clock[slot]:
-      state[slot] = value
-      clock[slot] = stamp -
-

The stamp is a triple, compared left to right:

-
    -
  • hlc — the hybrid logical clock. It advances with wall-clock time but is nudged forward whenever a higher clock is observed, so it stays causally monotonic: an operation that causally follows another always carries a higher (or equal) hlc.
    • -
  • generation — the operation's depth in the DAG (its longest path from genesis). This breaks ties where hlc alone can't. Governance operations in particular are authored with hlc = 0, so generation is what orders them — a later structural change always sits deeper in the DAG than the state it supersedes.
    • -
  • id — the content hash, a final deterministic tiebreaker so two genuinely concurrent writes still resolve the same way on every node.
    • -
-

A Delete records a stamp just like a write, so a stale re-add that arrives later cannot resurrect a deleted entity — its lower stamp loses.

-
- -
-

What the projection holds

-

The fold maintains one set of slots per plane. This is the complete materialized state of a scope:

- - - - - - - - -
PlaneSlotsResolved value
Dataentitiesthe latest value written to each entity (or absent, if deleted)
Access controlaclthe writer set and capability mask for each object
Membershipgroupseach group's members and their roles
Adminroot_admin, policythe scope's admin identity and policy bytes
Structuresubgroupsthe subgroup tree: parent links and visibility
Capabilitydefault_caps, member_capsdefault and per-member capability grants
-

The data plane is the application's state; the other planes are what Chapter 6 calls governance. They all fold the same way — that's the unification.

-
- -
-

The scope root

-

The whole projection collapses to one 32-byte hash. It is the concatenation of three plane hashes, each computed over its slots in sorted, deterministic order:

-
-scope_root = H( entities_hash ‖ acl_hash ‖ governance_hash )
-
-// each is a hash over its slots in sorted key order
-entities_hash  = H( for each entity: id ‖ len(value) ‖ value )
-acl_hash     = H( for each object: id ‖ writer ‖ capability_mask )
-governance_hash = H( memberships ‖ admin ‖ policy ‖ subgroups ‖ caps ) -
-

Because every node sorts the same way and hashes the same bytes, equal operation sets give equal scope roots — and unequal state gives unequal roots. That second half is the subtle, important part:

-
key property

A change with no data effect still moves the scope root.

-

Rotate an object's writers, or remove a member, and the data entities may be byte-for-byte identical — yet because acl_hash and governance_hash are folded into the root, the scope root changes. This closes the gap that would otherwise let a silent access-control or membership split-brain hide behind matching application data. Convergence is checked on the whole scope, not just its data.

-
- -
-

Convergence is a hash comparison

-

With an order-independent fold and a single root, checking whether two nodes agree is trivial: compare scope roots. No agreement on ordering, no quorum, no coordination — just two hashes.

-
-
-

Equal roots mean the nodes hold the same state and are done. Unequal roots mean one is missing operations the other has — which is exactly the signal the sync engine acts on in Chapter 10.

-
- -
-

Reading the past: authorization at a cut

-

The fold above produces the current state. But authorization (Chapter 6) often needs the state as it was at a particular point in history — specifically, the membership and access-control state an operation's author had observed when they wrote it.

-

The projection answers this by folding only the causal cut: the operation's parents and everything those parents transitively descend from, and nothing later. This yields an access-control view "as of" that point.

-
why it matters

Forward-only authorization.

-

A write authored before a member was revoked is judged against the access-control state that existed before the revocation — so it stays valid regardless of the order in which a peer happens to receive the write and the revocation. Authority is evaluated at the author's cut, never at the receiver's present. Chapter 6 builds the full membership and capability rules on top of this primitive.

-
- -
-

Specification

-

The exact composition of the scope root. Every node must hash identical bytes in identical order, so the rules below are normative for interop. All integers are little-endian; every map is iterated in ascending key order (the determinism source).

- -
-scope_root = SHA-256( entities_root ‖ acl_hash ‖ governance_hash ) // three 32-byte inputs -
- -
important

Two entity roots — pick the right one for the wire.

-

The entities_root input has two sources. The projection can hash its own entity map (a self-contained root, used in tests and local checks). But the convergence signal exchanged on the wire folds the storage layer's Merkle root as entities_root, with this projection's acl_hash and governance_hash on top. The storage Merkle root is authoritative for the data plane; the projection never re-hashes entity state for the wire signal. A second implementation must fold its storage root here, not a re-hash of the entity map.

- -

entities_hash (projection-local data plane)

-
// for each (id, value) in entities, ascending by id
-h.update( id )      // 32 bytes
-h.update( u64_le(len(value)) )
-h.update( value )
- -

acl_hash (access-control plane)

-
// for each (object_id, writers) ascending; writers ascending by key
-h.update( object_id ) // 32 bytes
-for (writer, mask) in writers:
-  h.update( writer ) // 32 bytes
-  h.update( [mask.bits()] ) // 1 byte bitflag
- -

governance_hash (membership + admin + policy + structure + caps)

-
// 1) groups — ascending; empty groups are skipped entirely
-for (group, members) in groups, if members not empty:
-  h.update( group )
-  for (member, role) in members: h.update( member ‖ [role_byte] )
-// 2) root admin
-h.update( admin ? [1] ‖ admin : [0] )
-// 3) policy
-h.update( u64_le(len(policy)) ‖ policy )
-// 4) live subgroups — ascending; only those whose latest exists=true
-for (child, slot) in subgroups, if slot.live:
-  h.update( child ‖ [parent if set] ‖ [restricted_byte] )
-// 5) capability plane — each map ascending
-for (group, caps) in default_caps: h.update( group ‖ u32_le(caps) )
-for ((group,member), caps) in member_caps: h.update( group ‖ member ‖ u32_le(caps) )
-for (group, admin) in group_admin: h.update( group ‖ admin )
- -

Stable role encoding

-

Roles hash to an explicit byte — fixed independently of any in-memory enum order, so the root survives refactors:

- - - - - - -
Rolerole_byte
Admin0
Member1
ReadOnly2
ReadOnlyTee3
-

Note the empty-group skip and the live-subgroup filter: materialized state must hash identically no matter how it was reached, so a phantom empty group or a deleted subgroup must never perturb the root.

-
- -
-

Where this leads

-

Chapters 3 and 4 are the core: a signed operation, a causally-ordered DAG, an order-independent fold to state, and a single scope root that makes convergence a hash comparison. Everything else is this machinery applied to a purpose — Chapter 5 shows where data operations come from (running a WASM application), and Chapter 6 shows how the governance planes encode membership and authority.

-
- -
-← Operations & the Causal DAG -Application Execution → -
- -
-
- - - - - diff --git a/architecture/protocol/receive-path.html b/architecture/protocol/receive-path.html deleted file mode 100644 index 3ae15a3f18..0000000000 --- a/architecture/protocol/receive-path.html +++ /dev/null @@ -1,79 +0,0 @@ - - - - -The Receive & Apply Path — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/The Receive & Apply Path
-

The Receive & Apply Path

-

Chapter 9 — verifying, ordering, and applying an operation from a peer.

- -
-

The right half of the operation's life

-

An operation arrives — from a gossip broadcast or a sync stream; the path is identical. The receiver must establish three things before the operation can change any state: that it is authentic, that its author was authorized, and that it is causally ready. Only then does it fold. Two of the steps can pause the operation in a buffer rather than reject it — that resilience is the interesting part.

-
-
-
- -
-

Step by step

-
1

Verify the signature.

Check the operation's signature against its author key. This is the trust boundary from Chapter 3: an operation that fails verification is dropped here and never reaches the fold. Everything downstream may assume authenticity.

-
2

Authorize at the author's cut.

Resolve the author's membership and writer rights at the operation's causal cut — the governance state they referenced — and check they were allowed to write it (Chapters 4 and 6). Because the verdict is computed at the cut, it is the same on every node regardless of receive order: forward-only authority.

-
3

Decrypt — or buffer for the key.

Decrypt the payload with the scope key. A node that is a member but hasn't yet received the key (the join race from Chapter 6) holds the operation in a key-pending buffer and drains it the moment the wrapped key arrives.

-
4

Insert into the DAG — or buffer for parents.

If all parents are applied, the operation is ready. If a parent hasn't arrived (gossip delivered a child before its parent), it waits in the pending buffer keyed by the missing parent — the same buffer-and-cascade from Chapter 3.

-
5

Fold and cascade.

Apply the operation into the projection (Chapter 4), advancing state and heads, then re-check both buffers: a just-applied operation may be the missing parent of a pending child, or the key that unlocks a queued operation. Cascade until nothing new is ready.

-
6

Run event handlers.

If the operation carried application events, dispatch their handlers and notify subscribed clients. This is a local side effect of applying — it never feeds back into consensus.

-
- -
-

Why buffering, not rejection

-

A best-effort gossip mesh delivers operations out of order and sometimes before their prerequisites. The receive path is built to tolerate that rather than fight it. A missing parent or an undelivered key is a "not yet," not a "no" — the operation is parked and revisited automatically. This is what lets a backfilled or reordered history snap into a correct state in a single cascade, and it is why convergence doesn't require any particular delivery order.

-
contrast

Reject only on authenticity or authority.

The two things a node will not wait on are a bad signature and an unauthorized author — those are permanent verdicts, identical on every node, so the operation is dropped outright. Everything else is a timing problem that resolves itself.

-
- -
-

Where this leads

-

Verified, authorized, ordered, applied: the operation has lived its full life on a receiving node, and if both peers hold the same set, their scope roots now match. But gossip is best-effort — a node that was offline never saw the broadcast at all. Chapter 10 closes that gap: detecting and repairing divergence.

-
- -
-← The Write Path -Sync & Convergence → -
- -
-
- - - - - diff --git a/architecture/protocol/storage.html b/architecture/protocol/storage.html deleted file mode 100644 index b5a3d2e45b..0000000000 --- a/architecture/protocol/storage.html +++ /dev/null @@ -1,114 +0,0 @@ - - - - -Appendix · Storage Schema — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/Appendix · Storage Schema
-

Appendix · Storage Schema

-

The on-disk format — column families, key byte-layouts, and value encodings.

- -
-

Scope

-

This appendix specifies the local persistence layer. A node stores everything in a key-value store (RocksDB) partitioned into column families. Two encodings are used for values: Identity (raw bytes, no framing) and Borsh (the same compact binary encoding used on the wire). Reimplementations are free to choose any backend — what matters for interop is the logical content, not the column layout; this appendix documents the reference layout because it makes the data model concrete.

-
- -
-

Column families

-

Some columns replicate (their content syncs between nodes); others are strictly node-local and never leave the machine.

- - - - - - - - - - - - - - -
ColumnScopeHolds
Metareplicatedper-context metadata (root hash, DAG heads, application)
Configreplicatedper-context revision counters
Identityreplicatedcontext member keypairs
Statereplicatedshared application state (the data plane)
Deltareplicatedcausal deltas (legacy per-context DAG)
UnifiedOpreplicatedthe unified op-store — one DAG per scope
Groupreplicatedall governance records (prefix-partitioned, below)
Blobs / Applicationreplicatedcontent-addressed binaries and WASM bundles
Alias / Genericreplicatedname→id mappings; uncategorized
PrivateStatenode-localper-node private state (never synced)
SortedIndexnode-localderived ordering index for sorted collections
ContextLocal, AbsorbBuffer, …node-localleave markers, straggler buffers, migration/blob breadcrumbs
-
- -
-

Key byte-layouts

-

Keys are fixed-size concatenations of 32-byte ids. The leading id is the partition prefix, so a range scan over it returns everything for one context or scope.

- - - - - - - -
KeyColumnLayoutBytes
ContextMetaMetacontext_id(32)32
ContextStateStatecontext_id(32) ‖ state_key(32)64
ContextPrivateStatePrivateStatecontext_id(32) ‖ state_key(32)64
ContextDagDeltaDeltacontext_id(32) ‖ delta_id(32)64
ScopeUnifiedOpUnifiedOpscope(32) ‖ op_id(32)64
-

The unified op store is the key one for the target model: keyed scope ‖ op_id, so all operations of a scope are a contiguous range, and the value is simply the Borsh-encoded Op from Chapter 3, stored verbatim (Identity codec).

-
- -
-

Key value structures

-

The two structures a reimplementation most needs. Both are Borsh.

-
-struct ContextMeta { // Meta column
-  application: [u8; 32],
-  root_hash: [u8; 32], // storage Merkle root = entity root
-  dag_heads: Vec<[u8; 32]>,
-  service_name: Option<str>,
-} -
-
-struct ContextDagDelta { // Delta column (legacy data DAG)
-  delta_id: [u8; 32], parents: Vec<[u8; 32]>,
-  actions: Vec<u8>, hlc: HybridTimestamp, applied: bool,
-  expected_root_hash: [u8; 32],
-  events: Option<Vec<u8>>, author_id: Option<PublicKey>,
-  governance_position_blob: Option<Vec<u8>>,
-  delta_signature: Option<[u8; 64]>,
-} -
-
- -
-

The Group column prefix scheme

-

All governance record types share one column, distinguished by a 1-byte prefix followed by the relevant ids. A representative selection:

- - - - - - - - - - - -
PrefixRecordKey layout
0x20GroupMeta0x20 ‖ group_id(32)
0x21GroupMember0x21 ‖ group_id(32) ‖ identity(32)
0x26MemberCapability0x26 ‖ group_id(32) ‖ identity(32)
0x30GroupOpLog0x30 ‖ group_id(32) ‖ sequence(8, big-endian)
0x34GroupParentRef0x34 ‖ child_id(32) → parent
0x36NamespaceIdentity0x36 ‖ namespace_id(32) → keypair
0x38NamespaceGovOp0x38 ‖ namespace_id(32) ‖ delta_id(32)
0x3AGroupKeyEntry0x3A ‖ group_id(32) ‖ key_id(32)
0x3CNonceWindow0x3C ‖ group_id(32) ‖ signer(32)
-

Note the GroupOpLog sequence is big-endian so keys sort in numeric order; id-based keys use the raw 32 bytes. The key_id in GroupKeyEntry is SHA-256(scope_key), matching the key_id a StateDelta advertises on the wire (Chapter 7).

-
- -
-

Back to the protocol

-

That is the on-disk shape of everything the ten chapters describe. Return to the Protocol Reference, or revisit Operations (whose Op is what the unified store holds) and Projection (whose root hash is the root_hash in ContextMeta).

-
- -
-← Sync & Convergence -Protocol Reference home → -
- -
-
- - - - diff --git a/architecture/protocol/sync.html b/architecture/protocol/sync.html deleted file mode 100644 index 0ba5fa9376..0000000000 --- a/architecture/protocol/sync.html +++ /dev/null @@ -1,190 +0,0 @@ - - - - -Sync & Convergence — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/Sync & Convergence
-

Sync & Convergence

-

Chapter 10 — detecting divergence and repairing it.

- -
-

Why sync exists

-

Gossip is best-effort. A node that was offline, partitioned, or simply not yet subscribed will miss broadcasts — so two nodes can hold different operation sets and, therefore, different scope roots. Sync is the reliable channel that closes that gap. Its whole job is built on one fact established back in Chapter 4: two nodes agree if and only if their scope roots are equal. So sync is just "compare roots; if they differ, find and transfer what's missing."

-
- -
-

What triggers it

-

A node starts a sync session for a scope when any of these fire:

-
    -
  • Periodically — a steady background tick, so nothing drifts indefinitely.
    • -
  • On join — a freshly joined member pulls the history it just gained access to.
    • -
  • On heartbeat mismatch — the periodic Heartbeat (Chapter 7) advertises a peer's heads and root; a root that doesn't match ours is a direct signal to reconcile.
    • -
  • After a data sync — a follow-up check that governance also agrees (below).
    • -
-
- -
-

The handshake and the verdict

-

A session opens a stream to a peer and the two exchange their heads and roots. From the comparison the node reaches one of three verdicts — and the verdict, not a guess, selects what happens next:

-
-
- - - - - - - scope roots equal? - H(entities ‖ acl ‖ gov) - - - - - yes - - Converged - nothing to do - - - - - no - - entity roots equal? - data plane only - - - - - yes - - Governance diverged - data agrees; pull governance ops - - - - - no - - Data diverged - locate & transfer missing state - - -
-

The two-step test matters: comparing the full scope root first catches divergence the data alone would hide (the hash-neutral case from Chapter 4 — an access-control or membership split with identical data). Only if the data plane also differs do we reach for the heavier state-transfer protocols.

-
- -
-

Choosing how to transfer

-

When state must move, the node picks the cheapest protocol that fits the gap:

-
-

Tree comparison

Walk the two state trees top-down, descending only into subtrees whose hashes differ, until the exact differing entities are found — then exchange just those. Cheap when the difference is small.

-

Level-wise

Compare the trees breadth-first, level by level. A better fit when differences are spread widely rather than concentrated.

-

Snapshot

Transfer the whole materialized scope. The fallback when a node is so far behind that incremental comparison isn't worth it — a new joiner, or one long offline.

-
-

Transferred entities are merged through the same conflict-free fold as everything else (Chapter 4), and the receiver re-checks its root afterward. The merge is authorized at the cut, so sync can never smuggle in a write the author wasn't entitled to make.

-
- -
-

A session, end to end

-
-
-
- -
-

Closing the governance gap

-

Data and governance live in different scopes and converge independently, so a successful data sync doesn't by itself guarantee the governance planes agree. After any data transfer completes, the node recomputes the scope root and, if it still differs from the peer's, pulls governance operations too — post-sync governance reconciliation.

-

The pull is idempotent: against an already-agreeing peer it returns nothing, so running it whenever roots still differ is safe and self-limiting. It is what ensures the whole scope — data and governance together — actually converges, not just the half that the state-transfer protocols touch.

-
- -
-

Specification

- -

The handshake

-

The initiator opens a stream and sends an Init carrying a DagHeadsRequest; the peer replies with:

-
-DagHeadsResponse {
-  dag_heads: Vec<[u8; 32]>,
-  root_hash: Hash, // storage Merkle (entity) root
-  scope_root: Option<Hash>, // None when the scope can't be folded (cold / non-group)
-} -
- -

The verdict

-

From the local and peer (scope_root, entity_root) pairs, the node computes one verdict. This is the exact decision:

-
-match (local_scope_root, peer_scope_root):
-  (Some l, Some p) if l == p            → Converged
-  (Some l, Some p) if entity_roots == → GovDiverged(l, p)
-  (Some _, Some _)              → DataDiverged
-  _ if entity_roots ==           → Converged  // asymmetric None ⇒ fall back to entity root
-  _                        → DataDiverged -
-
    -
  • Converged — nothing to do.
    • -
  • GovDiverged(local, peer) — entities agree, governance differs: pull governance operations.
    • -
  • DataDiverged — entities differ: run a state-transfer protocol.
    • -
- -

State-transfer protocols

-

When data diverges, the node selects by tree shape and how much differs: tree comparison (TreeNodeRequest walking only differing subtrees, request depth clamped to 16), level-wise (breadth-first when differences are widespread), or snapshot (a fresh or far-behind node bootstraps with full, compressed pages). Pushed entities pass an authorization gate — the host drops any leaf whose author is no longer a member, and per-operation signatures are still verified on apply — so sync can never introduce an unauthorized write.

- -

Post-sync governance reconciliation

-

After a data transfer completes, the node re-reads its (now possibly merged) root, recomputes scope_root, and if it still differs from the peer's it pulls governance. The pull is idempotent — it returns zero operations against an already-agreeing peer — so pulling whenever the root still differs is safe and self-limiting.

- -

Triggers & timing (defaults)

- - - - - - -
SettingDefaultMeaning
sync frequency10 speriodic check interval
min interval5 sminimum gap between syncs of one scope
session timeout30 sper-step / session budget
wedge grace~60 swatchdog that unsticks a stalled session
-

Beyond the periodic tick, a session also starts on a fresh join, on a heartbeat whose root doesn't match, and as the governance follow-up above.

-
- -
-

The operation's life, complete

-

That closes the loop the landing page opened. An operation is born in an execution (Chapter 8), broadcast over gossip (Chapter 7), received and folded by peers (Chapter 9) — and anything that channel missed is found and repaired here by comparing one 32-byte hash and transferring exactly what differs. No consensus, no quorum, no agreed order: just a signed operation DAG that folds to a state and a root, with divergence detected, not prevented.

-

From here the reference branches into appendices — the storage schema, the crate map, metrics, and migrations — but the protocol itself is the ten chapters you have just read.

-
- -
-← The Receive & Apply Path -Protocol Reference home → -
- -
-
- - - - - diff --git a/architecture/protocol/write-path.html b/architecture/protocol/write-path.html deleted file mode 100644 index 2ba0a3a3e0..0000000000 --- a/architecture/protocol/write-path.html +++ /dev/null @@ -1,82 +0,0 @@ - - - - -The Write Path — Calimero Core Architecture - - - - -
-
-
Home/Protocol Reference/The Write Path
-

The Write Path

-

Chapter 8 — from a request to a broadcast operation, end to end.

- -
-

The left half of the operation's life

-

The landing page promised to follow one operation from birth to convergence. This chapter is its birth: a request arrives, an application runs, and a signed operation is persisted and broadcast. Every step here was introduced earlier — this is where they connect into a single flow.

-
-
-
- -
-

Step by step

-
1

Request and lock.

A request (HTTP/RPC) names a context and a method. The node takes an execution lock on that context so concurrent writes don't interleave. Read-only calls take a shared lock and skip everything below — they produce no operation.

-
2

Execute.

The WASM application runs the method against current state (Chapter 5). It produces an outcome: a new root hash and an artifact of committed writes, plus events and any queued cross-calls. If it traps, the lock releases and nothing is written.

-
3

Build the operation.

The committed writes become data-plane operations in the context scope. Each takes the scope's current heads as parents, a fresh hybrid timestamp, the executor's member identity as author, and a content-addressed id (Chapter 3).

-
4

Sign.

The author signs the id with their member key. The signature binds authorship to content so a peer can verify it before trusting the operation — and so the operation can't be relabelled in flight.

-
5

Persist atomically.

In one atomic write the node stores the operation, the updated context metadata (new root hash and heads), and the materialized state. Atomicity matters: a node must never advance its heads without also holding the operation that produced them, or a crash could strand the two out of step.

-
6

Broadcast.

The signed operation is published to the context/<id> gossip topic, encrypted under the scope key. Broadcast is fire-and-forget and runs off the request's critical path — the writer has already durably committed; gossip is how everyone else finds out.

-
- -
-

What gets committed before the broadcast

-

The ordering is deliberate: persist, then broadcast. By the time a single byte leaves the node, the operation is already durable locally and the author's own state already reflects it. If the network is down, or no peer is listening, nothing is lost — the operation sits in the local DAG and will be served to peers later through the sync channel (Chapter 10). Convergence never depends on a broadcast being received; it only depends on the operation eventually being available.

-
- -
-

Governance writes take the same path

-

A membership change, a capability grant, or an admin action is built and broadcast exactly like a data write — the only differences are that the payload is a governance variant (Chapter 6) and the topic is the governance scope's (group/ or ns/) rather than the context's. One write path serves all four planes, because they are all just operations.

-
- -
-

Where this leads

-

The operation is now in flight. Chapter 9 is the right half of its life: what a receiving node does to verify, decrypt, order, and apply it.

-
- -
-← Networking & the Wire Protocol -The Receive & Apply Path → -
- -
-
- - - - - diff --git a/architecture/release.html b/architecture/release.html deleted file mode 100644 index c26c54e446..0000000000 --- a/architecture/release.html +++ /dev/null @@ -1,198 +0,0 @@ - - - - -Release Process — Calimero Core Architecture - - - -
-
-
Home/Release Process
-

Release Process

-

Versioning and release workflow

- -
-

Where the version lives

-

The release version has a single source of truth in the root Cargo.toml:

-
    -
  • [workspace.metadata.workspaces].version — authoritative version. The release workflow and tooling read it (e.g. cargo metadata | jq -r '.metadata.workspaces.version').
    • -
  • [workspace.package].version stays "0.0.0" as a placeholder; cargo-workspaces uses the metadata version when publishing.
    • -
-
Cargo.toml
Root workspace manifest — bump [workspace.metadata.workspaces].version only for a release.
-
- -
-

Cutting a release

-

Bump the version under [workspace.metadata.workspaces], merge (e.g. release PR), then the CI workflow:

-
    -
  • Reads the version from cargo metadata (no sed or in-repo string replacement).
    • -
  • Builds binaries, publishes crates (cargo ws publish), and creates the GitHub release with that version.
    • -
-
- -
-

Binaries, libraries, and types

-
    -
  • Binaries (merod, meroctl) — displayed version comes from build-time env vars in each crate’s build.rs (MEROD_VERSION, MEROCTL_VERSION), derived from workspace metadata plus git describe/commit.
    • -
  • Libraries that need a release string (e.g. bundle minRuntimeVersion) expose CALIMERO_RELEASE_VERSION from build.rs, sourced from [workspace.metadata.workspaces].version.
    • -
  • The Version type and related protocol types live in calimero_primitives::version.
    • -
-
- -
-Example metadata bump -
-
[workspace.metadata.workspaces]
-version = "0.11.0-rc.7" # current release candidate; was 0.11.0-rc.6
-
-
- -
-

CI workflow

-

Pushes to master run the main CI pipeline. The release workflow reads the version from root Cargo.toml workspace metadata (cargo metadata), builds merod and meroctl binaries for Linux and macOS, publishes workspace crates with cargo-workspaces, and creates a GitHub release attaching those artifacts.

-

Clippy gate: The CI checks step runs clippy across the full workspace with warnings treated as errors:

-
cargo clippy --workspace --all-targets --features calimero-storage/testing -- -D warnings
-

This covers all targets (binaries, libraries, tests, examples) and includes the feature-gated testing harness. Any clippy or rustc warning in any target fails CI. Run this command locally before pushing to avoid failures.

-
- -
-

Docker images

-

The merod image is published to ghcr.io/calimero-network/merod. Images are tagged with the release version; edge tracks the latest master build.

-
- -
-

Changelog

-

Release notes are driven by CHANGELOG.md at the repository root. It follows the Keep a Changelog format and should be updated in the same PR that bumps the workspace version.

-

The [Unreleased] compare link base tracks the current tip: 0.11.0-rc.7...HEAD. A new [0.11.0-rc.7] diff entry (0.11.0-rc.6...0.11.0-rc.7) was added for this release candidate, along with a [#2855] pull-request reference.

-
- -

ABI contract check workflow

-

A dedicated workflow at .github/workflows/abi-contract.yml automates cross-repo ABI compatibility checks in CI.

- -

Triggers

-
    -
  • Push to master.
    • -
  • Pull requests that touch ABI-related paths (path filters defined in the workflow).
    • -
- -

What the job does

-
    -
  1. Checks out the core repo. The checkout ref falls back from pull_request.head.sha to github.sha so both PR and push events resolve correctly.
    2. -
  2. Checks out mero-devtools-js at the ref controlled by the DEVTOOLS_REF variable (defaults to main).
    3. -
  3. Builds abi-codegen.
    4. -
  4. Runs the contract check script against the checked-out devtools.
    5. -
- -

Status and pinning

-

The job is non-required and is expected to show red until the matching schema-sync lands in mero-devtools-js. Once the downstream schema update is merged, bump DEVTOOLS_REF from main to a release tag to pin enforcement and prevent future regressions from an uncontrolled main tip.

-
# Example: pin to a specific devtools release
-DEVTOOLS_REF: v1.4.0
- -

Purpose

-

Without this workflow, ABI regressions were only discovered after mero-devtools-js vendored new generated types. The check now runs on every relevant PR so breakage is caught before merging rather than after the downstream tool picks up the change.

-
-

E2E op-store completeness reporting (C3 Stage 0)

-

A CI step named Report unified op-store completeness (C3 Stage 0, observe-only) runs inside the end-to-end workflow after Docker logs are collected. It makes known structural incompleteness in the op-store visible per scenario rather than silent.

- -

What the step does

-
    -
  1. Greps docker-logs/ for two marker strings: -
      -
    • op_store_incomplete — signals gaps in op-store coverage.
      • -
    • op_store_gate_unavailable — signals that the gate-load mechanism itself could not be reached.
      • -
    -
    2. -
  2. Classifies the result into one of three outcomes: -
      -
    • Gaps found: emits a ::notice:: annotation with the count of incomplete markers.
      • -
    • Gate-load failures only (no gap markers): emits an UNVERIFIED notice — the step explicitly does not claim the store is clean.
      • -
    • No markers of either kind: prints a confirmed-clean message.
      • -
    -
    3. -
- -

Observe-only behaviour

-

The step never fails the job. It is purely informational at C3 Stage 0, giving each subsequent stage a visible baseline to track. The intent is that the marker count falls to zero across stages, at which point the step becomes a hard gate at C3 Stage 4.

- -

Why it was added

-

Previously, op-store incompleteness was silent in CI and only surfaced after downstream tooling consumed the data. Surfacing the markers per scenario on every relevant run allows regressions and progress to be observed incrementally without blocking merges during the remediation period.

- -
# Markers the step looks for in docker-logs/
-op_store_incomplete
-op_store_gate_unavailable
-
-

Live paired SDK e2e CI workflow

-

The .github/workflows/sdk-e2e.yml workflow provides a behavioral gate on every relevant PR: it builds merod from the PR branch, boots a real node, and drives the mero-js e2e suite against it over actual HTTP. Static fixture or schema tests cannot catch wiring errors introduced when a renamed or removed parameter causes the SDK to send the old request shape and the node returns a 4xx; this workflow will.

- -

What the workflow does

-
    -
  1. Builds merod from the PR branch.
    2. -
  2. Boots a single node with embedded auth on a configurable port.
    3. -
  3. Runs the mero-js e2e suite against that node via the NODE_BASE_URL environment variable.
    4. -
  4. Runs the endpoint-coverage check.
    5. -
  5. Stops merod via MEROD_PID. The stop step guards against accidentally signaling the whole process group when the variable is unset — if MEROD_PID is empty the signal is skipped rather than sent to PID 0 or the group leader.
    6. -
- -

Resolving the paired mero-js ref

-

The companion script resolve-paired-ref.sh determines which mero-js branch to test against using a three-level priority:

-
    -
  1. Explicit override: an sdk-ref: <branch> line anywhere in the PR body.
    2. -
  2. Same-named branch: a branch on the mero-js repo with the same name as the core PR branch.
    3. -
  3. Default: master.
    4. -
-

The resolved ref is validated before use. It is rejected if it:

-
    -
  • contains characters outside the safe git-ref character set,
    • -
  • contains .. (path traversal),
    • -
  • starts with - (option injection), or
    • -
  • starts with / (absolute path).
    • -
- -

Pointing a PR at a paired SDK branch

-

If your core PR requires a matching change in mero-js, add a line to the PR description:

-
sdk-ref: my-feature-branch
-

The workflow will check out that branch of mero-js instead of master. Without this line the script falls back to a same-named branch if one exists, then to master.

- -

Why this catches what static tests miss

-

A renamed or removed parameter in a handler causes the SDK to send the old request shape; the node returns a 4xx and the workflow fails on the core developer's PR — before the downstream SDK picks up the change. Schema and ABI checks validate structure; the SDK e2e workflow validates behaviour across a live HTTP boundary.

-
-

Route coverage gate is now method-aware

-

The check-endpoint-coverage.sh script previously matched coverage by path alone. A recorded hit on any HTTP verb for a path would satisfy coverage for every verb on that same path. Coverage is now tracked and matched as METHOD /path pairs, so a hit on one verb never satisfies coverage for a different verb on the same path.

- -

What changed

-
    -
  • Each manifest entry is expected in the form METHOD /path, for example GET /admin-api/blobs/:blob_id.
    • -
  • Each recorded hit must also be a METHOD /path string. Both the method and the path must match before a route is marked covered.
    • -
  • Trailing slashes on recorded hits are normalized before comparison.
    • -
- -

Why the change was made

-

A real bug slipped through the old check: the SDK's getBlob was sending DELETE /admin-api/blobs/:id instead of GET /admin-api/blobs/:id. Because path-only coverage saw the path exercised via DELETE, it reported the route as covered and the CI gate stayed green. GET was never actually exercised. The method-aware check would have failed on that PR.

- -

Impact on the recorded-hits file

-

If your e2e suite or any tooling that feeds the recorded-hits file emits bare paths, those entries must be updated to include the method. Bare-path entries will no longer match any manifest route and will cause previously-green coverage checks to fail.

- -
    -
  • Old format (no longer accepted): /admin-api/blobs/:blob_id
    • -
  • Required format: GET /admin-api/blobs/:blob_id
    • -
- -

Manifest format

-

The endpoint manifest must list one route per line as METHOD /path. Routes that share a path but differ in method are separate entries and must each be hit independently to achieve full coverage:

-
GET  /admin-api/blobs/:blob_id
-POST /admin-api/blobs
-DELETE /admin-api/blobs/:blob_id
- -

What to check if coverage unexpectedly fails after this change

-
    -
  1. Confirm that the recorded-hits file produced by the e2e run contains METHOD /path strings, not bare paths.
    2. -
  2. Check for trailing-slash mismatches between the manifest and any SDK calls — the script normalizes recorded hits but not manifest entries.
    3. -
  3. Verify that every distinct METHOD /path pair in the manifest is actually exercised by at least one test; paths shared across verbs each need their own hit.
    4. -
-
-
-
- - - diff --git a/architecture/seq-diagram.js b/architecture/seq-diagram.js deleted file mode 100644 index 95bfb4c4b0..0000000000 --- a/architecture/seq-diagram.js +++ /dev/null @@ -1,133 +0,0 @@ -/* Calimero Core — shared animated sequence-diagram engine. - * - * Extracted from sequence-diagrams.html so any page can render the same - * step-cascade animated SVGs from data instead of hand-placed coordinates. - * - * Usage on a page: - *
- *
- * - * - * - * A `lane` = { label, color }. - * A `step` = { from, to, label, dashed?, color? } (from===to draws a self-call) - * | { type:'divider', label }. - * - * The renderer wraps each step in ; the - * cascade animation lives in styles.css and fires whenever the container has - * the `animating` class. Define SEQ_DIAGRAMS either before or after this script. - */ -(function () { - 'use strict'; - - var NS = 'http://www.w3.org/2000/svg'; - - function S(tag, a, txt) { - var e = document.createElementNS(NS, tag); - if (a) Object.keys(a).forEach(function (k) { e.setAttribute(k, a[k]); }); - if (txt != null) e.textContent = txt; - return e; - } - - function render(id, data) { - var box = document.getElementById(id); - if (!box) return; - box.textContent = ''; - - var lanes = data.lanes, steps = data.steps; - var n = lanes.length, m = steps.length; - - var L = 48, R = 18; - var W = Math.max(960, L + n * 150 + R); - var SS = 90, SH = 52; - var H = SS + m * SH + 28; - var lw = (W - L - R) / n; - var cx = lanes.map(function (_, i) { return L + lw * i + lw / 2; }); - - var svg = S('svg', { viewBox: '0 0 ' + W + ' ' + H, role: 'img', 'aria-label': data.title + ' sequence diagram' }); - svg.appendChild(S('title', null, data.title)); - svg.appendChild(S('rect', { x: 0, y: 0, width: W, height: H, fill: '#0f0f16', rx: 10 })); - - lanes.forEach(function (lane, i) { - var x = cx[i], hw = Math.min(lw - 10, 134); - svg.appendChild(S('rect', { x: x - hw / 2, y: 10, width: hw, height: 32, rx: 6, fill: lane.color + '18', stroke: lane.color + '50', 'stroke-width': 1 })); - svg.appendChild(S('text', { x: x, y: 31, 'text-anchor': 'middle', fill: lane.color, 'font-family': "'JetBrains Mono', monospace", 'font-size': lane.label.length > 11 ? '9.5' : '10.5', 'font-weight': '600' }, lane.label)); - svg.appendChild(S('line', { x1: x, y1: 48, x2: x, y2: H - 14, stroke: '#252538', 'stroke-width': 1, 'stroke-dasharray': '3,4' })); - }); - - var stepNum = 0; - steps.forEach(function (step, i) { - var y = SS + i * SH; - var g = S('g', { class: 'seq-step', style: '--step-i:' + i }); - - if (step.type === 'divider') { - g.appendChild(S('line', { x1: L, y1: y, x2: W - R, y2: y, stroke: '#35354a', 'stroke-width': 1, 'stroke-dasharray': '6,4' })); - var dlw = step.label.length * 6.2 + 24; - g.appendChild(S('rect', { x: W / 2 - dlw / 2, y: y - 11, width: dlw, height: 20, fill: '#0f0f16', rx: 4 })); - g.appendChild(S('text', { x: W / 2, y: y + 4, 'text-anchor': 'middle', fill: '#6b6b80', 'font-family': "'DM Sans', sans-serif", 'font-size': '10.5', 'font-style': 'italic' }, step.label)); - svg.appendChild(g); - return; - } - - stepNum++; - g.appendChild(S('text', { x: 12, y: y + 4, 'text-anchor': 'start', fill: '#4a4a5a', 'font-family': "'JetBrains Mono', monospace", 'font-size': '9.5', 'font-weight': '700' }, String(stepNum))); - - var fx = cx[step.from], tx = cx[step.to]; - var col = step.color || lanes[step.to].color; - - if (step.from === step.to) { - var rightSide = step.from < n - 1; - var dir = rightSide ? 1 : -1; - var lp = 36, lh = 18; - var sx = fx + dir * 3; - - g.appendChild(S('path', { - d: 'M ' + sx + ',' + (y - lh / 2) + ' h ' + (dir * lp) + ' v ' + lh + ' h ' + (dir * -lp), - fill: 'none', stroke: col, 'stroke-width': 1.3 - })); - - var ax = sx, ay = y + lh / 2; - g.appendChild(S('polygon', { points: (ax + dir * 5) + ',' + (ay - 3.5) + ' ' + ax + ',' + ay + ' ' + (ax + dir * 5) + ',' + (ay + 3.5), fill: col })); - - var lx = fx + dir * (lp + 10); - g.appendChild(S('text', { x: lx, y: y + 3, 'text-anchor': rightSide ? 'start' : 'end', fill: col, 'font-family': "'JetBrains Mono', monospace", 'font-size': '9', 'font-weight': '500' }, step.label)); - } else { - var d = tx > fx ? 1 : -1; - var x1 = fx + d * 5, x2 = tx - d * 5; - - g.appendChild(S('circle', { cx: fx, cy: y, r: 2.5, fill: col + '70' })); - g.appendChild(S('circle', { cx: tx, cy: y, r: 3, fill: col })); - - var dashVal = step.dashed ? '5,3' : 'none'; - var opVal = step.dashed ? '.55' : '1'; - g.appendChild(S('line', { x1: x1, y1: y, x2: x2, y2: y, stroke: col, 'stroke-width': 1.5, 'stroke-dasharray': dashVal, 'stroke-opacity': opVal })); - g.appendChild(S('polygon', { points: x2 + ',' + y + ' ' + (x2 - d * 7) + ',' + (y - 3.5) + ' ' + (x2 - d * 7) + ',' + (y + 3.5), fill: col, 'fill-opacity': opVal })); - - var mx = (x1 + x2) / 2; - g.appendChild(S('text', { x: mx, y: y - 8, 'text-anchor': 'middle', fill: '#c4c4cc', 'font-family': "'JetBrains Mono', monospace", 'font-size': '9.5' }, step.label)); - } - - svg.appendChild(g); - }); - - box.appendChild(svg); - } - - function replay(id) { - var el = document.getElementById(id); - if (!el) return; - el.classList.remove('animating'); - void el.offsetWidth; // force reflow so the animation re-runs - el.classList.add('animating'); - } - - window.replay = replay; - window.renderSeq = render; - - document.addEventListener('DOMContentLoaded', function () { - (window.SEQ_DIAGRAMS || []).forEach(function (d) { render(d.id, d); }); - }); -})(); diff --git a/architecture/sequence-diagrams.html b/architecture/sequence-diagrams.html deleted file mode 100644 index 1dec1fd293..0000000000 --- a/architecture/sequence-diagrams.html +++ /dev/null @@ -1,1090 +0,0 @@ - - - - - - Sequence Diagrams — Calimero Core Architecture - - - - -
-
-
- Home/Sequence Diagrams -
- -

Sequence Diagrams

-

Step-by-step animated flows for 27 key operations

- -
-

Core Operations

-
- - - - - - -
-

Sync & Replication

-
- - - - - - -
-

Infrastructure

-
- - -
-

Group Management

-
- - - - - -
-

Namespace Operations

-
- - - - -
-

Application Flows

-
- - - - -
-
- - -
-
-

Admin creates a new context with a generated identity, registers it in the group store, and publishes the operation across the network.

-
-
-
-
- - -
-
-

Client invokes a WASM method via JSON-RPC. The runtime executes it with host functions, commits state deltas, and broadcasts them to peers for convergence.

-
-
-
-
- - -
-
-

An admin creates a group invitation and delivers it out-of-band. The joiner uses the invitation to join, and their membership is propagated and verified across the network.

-
-
-
-
- - -
-
-

Two nodes exchange state hashes and DAG heads to detect divergence, then select and execute the appropriate sync protocol (Hash, Level, Snapshot, or Delta).

-
-
-
-
- - -
-
-

Client uploads a blob to the server which stores it locally and advertises via Kademlia DHT. Remote nodes discover providers and stream the blob on demand.

-
-
-
-
- - -
-
-

Admin triggers a group-level application upgrade. The new target is propagated to peers and applied to each context based on its upgrade policy (Automatic, Coordinated, or LazyOnAccess).

-
-
-
-
- - -
-
-

A group member joins a context within their group. Verifies group membership before generating a context identity and subscribing to the context topic.

-
-
-
-
- - -
-
-

HTTP authentication: how the AuthGuardLayer verifies JWT tokens from headers or query params and injects the authenticated identity into the request.

-
-
-
-
- - -
-
-

How an incoming StateDelta gossip message flows through the node: buffering during sync, key exchange, decryption, DAG ordering, application, and event emission.

-
-
-
-
- - -
-
-

A complete sync session: SyncManager opens a stream, shares keys, optionally shares blobs, then runs the selected DAG sync protocol.

-
-
-
-
- - -
-
-

How a SignedGroupOpV1 gossip message is received, verified, and applied through the governance DAG with pending queue handling.

-
-
-
-
- - -
-
-

When a GroupStateHeartbeat reveals missing DAG heads, the node opens a stream and requests the missing governance ops.

-
-
-
-
- - -
-
-

Current fleet-admission flow (supersedes the legacy specialized-node invite handshake). A TEE node broadcasts TeeAttestationAnnounce on ns/<hex> and re-announces for up to ~30s (MAX_ADMISSION_WAIT). A verifier runs verify_attestation, then namespace-scoped admit_tee_node (policy check + quote-hash replay guard), publishes MemberJoinedViaTeeAttestation (a ReadOnlyTee row) and a RootOp::KeyDelivery; the node applies the key and is admitted. On disable, a TeeMemberRemoved op triggers self_purge on the evicted node. See TEE Fleet HA.

-
-
-
-
- - -
-
-

When a node subscribes to a group topic, three operations fire in parallel: sync group state, broadcast aliases, and broadcast local state.

-
-
-
-
- - -
-
-

Admin creates a child group under an existing parent. The SubgroupCreated governance op is published on the parent's gossip topic so all peers learn about the hierarchy.

-
-
-
-
- - -
-
-

A ReadOnly member is added to a group. They can join contexts and read state, but any state mutations are discarded locally and rejected by remote nodes.

-
-
-
-
- - -
-
-

Admin grants capability bits to a group member via a governance operation, enabling them to perform specific actions like creating contexts or inviting members.

-
-
-
-
- - -
-
-

When a member is removed from a parent group, their ContextIdentity entries are cascaded through all descendant subgroup contexts. Child groups where they have direct membership are skipped.

-
-
-
-
- - -
-
-

A parent admin tries to join a context in a child subgroup where they lack direct membership. The platform enforces that context access requires membership in the owning group.

-
-
-
-
- - -
-
-

Developer generates a signing key, builds the WASM app, signs the manifest with JCS + SHA-256 + Ed25519, packages the .mpk bundle, and installs it on a node.

-
-
-
-
- - -
-
-

Context update with state migration: the new WASM module runs the migration function against the old state, produces new state bytes, and the context switches to the new application version.

-
-
-
-
- - -
-
-

A WASM method in Context A queues a cross-context call to Context B. After Context A commits, the node executes the target method on Context B using an owned member identity.

-
-
-
-
- - -
-
-

Admin creates a namespace (root group). The node generates a namespace identity keypair, creates group metadata, generates a group encryption key, subscribes to the namespace gossipsub topic, and publishes a RootOp::GroupCreated to the namespace governance DAG.

-
- - Admin - ContextMgr - GroupStore - Network - - - - - - POST /namespaces - - generate namespace identity keypair - - create group metadata + encryption key - - subscribe ns/<hex> topic - - sign & apply RootOp::GroupCreated - - gossip NamespaceGovernanceDelta - - { namespace_id, identity_pk } - -
-
-
- - -
-
-

A joiner receives an invitation out-of-band, generates their namespace identity, subscribes to the namespace topic, requests join via P2P stream. The peer responds with a wrapped group key, governance snapshot, and context IDs. The joiner unwraps the key, applies the snapshot, and announces MemberJoined.

-
- - Joiner - Peer (existing member) - Namespace DAG - - - - - gen namespace identity - - subscribe namespace topic + wait for mesh - - P2P stream: NamespaceJoinRequest(invitation, joiner_pk) - - ECDH wrap group key for joiner_pk - - { key_envelope, gov_ops_snapshot, context_ids } - - ECDH unwrap group key - - apply governance snapshot - - publish RootOp::MemberJoined - - auto-join group contexts - - joined ✓ - -
-
-
- - -
-
-

When a new member joins a namespace and publishes MemberJoined, existing members automatically detect this on the namespace DAG and publish a KeyDelivery op with the ECDH-wrapped group encryption key. The joiner unwraps the key and can then decrypt GroupOp payloads.

-
- - Joiner - Existing Member - Namespace DAG - - - - - RootOp::MemberJoined { member: joiner_pk } - - detect MemberJoined - - ECDH(my_sk, joiner_pk) wrap key - - RootOp::KeyDelivery { envelope } - - receive KeyDelivery via gossip - - ECDH(my_sk, sender_pk) unwrap - - can decrypt GroupOps ✓ - -
-
-
- - -
-
-

Namespace governance sync via heartbeat. Every 30s, nodes publish NamespaceStateHeartbeat with their DAG heads. When a peer detects missing ops, it uses gossipsub to push ops the peer needs, and opens a P2P backfill stream to request ops it is missing.

-
- - Node A - Node B - - - - NamespaceStateHeartbeat { dag_heads: [h1, h2] } - - compare: has h1, missing h2 - - P2P stream: NamespaceBackfillRequest { delta_ids: [h2] } - - scan namespace gov op log - - NamespaceBackfillResponse { deltas: [SignedNamespaceOp] } - - apply_signed_namespace_op - - converged ✓ - -
-
-
- - -
-
-

Two nodes concurrently edit the same document using the RGA CRDT. Each character insert generates a CharId with HLC timestamp. Both edits sync and merge deterministically.

-
-
-
-
- -
-
- - - - - - diff --git a/architecture/storage-schema.html b/architecture/storage-schema.html deleted file mode 100644 index 291d92b780..0000000000 --- a/architecture/storage-schema.html +++ /dev/null @@ -1,724 +0,0 @@ - - - - -Storage Schema — Calimero Core Architecture - - - - -
-
-
Home/Storage Schema
- -

Storage Schema

-

Every key/value type across all 11 columns with byte layouts

- -
-
12
column families
-
26
group prefixes
-
Borsh
default codec
-
RocksDB
storage engine
-
- - -
-

Column Enum

-

All persistent data is partitioned into 11 column families. Each variant maps to a dedicated RocksDB column family with independent compaction and bloom filters.

- -
-pub enum Column {
-    Meta,          // schema version, node id
-    Config,        // node configuration
-    Identity,      // keypairs per context
-    State,         // shared context state KV
-    PrivateState,  // per-node private KV (not synced)
-    Delta,         // causal deltas (DAG)
-    Blobs,         // content-addressed binary data
-    Application,   // WASM binaries + metadata
-    Alias,         // human-readable name → id
-    Generic,       // uncategorized data
-    Group,         // governance: 20 prefix-partitioned sub-namespaces
-    UnifiedOp,     // unified causal-log op-store: scope‖op_id → borsh(calimero_op::Op)
-} -
-
- - -
-

Context Column Types

-

Key/value types for the columns that store per-context data: metadata, configuration, shared state, private state, identity, deltas, and unified ops.

- -

Meta Column

- - - - - - - - - - - -
Key StructKey LayoutValue TypeCodecDescription
ContextMetacontext_id (32)ContextMetaValueBorshContext metadata including application info and DAG state
- -
-struct ContextMetaValue {
-    application: ApplicationMeta,  // app_id + version + services
-    root_hash: Hash,          // Merkle root of context state
-    dag_heads: Vec<[u8; 32]>,   // current DAG head delta IDs
-} -
- -

Config Column

- - - - - - - - - - - -
Key StructKey LayoutValue TypeCodecDescription
ContextConfigcontext_id (32)ContextConfigValueBorshPer-context revision counters
- -
-struct ContextConfigValue {
-    application_revision: u64,  // incremented on app upgrades
-    members_revision: u64,     // incremented on membership changes
-} -
- -

Identity Column

- - - - - - - - - - - -
Key StructKey LayoutValue TypeCodecDescription
ContextIdentitycontext_id (32)ContextIdentityValueBorshKeypair for this node's identity within a context
- -
-struct ContextIdentityValue {
-    private_key: Option<[u8; 32]>,  // Ed25519 private key
-    sender_key: Option<[u8; 32]>,   // sender public key
-} -
- -

State & PrivateState Columns

- - - - - - - - - - - - - - - - - - -
Key StructKey LayoutValue TypeCodecDescription
ContextStatecontext_id (32)+app_key (var)SliceIdentityShared context state — raw bytes, no serialization overhead
ContextPrivateStatecontext_id (32)+app_key (var)SliceIdentityPer-node private state — NOT synced to other nodes
- -

Delta Column

- - - - - - - - - - - -
Key StructKey LayoutValue TypeCodecDescription
ContextDagDeltacontext_id (32)+delta_id (32)ContextDagDeltaValueBorshCausal delta with parents, actions, and verification data
- -

UnifiedOp Column

-

Holds one calimero_op::Op per row, written by the context layer during the dual-write transition. This column is kept separate from Column::Delta to prevent key collisions; the two are intended to merge once legacy delta rows retire in a later cutover slice.

- - - - - - - - - - - -
Key StructKey LayoutValue TypeCodecDescription
ScopeUnifiedOpscope (32)+op_id (32)Slice (raw bytes)IdentityBorsh-serialized calimero_op::Op; calimero-store treats the value as opaque — callers must borsh-serialize/deserialize the Op themselves
- -
-// key/context.rs — re-exported from crate::key
-pub struct ScopeUnifiedOp {
-    scope: [u8; 32],  // typically a context_id
-    op_id: [u8; 32],
-}  // Size = U64, COLUMN = Column::UnifiedOp
-
-// types/context.rs — re-exported from crate::types
-// Value type: Identity codec (raw bytes); calimero_store has no dep on calimero_op -
- -
-struct ContextDagDeltaValue {
-    delta_id: [u8; 32],
-    parents: Vec<[u8; 32]>,        // causal parent IDs
-    actions: Vec<Action>,         // state mutations
-    hlc: HybridTimestamp,         // hybrid logical clock
-    applied: bool,               // has been applied to state
-    expected_root_hash: Hash,    // post-apply verification
-    events: Vec<Event>,          // emitted events
-} -
-
- - -
-

Group Column Detail

-

The Group column uses a single-byte prefix to partition logical sub-namespaces within one RocksDB column family. Each prefix isolates a governance concern — membership, capabilities, operations log, upgrades, aliases, and namespace identity.

- -

Membership & Metadata (0x20–0x23)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x20GroupMeta0x20+group_id (32)GroupMetaValueGroup metadata (app_key, target_app, upgrade_policy, admin, migration)
0x21GroupMember0x21+group_id (32)+identity (32)GroupMemberRoleMember role in group
0x22GroupContextIndex0x22+group_id (32)+context_id (32)()Context belongs to group (presence key)
0x23ContextGroupRef0x23+context_id (32)[u8; 32]Reverse index: context → group_id
- -
- GroupMetaValue definition -
-
-struct GroupMetaValue {
-    app_key: [u8; 32],              // application identity
-    target_application_id: [u8; 32], // target WASM application
-    upgrade_policy: UpgradePolicy,  // auto / manual / frozen
-    created_at: u64,               // unix timestamp
-    admin_identity: [u8; 32],       // group administrator
-    migration: Option<String>,       // migration method name
-} -
-
-
- -

Upgrades & Signing (0x24–0x25)

- - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x24GroupUpgradeKey0x24+group_id (32)GroupUpgradeValueActive upgrade state
0x25GroupSigningKey0x25+group_id (32)+key_id (32)GroupSigningKeyValueEd25519 signing key for group
- -
- GroupUpgradeValue & GroupUpgradeStatus definitions -
-
-struct GroupUpgradeValue {
-    from_version: u64,
-    to_version: u64,
-    migration: Option<String>,   // migration method name
-    initiated_at: u64,         // unix timestamp
-    initiated_by: [u8; 32],     // admin identity
-    status: GroupUpgradeStatus,
-} -
-
-enum GroupUpgradeStatus {
-    InProgress {
-        total: u64,     // total contexts to migrate
-        completed: u64, // successfully migrated
-        failed: u64,    // failed migrations
-    },
-    Completed {
-        completed_at: u64,  // unix timestamp
-    },
-} -
-
-struct GroupSigningKeyValue {
-    private_key: [u8; 32],  // Ed25519 private key bytes
-} -
-
-
- -

Capabilities & Defaults (0x26–0x29)

- - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x26GroupMemberCapability0x26+group_id (32)+identity (32)GroupMemberCapabilityValuePer-member capability bitmask
0x29GroupDefaultCaps0x29+group_id (32)GroupDefaultCapsValueDefault capabilities for new members
- -
- Capability value definitions -
-
-struct GroupMemberCapabilityValue {
-    capabilities: u32,  // bitmask: bit 0 = manage_members, bit 1 = manage_contexts, ...
-} -
-
-struct GroupDefaultCapsValue {
-    capabilities: u32,  // default bitmask for new members
-} -
-
-
- -

Migration & Nonce (0x2B–0x2C)

- - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x2BGroupContextLastMigration0x2B+group_id (32)+context_id (32)GroupContextLastMigrationValueLast migration method applied to this context
0x2CGroupLocalGovNonce0x2C+group_id (32)+signer (32)u64Per-signer monotonic nonce for replay protection
- -
- GroupContextLastMigrationValue definition -
-
-struct GroupContextLastMigrationValue {
-    method: String,  // WASM method name used for migration
-} -
-
-
- -

Metadata records (0x2D–0x2F)

-

Each entry holds a MetadataRecord { name: Option<String>, data: BTreeMap<String,String>, updated_at: u64, updated_by: PublicKey }. The data map is opaque to core (stored/replicated verbatim, never interpreted). These rows are replicated governance state but — like the former alias rows — are deliberately excluded from compute_group_state_hash.

- - - - - - - - - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x2DGroupMemberMetadata0x2D+group_id (32)+identity (32)MetadataRecordPer-member metadata (name + opaque data)
0x2EGroupMetadata0x2E+group_id (32)MetadataRecordPer-group metadata (name + opaque data; a namespace is a root group)
0x2FGroupContextMetadata0x2F+group_id (32)+context_id (32)MetadataRecordPer-context metadata within a group (name + opaque data)
- -

Operation Log & DAG (0x30–0x31)

- - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x30GroupOpLog0x30+group_id (32)+seq (8 BE)Vec<u8>Serialized SignedGroupOp — the governance operation DAG
0x31GroupOpHead0x31+group_id (32)GroupOpHeadValueLatest sequence number + DAG heads
- -
- GroupOpHeadValue definition -
-
-struct GroupOpHeadValue {
-    sequence: u64,                // latest sequence number
-    dag_heads: Vec<[u8; 32]>,     // current DAG head op IDs
-} -
-

The sequence number is stored as 8-byte big-endian in the key to enable efficient range scans in chronological order. The DAG heads track the frontier of the governance operation graph for catch-up protocol.

-
-
- -

Member-Context Tracking (0x32–0x33)

- - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x32GroupMemberContext0x32+group_id (32)+identity (32)+context_id (32)[u8; 32]Member-context join tracking
0x33GroupContextMemberCap0x33+group_id (32)+context_id (32)+identity (32)u8Per-context member capability override
-
- -

Subgroup Hierarchy (0x35–0x36)

- - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x35GroupChildIndex0x35+parent_id (32)+child_id (32)()Bidirectional index: lists children of a group (presence key). Used by collect_descendant_groups() for cascade operations.
0x36GroupParentRef0x36+child_id (32)[u8; 32]Maps child group → parent group. Used by resolve_namespace() to walk up the tree and find the root (namespace).
- -

Namespace Identity & Keys (0x37–0x3A)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
PrefixKey StructKey LayoutValue TypeDescription
0x37NamespaceIdentity0x37+namespace_id (32)NamespaceIdentityValueNode's Ed25519 keypair for this namespace (root group). Auto-generated on first create/join, shared across all subgroups and contexts in the namespace.
0x38NamespaceGovOp0x38+namespace_id (32)+delta_id (32)NamespaceGovOpValueNamespace governance operation log entry. Stores SignedNamespaceOp or opaque skeleton (for ops the node cannot decrypt).
0x39NamespaceGovHead0x39+namespace_id (32)NamespaceGovHeadValueDAG head tracking for the namespace governance DAG: next nonce and current head delta IDs.
0x3AGroupKeyEntry0x3A+group_id (32)+key_id (32)Vec<u8>Per-group encryption key, indexed by key_id. Used for encrypting GroupOp payloads within namespace governance. Rotated on member removal.
- -
NamespaceIdentityValue fields -
-struct NamespaceIdentityValue {
-    public_key: [u8; 32],   // Ed25519 public key (node's member identity in namespace)
-    private_key: [u8; 32],  // Ed25519 private key (signs ops, unwraps group keys via ECDH)
-    sender_key: [u8; 32],   // additional sender key for encrypted sync
-} -
-
- -
NamespaceGovOpValue & NamespaceGovHeadValue definitions -
-
-struct NamespaceGovOpValue {
-    skeleton_bytes: Vec<u8>,  // borsh(SignedNamespaceOp) or OpaqueSkeleton
-} -
-
-struct NamespaceGovHeadValue {
-    sequence: u64,              // next nonce for ops
-    dag_heads: Vec<[u8; 32]>,   // current DAG heads (multi-parent support)
-} -
-
-
- -

Application Services (ApplicationMeta extension)

-

ApplicationMeta now includes a services: Vec<ServiceMeta> field for multi-service bundles. When empty, the application is single-service (backward compat) and uses the top-level bytecode/compiled fields.

-
ServiceMeta fields -
-struct ServiceMeta {
-    name: Box<str>,         // e.g. "chat", "user-mgmt"
-    bytecode: BlobMeta,    // WASM blob for this service
-    compiled: BlobMeta,    // precompiled WASM blob
-} -
-
-
- - -
-

Key Model

-

All keys are statically typed via Key<T>, a newtype over GenericArray<u8, T::Size>. The AsKeyParts / FromKeyParts traits define how a key is decomposed into column + byte components, giving compile-time column assignment and fixed-size key guarantees.

- -
-
-

Key<T> struct

-
-pub struct Key<T: KeyParts>(
-    GenericArray<u8, T::Size>
-);
-
-

The key's byte length is determined at compile time by the associated Size type (a typenum unsigned integer). This means key construction is zero-allocation and key comparison is a simple memcmp.

-
-
-

Column Assignment

-
-pub trait AsKeyParts {
-    type Size: ArrayLength<u8>;
-    const COLUMN: Column;
-    fn as_key(&self) -> Key<Self>;
-}

-pub trait FromKeyParts: AsKeyParts {
-    fn from_key(key: Key<Self>) -> Self;
-} -
-

Each key type declares its COLUMN at the type level. The storage layer uses this to route operations to the correct RocksDB column family without runtime dispatch.

-
-
- -

How It Works

-
-
1
-

Type-Level Sizing

-

A key type like GroupMember declares Size = U65 (1 prefix + 32 group_id + 32 identity). The compiler enforces this — you can't accidentally write a 64-byte key into a 65-byte slot.

-
-
2
-

Column Routing

-

When you call db.get::<GroupMember>(key), the COLUMN associated constant resolves to Column::Group at compile time. No runtime map lookups needed.

-
-
3
-

Prefix Scans

-

For iteration within a column, the prefix byte enables efficient set_iterate_range scans. For example, scanning all members of a group uses prefix 0x21 || group_id as the range start.

-
-
4
-

Codec Selection

-

Values use Borsh serialization by default. State and PrivateState columns use Identity codec (raw bytes) to avoid serialization overhead for application-controlled data.

-
-
- -
- Example: GroupMember key implementation -
-
-pub struct GroupMember {
-    group_id: [u8; 32],
-    identity: [u8; 32],
-}

-impl AsKeyParts for GroupMember {
-    type Size = U65;  // 1 (prefix) + 32 + 32
-    const COLUMN: Column = Column::Group;

-    fn as_key(&self) -> Key<Self> {
-        let mut arr = GenericArray::default();
-        arr[0] = 0x21;  // prefix byte
-        arr[1..33].copy_from_slice(&self.group_id);
-        arr[33..65].copy_from_slice(&self.identity);
-        Key(arr)
-    }
-} -
-
-
- -
- Byte layout legend -
-

The byte layout diagrams throughout this page use the following color scheme:

-
- prefix (1) Single-byte namespace discriminator -
-
- id (32) Primary 32-byte identifier (group, context, etc.) -
-
- id2 (32) Secondary identifier (member, context, etc.) -
-
- id3 (32) Tertiary identifier (for triple-indexed keys) -
-
- seq (8 BE) Big-endian sequence number for ordered scans -
-
- var Variable-length suffix (application keys) -
-
-
-
- -

Transaction Merge: Key vs. Value Correction

-

Prior to this fix, Transaction::merge contained two bugs in the inner map insert logic that corrupted the column keyspace and caused panics during transaction composition.

- -

What Was Wrong

-
    -
  • Put branch: The map insert used the value bytes from Operation::Put as the map key instead of the entry's actual store key. Because multiple distinct store entries can share the same value bytes, this caused key collisions — different entries would overwrite each other in the merged map.
    • -
  • Delete branch: The code called unreachable!() for Operation::Delete, producing a panic whenever a delete operation was present in the transaction being merged. Deletes were never propagated.
    • -
- -

The Fix

-

Both branches are replaced with a single insert that uses entry.key().to_owned() (wrapped in Slice) as the map key and op.clone() as the value. This correctly handles all operation variants:

-
// Before (wrong — uses value as map key, panics on Delete)
-Operation::Put { value } => map.insert(Slice(value.to_owned()), op.clone()),
-Operation::Delete    => unreachable!(),
-
-// After (correct — uses the entry's store key for both variants)
-map.insert(Slice(entry.key().to_owned()), op.clone())
- -

Why This Matters

-
    -
  • A store entry's key and value are entirely separate concepts. The key identifies the row inside a column family (e.g. a GroupMember key laid out as 0x21 || group_id || identity); the value is the Borsh-serialized payload stored at that row. Using the value as the map key violates the column keyspace structure described in the Key Model section above.
    • -
  • With the value-as-key bug in place, any two entries that happened to serialise to the same bytes (e.g. two members with identical role values) would silently collapse into one row during a merge, causing silent data loss for the second entry.
    • -
  • Deletes are a legitimate and necessary operation — for example, removing a member (0x21 prefix row) must propagate through merged transactions to be applied atomically. Panicking on delete made any transaction that mixed puts and deletes impossible to merge.
    • -
- -

Correctness Invariant

-

After the fix, Transaction::merge preserves the following invariant: for every column family, the merged map contains exactly one Operation per distinct store key, and that operation is either Put or Delete. Later operations on the same key win (last-write semantics within the transaction), which is the expected behaviour for an in-memory transaction accumulator.

-
-
- - - - diff --git a/architecture/styles.css b/architecture/styles.css deleted file mode 100644 index 6f8dc9fe60..0000000000 --- a/architecture/styles.css +++ /dev/null @@ -1,824 +0,0 @@ -/* Calimero Core Architecture — Design System */ -@import url('https://fonts.googleapis.com/css2?family=DM+Sans:ital,wght@0,300;0,400;0,500;0,600;1,300&family=JetBrains+Mono:wght@300;400;600;700&display=swap'); - -:root { - /* Backgrounds */ - --bg: #0d1117; - --surface: #161b22; - --surface2: #1c2128; - --surface3: #22272e; - /* Borders */ - --border: #30363d; - --border-hi: #444c56; - /* Brand */ - --accent: #a5ff11; - --accent-glow: rgba(165, 255, 17, 0.18); - --accent-dim: rgba(165, 255, 17, 0.08); - --accent-border: rgba(165, 255, 17, 0.35); - /* Semantic colors */ - --blue: #58a6ff; - --green: #3fb950; - --red: #f85149; - --violet: #a371f7; - --cyan: #39d0d8; - --pink: #f778ba; - --orange: #f0883e; - --amber: #e3b341; - --lime: #a5ff11; - /* Text */ - --text: #c9d1d9; - --text-dim: #8b949e; - --text-bright: #f0f6fc; - /* Layout */ - --sidebar-w: 260px; - --radius-sm: 6px; - --radius-md: 10px; - --radius-lg: 14px; - /* Fonts */ - --font-body: 'DM Sans', system-ui, sans-serif; - --font-mono: 'JetBrains Mono', monospace; -} - -*, *::before, *::after { margin: 0; padding: 0; box-sizing: border-box; } -html { scroll-behavior: smooth; } -body { - background: var(--bg); - color: var(--text); - font-family: var(--font-body); - display: flex; - min-height: 100vh; - -webkit-font-smoothing: antialiased; -} - -::selection { background: var(--accent-glow); color: #fff; } - -a { color: var(--blue); text-decoration: none; transition: color .15s; } -a:hover { color: var(--accent); } - -/* ── SIDEBAR ── */ -.sidebar { - position: fixed; top: 0; left: 0; bottom: 0; width: var(--sidebar-w); - background: var(--surface); - border-right: 1px solid var(--border); - display: flex; flex-direction: column; z-index: 100; - overflow-y: auto; overflow-x: hidden; - scrollbar-width: thin; - scrollbar-color: var(--border) transparent; -} -.sidebar::before { - content: ''; - position: absolute; top: 0; left: 0; right: 0; height: 1px; - background: linear-gradient(90deg, transparent, var(--accent), transparent); - opacity: 0.6; -} - -.sidebar-logo { - padding: 20px 20px 16px; - border-bottom: 1px solid var(--border); - position: relative; -} -.sidebar-logo-inner { - display: flex; align-items: center; gap: 10px; -} -.sidebar-logo-icon { - width: 26px; height: 26px; flex-shrink: 0; overflow: visible; -} -.sidebar-logo-text { display: flex; flex-direction: column; gap: 1px; } -.sidebar-logo-text strong { - font-family: var(--font-body); - font-weight: 600; - font-size: .88rem; - color: var(--text-bright); - letter-spacing: -.2px; - line-height: 1.2; -} -.sidebar-logo-text strong em { font-style: normal; color: var(--accent); } -.sidebar-logo-text span { - font-family: var(--font-mono); - font-size: .6rem; - color: var(--text-dim); - letter-spacing: .5px; - text-transform: uppercase; -} - -.sidebar-search { padding: 12px 14px; } -.sidebar-search input { - width: 100%; padding: 7px 10px; - border: 1px solid var(--border); - border-radius: var(--radius-sm); - background: var(--surface2); - color: var(--text); - font-family: var(--font-body); - font-size: .82rem; - outline: none; - transition: border-color .15s, box-shadow .15s; -} -.sidebar-search input::placeholder { color: var(--text-dim); } -.sidebar-search input:focus { - border-color: rgba(165, 255, 17, 0.4); - box-shadow: 0 0 0 3px rgba(165, 255, 17, 0.06); -} - -.sidebar-nav { flex: 1; padding: 6px 0; } -.nav-section { - padding: 8px 18px 3px; - font-family: var(--font-mono); - font-size: .6rem; - color: var(--text-dim); - text-transform: uppercase; - letter-spacing: 1.2px; - margin-top: 8px; -} -.nav-link { - display: flex; align-items: center; gap: 8px; - padding: 7px 18px; - font-size: .84rem; - color: var(--text-dim); - transition: all .12s; - border-left: 2px solid transparent; - line-height: 1.3; -} -.nav-link:hover { color: var(--text); background: var(--surface2); } -.nav-link.active { - color: var(--accent); - border-left-color: var(--accent); - background: rgba(165, 255, 17, 0.05); - font-weight: 500; - box-shadow: inset 0 0 20px rgba(165, 255, 17, 0.03); -} -.nav-dot { width: 7px; height: 7px; border-radius: 2px; flex-shrink: 0; } -.nav-link.sub { padding-left: 38px; font-size: .8rem; } - -.sidebar-footer { - padding: 14px 18px; - border-top: 1px solid var(--border); -} -.sidebar-footer-brand { - font-size: .68rem; - color: var(--text-dim); - font-family: var(--font-mono); - margin-bottom: 10px; - letter-spacing: .3px; -} -.sidebar-footer-links { - display: flex; - flex-wrap: wrap; - gap: 6px; -} -.sidebar-footer-links a { - display: flex; - align-items: center; - gap: 5px; - padding: 4px 8px; - border-radius: var(--radius-sm); - border: 1px solid var(--border); - background: var(--surface2); - color: var(--text-dim); - font-size: .72rem; - font-family: var(--font-body); - transition: all .15s; - white-space: nowrap; -} -.sidebar-footer-links a:hover { - color: var(--accent); - border-color: var(--accent-border); - background: var(--accent-dim); -} - -/* ── MAIN CONTENT ── */ -.main { margin-left: var(--sidebar-w); flex: 1; min-width: 0; display: flex; } -.content { flex: 1; max-width: 1060px; margin: 0 auto; padding: 40px 36px 80px; min-width: 0; } - -/* ── RIGHT SIDEBAR (Table of Contents) ── */ -.toc-sidebar { - position: sticky; top: 0; align-self: flex-start; - width: 220px; flex-shrink: 0; - padding: 40px 16px 40px 0; - height: 100vh; overflow-y: auto; - scrollbar-width: none; - display: none; -} -.toc-sidebar::-webkit-scrollbar { display: none; } -@media (min-width: 1280px) { .toc-sidebar { display: block; } } - -.toc-title { - font-family: var(--font-mono); font-size: .6rem; - color: var(--text-dim); text-transform: uppercase; - letter-spacing: 1.2px; padding: 0 0 8px 12px; - border-bottom: 1px solid var(--border); margin-bottom: 8px; -} -.toc-link { - display: block; padding: 4px 12px; - font-size: .78rem; color: var(--text-dim); - text-decoration: none; border-left: 2px solid transparent; - transition: all .12s; line-height: 1.4; margin-bottom: 1px; -} -.toc-link:hover { color: var(--text); } -.toc-link.active { - color: var(--accent); border-left-color: var(--accent); - background: var(--accent-dim); -} -.toc-link.toc-h3 { padding-left: 24px; font-size: .74rem; } - -.breadcrumb { - font-family: var(--font-mono); font-size: .72rem; - color: var(--text-dim); margin-bottom: 28px; - display: flex; gap: 6px; align-items: center; flex-wrap: wrap; -} -.breadcrumb .sep { color: var(--border-hi); } -.breadcrumb a { color: var(--text-dim); } -.breadcrumb a:hover { color: var(--accent); } - -h1 { - font-family: var(--font-body); - font-weight: 600; - font-size: 2.6rem; - letter-spacing: -1px; - line-height: 1.1; - margin-bottom: 8px; - background: linear-gradient(135deg, #ffffff 30%, rgba(165, 255, 17, 0.9) 100%); - -webkit-background-clip: text; - -webkit-text-fill-color: transparent; - background-clip: text; -} -h1 em { - font-style: normal; - background: linear-gradient(135deg, var(--accent) 0%, #c9ff73 100%); - -webkit-background-clip: text; - -webkit-text-fill-color: transparent; - background-clip: text; -} -.page-subtitle { - font-family: var(--font-mono); font-size: .78rem; - color: var(--text-dim); margin-bottom: 40px; -} - -h2 { - font-family: var(--font-body); - font-size: 1.35rem; - color: var(--text-bright); - margin-bottom: 16px; - font-weight: 600; - letter-spacing: -.3px; -} -h3 { font-size: .95rem; color: var(--accent); margin: 20px 0 8px; font-weight: 600; } -h4 { - font-family: var(--font-mono); font-size: .85rem; - color: var(--text-bright); margin-bottom: 5px; -} -p, li { font-size: .875rem; color: var(--text-dim); line-height: 1.75; } -ul, ol { padding-left: 18px; margin-bottom: 12px; } -strong { color: var(--text); font-weight: 500; } - -/* ── CARDS ── */ -.card { - background: var(--surface); - border: 1px solid var(--border); - border-radius: var(--radius-lg); - padding: 26px 30px; - margin-bottom: 20px; - position: relative; - overflow: hidden; - backdrop-filter: blur(4px); - transition: border-color .2s, box-shadow .2s; -} -.card:hover { - border-color: var(--border-hi); - box-shadow: 0 4px 24px rgba(0, 0, 0, 0.3); -} -.card::before { content: ''; position: absolute; top: 0; left: 0; right: 0; height: 2px; } -.card.ga::before { background: linear-gradient(90deg, var(--accent), var(--cyan), var(--green)); } -.card.gb::before { background: linear-gradient(90deg, var(--violet), var(--pink), var(--orange)); } -.card.gc::before { background: linear-gradient(90deg, var(--cyan), var(--green), var(--accent)); } -.card.gd::before { background: linear-gradient(90deg, var(--red), var(--accent), var(--violet)); } - -svg { width: 100%; height: auto; display: block; } -/* UI icon SVGs must not stretch to 100% width */ -button svg, a svg, -.docs-search-btn svg, -.search-icon, -.search-input-row svg, -.sidebar-footer-links svg, -.menu-toggle svg { width: auto; height: auto; display: inline; flex-shrink: 0; } -.code { - font-family: var(--font-mono); - background: var(--bg); - color: var(--green); - padding: 2px 6px; - border-radius: 4px; - font-size: .77rem; - border: 1px solid var(--border); -} -.gh-link { - font-family: var(--font-mono); font-size: .72rem; - color: var(--blue); opacity: .7; transition: opacity .15s; -} -.gh-link:hover { opacity: 1; color: var(--accent); } - -/* ── GRIDS ── */ -.g2 { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; } -.g3 { display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 16px; } -.g4 { display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 12px; } -@media (max-width: 900px) { .g2, .g3 { grid-template-columns: 1fr; } } - -/* ── STATS ── */ -.stat { - background: var(--surface2); - border: 1px solid var(--border); - border-radius: var(--radius-md); - padding: 18px; - text-align: center; - transition: border-color .2s, box-shadow .2s; -} -.stat:hover { - border-color: var(--accent-border); - box-shadow: 0 0 20px rgba(165, 255, 17, 0.05); -} -.stat .v { - font-family: var(--font-body); - font-size: 2rem; - font-weight: 600; - color: var(--accent); - line-height: 1; - text-shadow: 0 0 20px rgba(165, 255, 17, 0.3); -} -.stat .l { font-size: .72rem; color: var(--text-dim); margin-top: 4px; font-family: var(--font-mono); } - -/* ── FILE ROW ── */ -.fr { - display: grid; grid-template-columns: 220px 1fr; gap: 12px; - padding: 8px 12px; - background: var(--surface2); - border-radius: var(--radius-sm); - border: 1px solid transparent; - transition: border-color .15s; - margin-bottom: 6px; - align-items: start; -} -.fr:hover { border-color: var(--border-hi); } -.fr .fp { font-family: var(--font-mono); font-size: .75rem; color: var(--blue); word-break: break-all; } -.fr .fd { font-size: .82rem; color: var(--text-dim); line-height: 1.5; } - -/* ── STEP ── */ -.step { - background: var(--surface2); - border: 1px solid var(--border); - border-radius: var(--radius-md); - padding: 18px 20px; - position: relative; - transition: border-color .15s, box-shadow .15s; - margin-bottom: 12px; -} -.step:hover { - border-color: var(--accent-border); - box-shadow: 0 0 16px rgba(165, 255, 17, 0.04); -} -.step .n { - position: absolute; top: -9px; left: 14px; - background: var(--accent); color: #0d1117; - font-family: var(--font-mono); font-weight: 700; font-size: .68rem; - padding: 1px 9px; border-radius: 4px; - letter-spacing: .5px; -} -.step h4 { margin-top: 2px; } - -/* ── CRATE BOX ── */ -.cb { - background: var(--surface2); - border: 1px solid var(--border); - border-radius: var(--radius-md); - padding: 16px; - transition: border-color .15s, box-shadow .15s; -} -.cb:hover { - border-color: var(--accent-border); - box-shadow: 0 0 16px rgba(165, 255, 17, 0.04); -} -.cb h4 { font-size: .85rem; margin-bottom: 6px; } - -/* ── TAGS ── */ -.tag { - display: inline-block; padding: 2px 8px; - border-radius: 4px; font-size: .7rem; - font-family: var(--font-mono); font-weight: 600; - letter-spacing: .3px; -} -.tag-actor { background: rgba(88, 166, 255, 0.12); color: var(--blue); } -.tag-lib { background: rgba(63, 185, 80, 0.12); color: var(--green); } -.tag-cli { background: rgba(165, 255, 17, 0.12); color: var(--accent); } -.tag-types { background: rgba(163, 113, 247, 0.12); color: var(--violet); } -.tag-wasm { background: rgba(247, 120, 186, 0.12); color: var(--pink); } -.tag-store { background: rgba(57, 208, 216, 0.12); color: var(--cyan); } -.tag-done { background: rgba(63, 185, 80, 0.12); color: var(--green); } -.tag-gap { background: rgba(248, 81, 73, 0.12); color: var(--red); } -.tag-partial{ background: rgba(165, 255, 17, 0.12); color: var(--accent); } - -/* ── LEGEND ── */ -.legend { - display: flex; gap: 18px; flex-wrap: wrap; - margin-top: 14px; padding-top: 14px; - border-top: 1px solid var(--border); -} -.legend-item { display: flex; align-items: center; gap: 7px; font-size: .78rem; color: var(--text-dim); } -.ld { width: 10px; height: 10px; border-radius: 3px; flex-shrink: 0; } - -/* ── EXPANDABLE SECTIONS ── */ -details { margin-bottom: 12px; } -details summary { - cursor: pointer; padding: 10px 14px; - background: var(--surface2); - border: 1px solid var(--border); - border-radius: var(--radius-md); - font-family: var(--font-mono); font-size: .84rem; - color: var(--text-bright); - list-style: none; display: flex; align-items: center; gap: 8px; - transition: border-color .15s; -} -details summary:hover { border-color: var(--accent-border); } -details summary::before { - content: '\25b6'; font-size: .6rem; - color: var(--text-dim); transition: transform .2s; -} -details[open] summary::before { transform: rotate(90deg); } -details[open] summary { border-radius: var(--radius-md) var(--radius-md) 0 0; border-bottom-color: transparent; } -details .detail-body { - padding: 16px; - background: var(--surface); - border: 1px solid var(--border); - border-top: none; - border-radius: 0 0 var(--radius-md) var(--radius-md); -} - -/* ── TYPE DEFINITION ── */ -.typedef { - background: var(--bg); - border: 1px solid var(--border); - border-radius: var(--radius-sm); - padding: 14px 16px; margin-bottom: 10px; - font-family: var(--font-mono); font-size: .77rem; - line-height: 1.8; color: var(--text); overflow-x: auto; - white-space: pre-wrap; word-break: break-word; -} -.typedef .kw { color: var(--violet); } -.typedef .ty { color: var(--amber); } -.typedef .field { color: var(--green); } -.typedef .comment { color: var(--text-dim); font-style: italic; } - -/* ── TABS ── */ -.tabs { - display: flex; gap: 4px; margin-bottom: 24px; - background: var(--surface); padding: 4px; - border-radius: var(--radius-md); - border: 1px solid var(--border); - overflow-x: auto; flex-wrap: nowrap; -} -.tab { - padding: 7px 16px; border: none; - background: transparent; color: var(--text-dim); - font-family: var(--font-body); font-weight: 500; font-size: .84rem; - cursor: pointer; border-radius: var(--radius-sm); - transition: all .15s; white-space: nowrap; -} -.tab:hover { color: var(--text); background: var(--surface2); } -.tab.on { - background: var(--accent); color: #0d1117; - font-weight: 600; - box-shadow: 0 0 12px rgba(165, 255, 17, 0.25); -} -.panel { display: none; animation: fadeUp .25s ease; } -.panel.on { display: block; } - -/* ── HERO (index page) ── */ -.hero { text-align: center; padding: 60px 0 40px; } -.hero h1 { - font-size: 3.4rem; - margin-bottom: 16px; - display: block; -} -.hero .tagline { - font-size: 1.05rem; - color: var(--text-dim); - max-width: 580px; - margin: 0 auto 36px; - line-height: 1.7; - font-weight: 300; -} -.hero-cards { - display: grid; - grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); - gap: 14px; margin-top: 32px; -} -.hero-card { - background: var(--surface); - border: 1px solid var(--border); - border-radius: var(--radius-lg); - padding: 22px; - text-align: left; - transition: border-color .2s, transform .2s, box-shadow .2s; - display: block; -} -.hero-card:hover { - border-color: rgba(165, 255, 17, 0.3); - transform: translateY(-2px); - box-shadow: 0 8px 24px rgba(0, 0, 0, 0.3), 0 0 20px rgba(165, 255, 17, 0.04); - color: var(--text); -} -.hero-card h3 { margin-top: 0; font-size: .9rem; } -.hero-card p { font-size: .81rem; } -.hero-card .card-icon { font-size: 1.3rem; margin-bottom: 10px; display: block; } - -/* ── MENU TOGGLE ── */ -.menu-toggle { - display: none; - position: fixed; top: 12px; left: 12px; z-index: 300; - background: var(--surface); - border: 1px solid var(--border); - border-radius: var(--radius-sm); - padding: 7px 11px; color: var(--text); cursor: pointer; font-size: 1rem; - transition: left .25s, border-color .15s, color .15s; - line-height: 1; -} -.menu-toggle:hover { border-color: var(--accent-border); color: var(--accent); } -.menu-toggle.sidebar-open { left: calc(var(--sidebar-w) + 12px); } - -/* ── THEME TOGGLE ── */ -.theme-toggle { - position: absolute; top: 18px; right: 14px; - background: var(--surface2); - border: 1px solid var(--border); - border-radius: var(--radius-sm); - color: var(--text-dim); font-size: 1rem; cursor: pointer; - padding: 4px 8px; - transition: all .15s; line-height: 1; -} -.theme-toggle:hover { color: var(--accent); border-color: var(--accent-border); } - -/* ── ANIMATIONS ── */ -@keyframes fadeUp { - from { opacity: 0; transform: translateY(6px); } - to { opacity: 1; transform: translateY(0); } -} -@keyframes slideIn { - from { opacity: 0; transform: translateY(8px); } - to { opacity: 1; transform: translateY(0); } -} - -/* ── LIGHT THEME ── */ -html.light { - --bg: #ffffff; --surface: #f6f8fa; --surface2: #eaeef2; --surface3: #d0d7de; - --border: #d0d7de; --border-hi: #b1bac4; - --accent: #5b9700; --lime: #5b9700; - --accent-glow: rgba(91, 151, 0, 0.15); - --accent-dim: rgba(91, 151, 0, 0.08); - --accent-border: rgba(91, 151, 0, 0.35); - --text: #24292f; --text-dim: #57606a; --text-bright: #1f2328; -} -html.light .sidebar { background: #f6f8fa; } -html.light h1 { - background: linear-gradient(135deg, #1f2328 40%, #5b9700 100%); - -webkit-background-clip: text; - -webkit-text-fill-color: transparent; - background-clip: text; -} -html.light .code { background: #eaeef2; color: #1a7f37; border-color: #d0d7de; } -html.light .typedef { background: #f0f3f6; border-color: #d0d7de; color: #24292f; } - -/* ── DOCS SEARCH BUTTON (sidebar) ── */ -.docs-search-btn { - display: flex; align-items: center; gap: 8px; - width: 100%; padding: 7px 10px; - border: 1px solid var(--border); - border-radius: var(--radius-sm); - background: var(--surface2); - color: var(--text-dim); - font-family: var(--font-body); - font-size: .82rem; - cursor: pointer; - text-align: left; - transition: border-color .15s, color .15s, box-shadow .15s; - margin-bottom: 8px; -} -.docs-search-btn:hover { - border-color: rgba(165, 255, 17, 0.4); - color: var(--text); - box-shadow: 0 0 0 3px rgba(165, 255, 17, 0.06); -} -.docs-search-btn span { flex: 1; } -.docs-search-btn kbd { - font-family: var(--font-mono); - font-size: .65rem; - padding: 2px 5px; - border: 1px solid var(--border-hi); - border-radius: 4px; - color: var(--text-dim); - background: var(--surface3); - line-height: 1.4; -} - -/* ── SEARCH OVERLAY ── */ -.search-overlay { - position: fixed; inset: 0; z-index: 1000; - background: rgba(1, 4, 9, 0.7); - backdrop-filter: blur(4px); - display: flex; align-items: flex-start; justify-content: center; - padding-top: 12vh; - opacity: 0; pointer-events: none; - transition: opacity .18s ease; -} -.search-overlay.open { - opacity: 1; pointer-events: all; -} - -.search-modal { - width: 100%; max-width: 620px; - background: var(--surface); - border: 1px solid var(--border-hi); - border-radius: var(--radius-lg); - box-shadow: 0 24px 64px rgba(0,0,0,0.6), 0 0 0 1px rgba(165,255,17,0.06); - overflow: hidden; - transform: translateY(-8px); - transition: transform .18s ease; - display: flex; flex-direction: column; - max-height: 70vh; -} -.search-overlay.open .search-modal { - transform: translateY(0); -} - -.search-input-row { - display: flex; align-items: center; gap: 10px; - padding: 14px 16px; - border-bottom: 1px solid var(--border); - flex-shrink: 0; -} -.search-icon { color: var(--text-dim); flex-shrink: 0; } -#search-input { - flex: 1; background: transparent; border: none; outline: none; - color: var(--text-bright); font-family: var(--font-body); - font-size: 1rem; caret-color: var(--accent); -} -#search-input::placeholder { color: var(--text-dim); } -.search-esc-hint { - font-family: var(--font-mono); font-size: .65rem; - padding: 2px 6px; border-radius: 4px; - border: 1px solid var(--border-hi); - background: var(--surface2); color: var(--text-dim); - flex-shrink: 0; -} - -.search-results { - flex: 1; overflow-y: auto; padding: 6px; - scrollbar-width: thin; - scrollbar-color: var(--border) transparent; -} - -.search-hint { - padding: 20px 16px; - color: var(--text-dim); - font-size: .85rem; - text-align: center; -} -.search-hint strong { color: var(--text); } -.search-loading::before { - content: ''; - display: inline-block; width: 12px; height: 12px; - border: 2px solid rgba(165,255,17,0.2); - border-top-color: var(--accent); - border-radius: 50%; - animation: spin .7s linear infinite; - margin-right: 8px; vertical-align: middle; -} -@keyframes spin { to { transform: rotate(360deg); } } - -.search-result-item { - display: block; padding: 10px 12px; - border-radius: var(--radius-sm); - border: 1px solid transparent; - color: var(--text); text-decoration: none; - transition: background .1s, border-color .1s; - margin-bottom: 2px; -} -.search-result-item:hover, -.search-result-item.selected { - background: var(--surface2); - border-color: var(--border); - color: var(--text); -} -.search-result-item.selected { - border-color: var(--accent-border); - background: var(--accent-dim); -} -.search-result-header { - display: flex; align-items: center; gap: 8px; - margin-bottom: 4px; -} -.search-result-title { - font-size: .9rem; font-weight: 500; color: var(--text-bright); -} -.search-result-title mark { - background: rgba(165, 255, 17, 0.25); - color: var(--accent); border-radius: 2px; padding: 0 1px; -} -.search-result-excerpt { - font-size: .78rem; color: var(--text-dim); - line-height: 1.6; margin: 0; - display: -webkit-box; -webkit-line-clamp: 2; -webkit-box-orient: vertical; - overflow: hidden; -} -.search-result-excerpt mark { - background: rgba(165, 255, 17, 0.2); - color: var(--accent); border-radius: 2px; padding: 0 1px; -} - -.search-footer-bar { - display: flex; gap: 16px; align-items: center; - padding: 8px 14px; - border-top: 1px solid var(--border); - font-size: .72rem; color: var(--text-dim); - font-family: var(--font-mono); - flex-shrink: 0; -} -.search-footer-bar kbd { - display: inline-block; padding: 1px 5px; - border: 1px solid var(--border-hi); - border-radius: 3px; background: var(--surface2); - font-family: var(--font-mono); font-size: .65rem; - color: var(--text-dim); margin-right: 3px; -} - -/* ── SCROLLBAR ── */ -::-webkit-scrollbar { width: 6px; height: 6px; } -::-webkit-scrollbar-track { background: transparent; } -::-webkit-scrollbar-thumb { background: rgba(255,255,255,0.08); border-radius: 3px; } -::-webkit-scrollbar-thumb:hover { background: rgba(255,255,255,0.14); } - -/* ── STORAGE-PRIMITIVES TABLE / DECISION TREE (used in crates/store and crates/sdk) ── */ -.prim-table { width: 100%; border-collapse: collapse; margin-bottom: 18px; font-size: .82rem; } -.prim-table th { - text-align: left; padding: 10px 12px; font-family: var(--font-mono); font-size: .7rem; - color: var(--text-dim); text-transform: uppercase; letter-spacing: .5px; - border-bottom: 2px solid var(--border); background: var(--surface2); -} -.prim-table th:first-child { border-radius: 8px 0 0 0; } -.prim-table th:last-child { border-radius: 0 8px 0 0; } -.prim-table td { padding: 10px 12px; border-bottom: 1px solid var(--border); vertical-align: top; } -.prim-table tr:hover td { background: var(--surface2); } -.prim-table .pt-name { font-family: var(--font-mono); font-weight: 600; color: var(--green); white-space: nowrap; } -.prim-table .pt-code { font-family: var(--font-mono); font-size: .76rem; color: var(--cyan); } -.prim-table .pt-desc { color: var(--text-dim); font-size: .78rem; } -.prim-table tr.baseline td { background: rgba(255,255,255,0.02); } -.prim-table tr.baseline .pt-name { color: var(--text-dim); font-weight: 500; } - -.prim-tree { - font-family: var(--font-mono); font-size: .82rem; line-height: 1.7; - background: var(--surface2); border: 1px solid var(--border); border-radius: 8px; - padding: 14px 18px; margin: 6px 0 16px 0; white-space: pre; overflow-x: auto; -} -.prim-tree .arrow { color: var(--cyan); } -.prim-tree .target { color: var(--green); font-weight: 600; } -.prim-tree .public { color: var(--cyan); } - -/* ── MOBILE ── */ -@media (max-width: 860px) { - .sidebar { transform: translateX(-100%); transition: transform .25s; } - .sidebar.open { transform: translateX(0); } - .main { margin-left: 0; } - .menu-toggle { display: block; } - h1 { font-size: 2rem; } - .hero h1 { font-size: 2.4rem; } - .content { padding: 40px 18px 60px; } - /* Allow the type-name cell to wrap so the table doesn't force horizontal scroll. */ - .prim-table .pt-name { white-space: normal; } -} - -/* ── ANIMATED DIAGRAM ENGINE (shared; see seq-diagram.js) ── */ -.diagram-container { overflow-x: auto; -webkit-overflow-scrolling: touch; margin: 16px 0; } -.diagram-container svg { border-radius: 10px; max-height: 75vh; } -.diagram-container.seq svg { min-width: 700px; } -.seq-step { opacity: 0; } -.diagram-container.animating .seq-step { - animation: stepIn .4s ease-out forwards; - animation-delay: calc(var(--step-i) * .45s); -} -@keyframes stepIn { from { opacity: 0; transform: translateY(3px); } to { opacity: 1; transform: none; } } -.diagram-tools { display: flex; justify-content: flex-end; margin-bottom: 8px; } -.replay-btn { - padding: 6px 16px; border: 1px solid var(--border); - background: var(--surface2); color: var(--text-dim); - font-family: var(--font-mono); font-size: .74rem; - border-radius: 8px; cursor: pointer; transition: all .15s; -} -.replay-btn:hover { border-color: var(--accent); color: var(--accent); background: var(--surface); } -.diagram-desc { font-size: .84rem; color: var(--text-dim); margin-bottom: 14px; line-height: 1.65; } -@media (prefers-reduced-motion: reduce) { - .seq-step { opacity: 1; } - .diagram-container.animating .seq-step { animation: none; } -} - -/* ── SHARED SPEC TABLE (protocol reference specification sections) ── */ -.spec-tbl { width: 100%; border-collapse: collapse; margin: 10px 0; font-size: .85rem; } -.spec-tbl th, .spec-tbl td { text-align: left; padding: 8px 12px; border-bottom: 1px solid var(--border); vertical-align: top; } -.spec-tbl th { color: var(--text-bright); font-weight: 600; } -.spec-tbl td { color: var(--text-dim); } -.spec-tbl code { font-size: .82rem; white-space: nowrap; } -.spec-tbl td.wrap code { white-space: normal; } diff --git a/architecture/system-overview.html b/architecture/system-overview.html deleted file mode 100644 index dbcb36c420..0000000000 --- a/architecture/system-overview.html +++ /dev/null @@ -1,716 +0,0 @@ - - - - -System Overview — Calimero Core Architecture - - - -
-
-
Home/System Overview
- -

System Overview

-

Calimero Core v0.10.1-rc.8 — 21 crates Rust workspace

- - -
-
21
crates
-
11
core libraries
-
1
CLI binaries
-
14+
sample apps
-
- - -
-

High-Level Architecture

-

The system is organized in horizontal layers. External clients and CLIs connect through the Server layer, which delegates to Actix actors. Actors coordinate via typed messages and share a common storage layer at the bottom.

- - - - - - - - - - - - meroctl CLI - operator / admin - - - Admin Dashboard - embedded SPA (React) - - - External Clients - RPC / WS / SSE - - - - - - - - - Server Layer - Axum · REST Admin API · JSON-RPC 2.0 · WebSocket · SSE Events · Prometheus Metrics - - - - - - - - - - - NodeManager - - actor - SRP split: NodeClients + NodeManagers + NodeState - Delta handling, namespace events, key delivery - Blob cache (LRU), sync sessions, heartbeat - crates/node · Handler<NodeMessage> - - - - ContextManager - Contexts, groups, namespace governance DAGs - Lifecycle recovery, metrics, 30+ handlers - GroupStore (modular), namespace identity - crates/context · Handler<ContextMessage> - - actor - - - - - - - - - - - - - - NetworkManager - - actor - libp2p: gossipsub, Kademlia, mDNS, relay - Streams, AutoNAT v2, DCUtR, rendezvous - crates/network · Handler<NetworkMessage> - - - - SyncManager - Hash compare - Level-wise sync - Snapshot / Delta - - - - WASM Runtime - Wasmer + Cranelift - 50+ host functions - CRDT collections - - - - - - - - - Storage Layer - RocksDB · 11 column families · typed keys · causal DAG · CRDT collections · blob store - - - - mero-auth - - crate - JWT issuing + verification - Pluggable providers (NEAR, ETH, …) - Challenge / nonce auth flow - Embedded or reverse-proxy mode - crates/mero-auth - - - - - - - - P2P Network - Other Calimero nodes - gossipsub · streams · DHT - topic-based message routing - - - - - - - - - Node - - Context - - Network - - Sync - - Runtime - - Storage - - Server - - Auth - - - direct - - async / indirect - -
- - -
-

Crate Dependency Graph

-

Directed edges show compile-time dependencies. Top-level binaries depend on the server and node crates, which fan out through the core libraries, all converging on shared primitives at the bottom.

- - - - - - - - - - - - - - - merod - - meroctl - - - - server - - - - - - - - - node - - - context-primitives - - - node-primitives - - - mero-auth - - - - - - - - - - context - - - network - - - store - - - runtime - - - sync - - - - - - - - - - - dag - - - network-primitives - - - store-rocksdb - - - storage - - - sys - - - calimero-sdk - - - - - - - - - - - - primitives - - - - - - - - - - -
-
Binary / Primitives
-
Server / SDK
-
Node
-
Context
-
Network
-
Storage
-
Runtime
-
Auth
-
-
- - -
-

Actor System

-

Calimero uses Actix for its actor model. Three primary actors manage the system, plus a long-running async task for synchronization and a background garbage-collection actor.

- -
-
-

NodeManager actor

-

Central orchestrator, decomposed into three layers: NodeClients (injected service facades), NodeManagers (BlobStore, SyncManager), and NodeState (mutable runtime state with DashMap for lock-free concurrent access). Handles namespace events, key delivery, and blob cache with multi-phase LRU eviction.

-

Key messages:

-
    -
  • HandleNetworkEvent — process gossip, stream, namespace, specialized node events
    • -
  • HandleStateDelta — route incoming delta to context
    • -
  • HandleNamespaceOp — forward namespace governance ops, trigger key delivery
    • -
  • BootContext — initialize context after creation
    • -
-
-
-

ContextManager actor

-

Manages all contexts, groups, and namespace governance. 30+ message handlers for execution, governance, membership, upgrades, and namespace operations. Holds per-namespace DagStores (via HashMap<namespace_id, Arc<Mutex<DagStore>>>). Lifecycle module handles startup recovery and 30s namespace heartbeat. Prometheus metrics for execution and governance events.

-

Key messages:

-
    -
  • ExecuteMethod — run WASM method, produce delta (auto-resolves executor identity)
    • -
  • ApplySignedNamespaceOp — ingest namespace governance op into namespace DAG
    • -
  • CreateContext / JoinGroup / ListNamespaces
    • -
  • UpdateApplication — upgrade WASM binary
    • -
-
-
-

NetworkManager actor

-

Wraps the libp2p swarm. Publishes and subscribes to gossipsub topics, manages Kademlia DHT, and handles stream-based request/response protocols. One topic per context + group topics.

-

Key messages:

-
    -
  • Publish — send gossip message on topic
    • -
  • Subscribe / Unsubscribe
    • -
  • OpenStream — direct peer-to-peer stream
    • -
  • ProvideRecord / GetRecord — DHT ops
    • -
-
-
- -
-
-

SyncManager async task

-

Not an Actix actor — runs as a long-lived async task spawned at startup. Implements four sync protocols: hash comparison (cheap), level-wise (medium), snapshot (heavy), and direct delta exchange. Triggered by heartbeat mismatches.

-

Communicates via tokio::broadcast channels and direct LazyRecipient sends to NodeManager.

-
-
-

GC Actor background

-

Garbage collection actor that periodically cleans up expired blobs, stale peer entries, and orphaned context data. Runs on a configurable timer. Low-priority Actix actor.

-

Reads from storage columns and issues deletes for entries past their TTL.

-
-
- -

Communication Patterns

- - - - - - - - NodeManager - - - ContextManager - - - NetworkManager - - - SyncManager - - - - LazyRecipient - - - LazyRecipient - - - LazyRecipient - - - - tokio::broadcast channel - - - - - - - - Heartbeat events, sync triggers, and state updates flow through the broadcast channel - -
- - -
-

Data Flow: Method Execution

-

End-to-end flow from a client calling a WASM method to state convergence across all peers.

- -
-
-
1
-

Client → Server

-

External client sends a JSON-RPC 2.0 request to the /rpc endpoint. The request specifies context_id, method, and args. Authenticated via JWT bearer token (mero-auth).

-
-
2
-

Server → ContextManager

-

The JSON-RPC handler constructs an ExecuteMethod message and sends it to ContextManager via LazyRecipient<ContextMessage>. Includes caller identity, payload, and executor kind.

-
-
3
-

ContextManager → Runtime

-

ContextManager loads the WASM module (cached in-memory), builds a VMContext with storage handles, caller identity, and method metadata. Invokes runtime::execute().

-
-
4
-

Runtime → WASM

-

Wasmer executes the compiled WASM function. The application calls host functions for storage reads/writes, CRDT operations, event emissions, and cross-context calls. All mutations are collected in a pending changeset.

-
-
-
-
5
-

State Commit

-

On successful return, the changeset is committed to storage. A CausalDelta is constructed containing the diff, new root_hash, parent hashes, and the operation metadata. Appended to the context's DAG.

-
-
6
-

Gossip Broadcast

-

The delta is serialized and published via NetworkManager::Publish on the context's gossipsub topic. The message includes the StateDelta payload and the sender's peer ID.

-
-
7
-

Remote Nodes Receive + Apply

-

Peer nodes receive the gossip message. NodeManager::HandleStateDelta routes it to the correct context. ContextManager verifies the delta's parent hash exists in the local DAG, then applies it to storage.

-
-
8
-

Convergence

-

Heartbeats carry each context's root_hash. If peers disagree, SyncManager triggers a catch-up protocol (hash comparison → level-wise → snapshot fallback). All peers eventually converge to the same state.

-
-
-
- - - - - - - - Client - JSON-RPC - - - - - Server - Axum - - - - - ContextMgr - ExecuteMethod - - - - - Runtime - WASM exec - - - - - Storage - commit delta - - - - - Gossip - broadcast - - - - - Peers - apply + verify - - - - - Converge - heartbeat sync - -
- - -
-

Data Flow: Governance Operation

-

How an admin action (e.g. adding a member, changing capabilities) propagates across the group and reaches eventual consistency.

- -
-
-
1
-

Admin Action

-

An admin (via REST API or meroctl) submits a governance action. The server translates this into a GroupOp variant — e.g. MemberAdded, CapabilitiesSet, TargetApplicationSet. Includes the target group ID.

-
-
2
-

Sign + Apply Locally

-

The operation is signed with the node's namespace identity (Ed25519 keypair stored per root group in the datastore), producing a SignedGroupOpV1. Includes: state_hash (current group state), nonce (monotonic), and parent hash. The op is applied to local GroupStore and appended to the OpLog DAG.

-
-
3
-

Gossip Broadcast

-

The signed operation is published via gossipsub on the group topic (separate from context data topics). All group members subscribed to this topic receive the message.

-
-
-
-
4
-

Remote Ingestion

-

Receiving nodes verify the Ed25519 signature and check the sender has sufficient capabilities to perform the operation. The op is added to the local DagStore. If parents are missing, the op enters a pending queue until parents arrive.

-
-
5
-

Catch-Up Protocol

-

Heartbeats carry each group's latest state_hash. On mismatch, peers exchange GroupDeltaRequest / GroupDeltaResponse messages to transfer missing operations. The DAG structure ensures causal ordering is preserved during catch-up.

-
-
-
- - - - - - - - Admin API - GroupOp - - - - - Sign + Apply - Ed25519 · OpLog - - - - - Gossip - group topic - - - - - Remote Verify - DagStore · pending - - - - - Catch-Up - GroupDeltaReq/Resp - - Each arrow represents a message boundary — failures at any stage are recoverable via the catch-up protocol - -
- - -
-

Cross-Crate Communication

-

Crates communicate via thin async façade structs that wrap LazyRecipient<M>. Each façade provides typed, async methods that hide the Actix message-passing details. This keeps crate boundaries clean — consumers never import Actix directly.

- -
-
-

NodeClient

-

LazyRecipient<NodeMessage>

-

Thin async façade used by Server and ContextManager to send commands to the NodeManager actor.

-

Key Methods

-
    -
  • boot_context()
    • -
  • handle_network_event()
    • -
  • get_peers()
    • -
  • broadcast_delta()
    • -
  • request_blob()
    • -
-
-
-

ContextClient

-

LazyRecipient<ContextMessage>

-

Used by Server (for RPC execution) and NodeManager (for delta routing and namespace governance) to invoke context-level operations.

-

Key Methods

-
    -
  • execute_method()
    • -
  • create_context()
    • -
  • join_group() / list_namespaces()
    • -
  • apply_signed_namespace_op()
    • -
  • update_application()
    • -
  • get_context_info()
    • -
-
-
-

NetworkClient

-

LazyRecipient<NetworkMessage>

-

Used by NodeManager, ContextManager, and SyncManager to publish messages, open streams, and manage topic subscriptions.

-

Key Methods

-
    -
  • publish()
    • -
  • subscribe()
    • -
  • unsubscribe()
    • -
  • open_stream()
    • -
  • get_peers()
    • -
  • provide_record()
    • -
-
-
- -

Façade Pattern

- - - - - - - - Server / RPC Handler - async fn handle_request() - - - NodeManager - fn handle(&mut self, msg) - - - - ContextClient - .execute_method(ctx, args) - await LazyRecipient::send() - - - - ContextManager - Handler<ContextMessage> - Actix actor mailbox - - - - - - - async send - - - typed call - typed call - - - - Result<T, Error> - returned to caller - - - -
- How LazyRecipient works -
-

LazyRecipient<M> is a wrapper around Actix's Recipient<M> that defers resolution. It stores an Arc<OnceCell<Recipient<M>>> internally. On first use, the address is resolved from the actor registry. Subsequent sends reuse the cached recipient.

-

This pattern breaks circular initialization dependencies — actors can hold LazyRecipients to each other without needing all actors to be started simultaneously. The send() method is async and returns the actor's response.

-
-pub struct LazyRecipient<M: Message> {
-    inner: Arc<OnceCell<Recipient<M>>>,
-}

-impl<M> LazyRecipient<M> {
-    pub async fn send(&self, msg: M) -> Result<M::Result> { /* ... */ }
-} -
-
-
- -
- Message envelope pattern -
-

Each actor defines a single top-level enum that wraps all possible messages. This allows a single Handler impl to dispatch on variants, simplifying the API surface.

-
-pub enum ContextMessage {
-    ExecuteMethod { ctx: ContextId, method: String, args: Vec<u8> },
-    CreateContext { params: CreateParams },
-    JoinGroup { invitation: GroupInvitation },
-    ApplySignedNamespaceOp { namespace_id: [u8; 32], op: SignedNamespaceOp },
-    HandleGroupOp { op: SignedGroupOpV1 },
-    ListNamespaces { filter: NamespaceFilter },
-    UpdateApplication { ctx: ContextId, blob: BlobId },
-    // ... 25+ more variants
-} -
-
-
-
- -
-
- - - diff --git a/architecture/tee-fleet-ha.html b/architecture/tee-fleet-ha.html deleted file mode 100644 index 285996b79c..0000000000 --- a/architecture/tee-fleet-ha.html +++ /dev/null @@ -1,406 +0,0 @@ - - - - -TEE Fleet High Availability — Calimero Core Architecture - - - -
-
-
Home/Fleet HA
- - -

TEE Fleet High Availability

-

Hardware-attested read replicas that hold group keys and serve a namespace's data

- -
-
ReadOnlyTee
replica role
-
Namespace
HA & entitlement boundary
-
TDX
attestation trust root
-
Per-group
encryption key model
-
- - -
-

What fleet HA is

-

-A TEE fleet is a pool of hardware-attested replica nodes that hold a namespace's -group keys and serve reads for its data. Each replica joins as a -ReadOnlyTee member — it can decrypt and answer sync requests, but it can -never write. The fleet exists so that a namespace's data stays available and queryable even when -the human-operated owner node is offline, NAT'd, or asleep. -

-

-Two planes cooperate. The control plane (the -mdma service) -is the source of truth for entitlement: a namespace is configured or paid for once, and -mdma decides which fleet nodes should serve it. A fleet sidecar (shipped -by the mero-tee -repo) runs alongside each merod replica and drives the per-node lifecycle -by calling meroctl. Core itself owns the on-chain governance: attestation -verification, admission, key delivery, and purge. -

-

-The key split to keep straight: entitlement is per-namespace (paid/configured -once), but admission is per-subgroup (each Restricted subgroup the replica should -read needs its own membership row and key). The sections below walk the full lifecycle — -fleet-join, admission, key delivery, transparent subgroup follow-on, and disable/leave/purge. -

-
- - -
-

Roles & boundaries

- -

The ReadOnlyTee role

-
    -
  • Directly-rowed, never inherited. A ReadOnlyTee membership is - always a stored (member, group) row written by admission. It is - never conferred by the inherited-membership parent-walk — a replica is only ever a member - of the scopes it was explicitly admitted to.
    • -
  • Read-only. The role cannot author governance ops or state deltas. It exists to - decrypt and serve, not to mutate.
    • -
  • Auto-follow on. Admission sets both auto-follow flags - (contexts and subgroups) to - true, so a replica tracks new contexts in the scopes it holds. See - Auto-Follow for the propagation machinery.
    • -
- -

The namespace as the HA boundary

-

-A namespace is its own root group — there is no separate -Namespace type; the namespace is the no-parent -ContextGroup at the top of a strict tree (see -Membership & Leave for the hierarchy recap). Fleet HA -is scoped to a namespace: entitlement, the admission policy, and the disable/purge boundary all -sit at the namespace root. A replica admitted at the root serves the namespace; per-subgroup rows -extend its reach into Restricted children. -

- -

Two distinct key families

-

-Do not conflate them: -

-
    -
  • Per-group encryption keys (GroupKeyring) — the AES keys that - protect a group's replicated data. These are delivered to a replica after admission and - are the subject of most of this page.
    • -
  • The node's own storage / disk key — the AES-256 key merod - fetches from the KMS at startup to encrypt its datastore and blobstore on disk. That is the - subject of TEE Mode, a separate concern from group keys.
    • -
-
- - -
-

Fleet-join & attestation announce

-

-The fleet sidecar starts a replica's join by calling -meroctl tee fleet-join, which hits -POST /admin-api/tee/fleet-join. The handler -(crates/server/src/admin/handlers/tee/fleet_join.rs): -

-
    -
  1. Resolves the node's namespace identity (the keypair the replica joins under).
    2. -
  2. Builds report_data = nonce(32) || Sha256(pubkey), binding a fresh - random nonce to the node's public key.
    3. -
  3. Calls generate_attestation to produce a real TDX quote - (crates/tee-attestation/src/generate.rs). Mock quotes are rejected on - this path — fleet-join is production hardware only.
    4. -
  4. Broadcasts - BroadcastMessage::TeeAttestationAnnounce { quote_bytes, public_key, nonce, node_type: ReadOnly } - on the gossip topic ns/<hex(namespace_id)>.
    5. -
-

-A single publish into an empty mesh is silently dropped — gossipsub has no replay, and at boot the -replica may have no peers yet. The handler defends against this by re-announcing: -it republishes the announce roughly every 2 s for up to ~30 s per call, issues a namespace -bootstrap pull each cycle to seed the mesh, and polls for its own admission between announces. If -30 s elapse without admission the call returns; the sidecar retries. -

-
-Announce loop, step by step -
-
    -
  1. Generate nonce, build report_data, generate the TDX quote bound to the namespace-identity pubkey.
    2. -
  2. Publish TeeAttestationAnnounce on ns/<hex(namespace_id)>.
    3. -
  3. Sleep ~2 s; run a namespace bootstrap pull to populate the mesh.
    4. -
  4. Poll: has a verifier written a ReadOnlyTee row for this node yet?
    5. -
  5. If not and the ~30 s budget remains, re-announce and loop. (No replay — every publish is fresh into whatever mesh now exists.)
    6. -
  6. On admission, fall through to joining contexts and publishing the self-signed auto-follow op (see Auto-Follow).
    7. -
-
-
-
- - -
-

Verifier admission & policy

-

-A verifier is any existing member of the namespace that receives the announce. The -inbound handler is crates/node/src/handlers/tee_attestation_admission.rs. -It verifies the quote before trusting anything: verify_attestation -(crates/tee-attestation/src/verify.rs) checks the DCAP signature, that the -nonce in report_data is fresh, and the measurements. Only then does it -call admit_tee_node -(crates/context/src/handlers/admit_tee_node.rs). -

- -

Policy is namespace-scoped

-

-The admission policy lives on the namespace root, never on a subgroup. -read_tee_admission_policy -(crates/governance-store/src/tee.rs) resolves whatever scope it is given -up to the namespace root before reading. Any policy bytes attached to a subgroup are inert — see -the Auto-Follow page's policy-scope note for why subgroup policies -were deliberately disabled. -

- -

Admission checks

-
    -
  • Measurement allowlist. allowed_mrtd must be non-empty (an - empty allowlist fails closed). The quote's mrtd, - rtmr0..3, and tcb_status are checked against - their allowlists.
    • -
  • Mock gate. Mock quotes are admitted only when accept_mock is - set — a development-only escape hatch.
    • -
  • Per-group quote-hash replay. is_quote_hash_used rejects a - quote whose hash was already consumed in this scope, blocking replay of a captured announce.
    • -
  • Idempotent. If a direct row for this member already exists, admission is a no-op rather - than an error — safe under the re-announce loop.
    • -
- -

What admission writes

-

-On success the verifier writes a ReadOnlyTee direct row plus a signing -key for the member, and publishes -GroupOp::MemberJoinedViaTeeAttestation. That op propagating is exactly -what the joiner's poll in fleet_join.rs is waiting on. -

-
- - -
-

Key delivery & recovery

-

-A row alone is not enough — the replica needs the group's encryption key to decrypt anything. After -admission, a key-holder (the admitting verifier, or any member that holds the key) -delivers it. -

- -

The one-shot delivery

-

-The key-holder calls deliver_group_key_to_member, which publishes -NamespaceOp::Root(RootOp::KeyDelivery { group_id, envelope }) on the -namespace DAG. The envelope is ECDH-wrapped for the recipient's public key. The recipient applies -it via apply_received_group_key and stores the key. -

- -

The durable recovery pull

-

-A one-shot delivery can be missed (offline at delivery time, dropped broadcast). The joiner side -therefore has a durable fallback: recover_missing_group_keys -(crates/node/src/sync/manager/namespace_sync.rs) sends a -GroupKeyRequest; a member-peer that holds the key answers with -GroupKeyResponse. The responder's authorization is -role-agnostic, so it will serve a ReadOnlyTee replica -just as it would any member. The recovery pull runs: -

-
    -
  • at the end of every namespace sync,
    • -
  • on an interval, and
    • -
  • on receipt of the relevant gossip event.
    • -
-

-This turns key acquisition from a fragile single broadcast into a self-healing loop. -

- -

Open subgroups need no per-subgroup key

-

-Data in an Open-chain subgroup is encrypted under the namespace key, not -a distinct per-subgroup key. A replica admitted at the namespace root already holds that key, so it -can read Open subgroups with no additional delivery. Per-subgroup keys only matter for -Restricted subgroups, which is what the next section is about. -

-
- - -
-

Transparent per-subgroup admission (PR #2772)

-

-Entitlement is per-namespace, but a Restricted subgroup is a private island: a root-admitted -replica has no row in it and no key for it. Manually re-attesting into every Restricted subgroup -would be brittle. PR #2772 makes this transparent — the replica is folded into the -Restricted subgroups it should read without a fresh quote. -

-

-Scope: Restricted subgroups only. Open subgroups are already readable via inherited -membership and the namespace key, so they need nothing here. -

- -

How it works

-

-A key-holder-side subscriber (tee_subgroup_admit) reacts to two op events: -

-
    -
  • OpEvent::SubgroupCreated — a new Restricted subgroup appears. - The subscriber admits the namespace's existing root ReadOnlyTee members - into it.
    • -
  • OpEvent::TeeMemberAdmitted at the root — a new TEE replica - joins the namespace. The subscriber admits it into the Restricted subgroups this node holds keys - for.
    • -
-

-Crucially, no new attestation is requested. The member's already-verified verdict is read back from -the root op log and reused. Both branches call the same -admit_tee_node used at the root — and because the actor holds the key, that -call writes the row and delivers the per-subgroup key in one step. -

- -

Two safety details

-
    -
  • Root-only loop guard. The subscriber acts only at the root, preventing fan-out echo - (an admission triggering an event that triggers another admission).
    • -
  • Bounded wake-then-reread. The apply path can emit the op event before the op log has - durably persisted it. A bounded retry absorbs that timing window so the verdict read-back does not - race the persist.
    • -
-
- - -
-

Disable → leave → purge

-

-Decommissioning a replica from a namespace is a coordinated, cross-repo flow. The control plane -(mdma) is the source of truth and never reaches into the node directly — it uses -soft disable. -

- -

Soft disable (control plane)

-

-mdma drops the namespace from the fleet node's should-join assignments. -That is the entire control-plane action: there is no kill switch sent to core. The mero-tee fleet -sidecar notices the namespace has disappeared from its assignments — but only on a -confirmed-good poll, so a transient assignment-fetch failure does not trigger an accidental -teardown. -

- -

Leave (node, via sidecar)

-

-On a confirmed drop, the sidecar runs meroctl namespace leave <namespace_id>. -That publishes GroupOp::MemberLeft -(crates/governance-store/src/ops/group/member_left.rs), which -cascades: it removes the node's direct row at the root and in -every subgroup it held a row in, emitting TeeMemberRemoved per -subgroup. (General leave/eviction semantics are covered in -Membership & Leave.) -

- -

Purge (node, role-scoped)

-

-self_purge (crates/context/src/self_purge.rs) -reacts to those removals. It is role-scoped: only -ReadOnlyTee removals hard-purge; a non-TEE removal stays a soft-leave. For -a TEE removal it runs PurgeAction::Namespace, which cascades the whole -subtree and: -

-
    -
  • deletes local replicated data,
    • -
  • deletes the signing keys,
    • -
  • deletes the AES group encryption keys (PR #2776), and
    • -
  • unsubscribes from the namespace gossip topic.
    • -
-

-A purge that is interrupted (crash mid-cascade) is not lost: a durable -pending-self-purge marker plus a startup reconcile sweep complete it on the next -restart (crates/governance-store/src/local_state.rs). -

-
-Disable → purge sequence -
-
    -
  1. mdma drops the namespace from the node's should-join assignments (soft disable).
    2. -
  2. Sidecar observes the drop on a confirmed-good poll and runs meroctl namespace leave.
    3. -
  3. Core publishes GroupOp::MemberLeft; it cascades, emitting TeeMemberRemoved for the root and each subgroup row.
    4. -
  4. self_purge sees the ReadOnlyTee removals and runs PurgeAction::Namespace.
    5. -
  5. Local data + signing keys are deleted (plus the AES group keys (PR #2776)); the node unsubscribes from the gossip topic.
    6. -
  6. If interrupted, the pending-self-purge marker + startup reconcile sweep finish the purge after restart.
    7. -
-
-
-

-Cross-repo contract. The disable decision and the assignment feed are owned by mdma; the -leave trigger lives in the mero-tee sidecar; core owns the leave op, the cascade, and the purge. -Changing the shape of any of these requires coordinating across the three repos. -

-
- - -
-

Operator notes

-
    -
  • Purge observability. Watch self_purge_failures_total, - self_purge_reconcile_total, and - self_purge_events_dropped_total. A rising failures or dropped-events - counter means purges are not completing cleanly; reconcile counts confirm the startup sweep is - doing its job. (See Metrics Reference.)
    • -
  • MRTD alignment. The node image's measurements and the control plane's - allowed_mrtd allowlist must agree, or every fleet-join fails admission. - After a node-image bump, update the namespace's admission policy before expecting replicas to - join. Compare against the published measurements as in TEE Mode.
    • -
  • Two key families. A replica that can fetch its KMS disk key but cannot decrypt group - data has a group-key problem (delivery/recovery), not a KMS problem — and vice versa. - Diagnose the right family.
    • -
  • Restricted vs Open reads. If a replica reads some subgroups but not others, the gap is - almost always missing per-subgroup admission/keys for Restricted children — the transparent - subgroup admission (PR #2772) is what closes it.
    • -
-
- - -
-

Related

-
    -
  • TEE Mode — the node's own storage/disk KMS key and the - startup attestation flow (distinct from group keys).
    • -
  • Membership & Leave — general leave and eviction - semantics that the fleet leave path builds on.
    • -
  • Auto-Follow — how an admitted replica tracks new contexts and - subgroups, and the namespace-scoped policy rationale.
    • -
  • Glossary — definitions for namespace, group, role, and quote - terms used above.
    • -
  • Producing repos: - mero-tee - (the fleet sidecar and node image) and - mdma (the - control plane that owns entitlement and disable). Their internals are out of scope for core; only - the contract surfaces named on this page are relied upon.
    • -
-
- -
-
- - - diff --git a/architecture/tee-mode.html b/architecture/tee-mode.html deleted file mode 100644 index 1538e3eba8..0000000000 --- a/architecture/tee-mode.html +++ /dev/null @@ -1,126 +0,0 @@ - - - - -TEE Mode — Calimero Core Architecture - - - -
-
-
Home/TEE Mode
-

TEE Mode (storage key / KMS)

-

Trusted Execution Environment operation — the node's storage/disk encryption key

- -
-

Which key this page is about

-

-This page covers the node's storage / disk encryption key obtained from the KMS — -a per-node, MRTD-gated key fetched at startup and used to encrypt the local datastore and -blobstore. It is distinct from the per-group GroupKeyring -encryption keys delivered via KeyDelivery, which protect fleet-HA application -data shared across a group. The two are unrelated mechanisms — do not conflate them. For the -fleet / per-group key story, see TEE Fleet HA. -

-
- -
-

Overview

-

merod in TEE mode runs inside a confidential VM and obtains its storage encryption key from a KMS at startup. Only nodes with a [tee] section perform the KMS key-fetch and attestation flow; other nodes use libp2p as usual without KMS.

-

Deployment on GCP, Phala, and related images is documented in mero-tee.

-
- -
-

KMS provider

-

Supported path today: Phala Cloud — mero-kms-phala from the mero-tee repository (merod in a Phala CVM with dstack).

-
- -
-

Configuration

-

Run merod … init first, then add KMS settings (CLI config or edit config.toml):

-
[tee]
-[tee.kms.phala]
-url = "https://<kms-host>:8443/"
-
-[tee.kms.phala.attestation]
-enabled = true
-accept_mock = false
-allowed_tcb_statuses = ["UpToDate"]
-allowed_mrtd = ["<trusted_kms_mrtd_hex>"]
-allowed_rtmr0 = ["…"] … allowed_rtmr1..3 similarly in production
-

Optional TLS/mTLS paths under [tee.kms.phala.tls]; external policy via policy_json_path / binding_b64.

-
- -
-

Release policy environment variables

-

When set, merod fetches attestation policy from the official release and verifies the KMS via POST /attest before key requests. Precedence:

-

MERO_KMS_RELEASE_TAG > MERO_KMS_VERSION > MERO_TEE_VERSION

-

(MERO_KMS_RELEASE_TAG accepts mero-kms-vX.Y.Z or X.Y.Z.) If a version is set and policy fetch fails, startup aborts (fail closed). Air-gapped: USE_ENV_POLICY=true with a verified local policy script.

-
- -
-

Startup flow

-
    -
  1. If attestation is enabled: KMS self-attestation (POST /attest) — quote validity, nonce/binding in reportData, policy for TCB, MRTD, RTMR0–3.
    2. -
  2. Request a challenge from the KMS.
    3. -
  3. Build a TDX quote with the challenge nonce and peer ID hash.
    4. -
  4. Sign with the node identity key from config.toml.
    5. -
  5. Submit to the KMS and receive the storage key for datastore/blobstore.
    6. -
-
- -
-

GCP operators: MRTD verification

-

Compare deployed nodes to published measurements using published-mrtds.json from mero-tee releases, e.g.:

-
https://github.com/calimero-network/mero-tee/releases/download/<X.Y.Z>/published-mrtds.json
-

Step-by-step: mero-tee verify-mrtd.

-
- -
-

Deployment model

-
    -
  • Isolate each merod release line with its own KMS; pin attestation inputs to a specific signed mero-tee release tag (do not auto-follow latest).
    • -
  • Avoid mixing old/new TEE cohorts against one KMS unless deliberately governed.
    • -
-
- -
-Troubleshooting -
-
    -
  • Key fetch fails — Confirm tee.kms.phala.url is reachable; KMS /health returns 200; KMS can access the TEE attestation socket (e.g. dstack on Phala).
    • -
  • KMS rejects attestation — Dev: mock attestation per mero-tee docs. Prod: align measurement policy with mero-kms-phala.
    • -
  • Mock vs production — accept_mock and MEROD_ALLOW_MOCK_KMS_ATTESTATION are for development only. The generate_mock_attestation(report_data) function in calimero-tee-attestation is a public, OS-independent API that produces a structurally valid but cryptographically invalid quote; it is accepted only by KMS policies with accept_mock = true and must never be used in production.
    • -
  • Local fleet testing without TDX hardware — Pass merod run --mock-tee to enable the dev/test-only mock-TEE harness. The node produces and accepts a synthetic attestation quote (via generate_mock_attestation) through the real fleet-join / announce / admit path, allowing full TEE/HA fleet lifecycle testing on a laptop. This mode is disabled by default and will refuse to start if a real KMS attestation endpoint is configured — it must never be used in production.
    • -
-
-
- -
-

Known fixes (PR #2855)

-
    -
  • Buffered ContextRegistered re-driven after GroupCreated — A buffered encrypted ContextRegistered event that arrived before its subgroup's GroupCreated had applied was previously stranded. It is now re-driven once GroupCreated applies, ensuring the context is correctly registered (fixes #2848).
    • -
  • Fleet-join first announce no longer 500s on empty mesh — fleet-join's initial announce into an empty gossipsub mesh previously returned HTTP 500. This is now handled gracefully (fixes #2491).
    • -
  • KeyDelivery-bootstrapped replica seeds CAN_JOIN_OPEN_SUBGROUPS — A TEE replica bootstrapped via a KeyDelivery seed now correctly seeds the CAN_JOIN_OPEN_SUBGROUPS capability so it can replicate Open subgroups via inheritance, matching the behaviour of nodes that joined through the normal fleet path.
    • -
-
- -
-

Attestation flow

-
    -
  1. merod starts and detects a [tee] configuration block.
    2. -
  2. Requests KMS self-attestation (POST /attest) — verifies the KMS quote, nonce, binding, and MRTD/RTMR policy.
    3. -
  3. Requests a challenge from the KMS.
    4. -
  4. Generates a TDX attestation quote embedding the challenge nonce and peer ID hash.
    5. -
  5. Signs the payload with the node identity key from config.toml.
    6. -
  6. Submits the signed request to the KMS.
    7. -
  7. Receives the storage encryption key (AES-256).
    8. -
  8. Uses that key for datastore and blobstore encryption.
    9. -
-
- -
-
- - - diff --git a/architecture/unified-causal-log-cutover-plan.html b/architecture/unified-causal-log-cutover-plan.html deleted file mode 100644 index 955a7122af..0000000000 --- a/architecture/unified-causal-log-cutover-plan.html +++ /dev/null @@ -1,212 +0,0 @@ - - - - -Unified Causal Log Cutover Plan — Calimero Core Architecture - - - -
-
-
Home/Unified Causal Log Cutover Plan
- -

Unified Causal Log Cutover Plan

-

C0 shadow strategy, wire protocol, divergence marker, and the e2e canary gate that gates C1

- -
-

Overview

-

The unified-causal-log cutover is staged across multiple milestones, conventionally labelled C0, C1, and beyond. This page documents the C0 shadow plan — the phase in which the new causal log is computed and propagated in parallel with the existing log without replacing it. No reads are served from the shadow log during C0; its sole purpose is to validate correctness before any irreversible cutover step is taken.

-

The design doc at docs/design/unified-causal-log-cutover-plan.md has been updated with a dedicated C0 section. The material here reflects that section and records the rationale so maintainers can judge when it is safe to proceed to C1.

-
- -
-

What C0 does

-

During C0 every node computes the unified causal log alongside the legacy log. The two logs are kept strictly separate:

-
    -
  • The legacy log continues to drive all reads, consensus decisions, and state machine transitions.
    • -
  • The shadow log is computed from the same sequence of events and stored in a parallel structure that is never exposed to application logic.
    • -
  • At the end of each epoch boundary the node compares the root hash of the shadow log against the root hash transmitted by peers and records any divergence via the scope_root_shadow_divergence marker (described below).
    • -
-

C0 is considered in-progress until the e2e canary gate defined in this document passes all required conditions. C1 must not begin before that gate clears.

-
- -
-

Shadow log computation method

-

The shadow log is built by replaying every committed event through the new causal-log assembler while the legacy path runs concurrently. The key properties of this computation are:

-
    -
  1. Deterministic ordering — events are fed to the assembler in the same total order that the consensus layer delivers them, so the resulting DAG is identical on every correctly-behaving replica.
    2. -
  2. Incremental hashing — each new event is appended to the shadow log using the same incremental Merkle construction that C1 will use for the live log, ensuring the hash function under test is the production one.
    3. -
  3. Isolation — the assembler writes to a separate storage column family (or equivalent namespace) tagged shadow:. Reads from application state never touch this namespace during C0.
    4. -
  4. No side effects — the shadow assembler is read-only with respect to consensus; it neither proposes nor votes and does not influence leader election.
    5. -
-

The computation is gated behind a feature flag unified_causal_log_shadow that defaults to on for all nodes running the C0 release and can be disabled per-node for performance triage without affecting correctness of the live path.

-
- -
-

Wire protocol additions for C0

-

C0 introduces a small extension to the gossip envelope so that peers can exchange shadow root hashes without polluting the legacy log protocol:

-
// Protobuf excerpt — added in C0
-message ShadowRootAnnouncement {
-  // Epoch sequence number this root covers.
-  uint64 epoch_seq = 1;
-
-  // Scope identifier — matches the existing ScopeId type.
-  bytes scope_id = 2;
-
-  // Merkle root of the shadow causal log at epoch_seq.
-  bytes shadow_root = 3;
-
-  // Sender's node identity for divergence attribution.
-  bytes sender_node_id = 4;
-}
-

Nodes broadcast a ShadowRootAnnouncement immediately after sealing each epoch. Receipt of an announcement triggers a local comparison:

-
    -
  • If the received shadow_root matches the locally computed root, no action is taken.
    • -
  • If they differ, the node emits a scope_root_shadow_divergence structured log event and increments the corresponding Prometheus counter.
    • -
-

The announcement messages are not stored in the legacy log and carry no consensus weight. They are fire-and-forget gossip with no acknowledgement requirement during C0.

-
- -
-

The scope_root_shadow_divergence marker

-

The scope_root_shadow_divergence marker is the primary observability signal for C0. It is emitted as a structured log event at WARN level and exported as a Prometheus counter with the following labels:

-
    -
  • scope_id — the affected scope.
    • -
  • epoch_seq — the epoch at which the divergence was first observed.
    • -
  • local_root — hex-encoded local shadow root (first 8 bytes for brevity).
    • -
  • remote_root — hex-encoded remote shadow root (first 8 bytes).
    • -
  • remote_node_id — the peer whose announcement triggered the comparison.
    • -
-
// Example structured log line
-{
-  "level": "WARN",
-  "target": "calimero_causal_log::shadow",
-  "message": "scope_root_shadow_divergence",
-  "scope_id": "abc123",
-  "epoch_seq": 4821,
-  "local_root": "d4e8f1a2...",
-  "remote_root": "9c3b77e0...",
-  "remote_node_id": "node-07"
-}
-

A divergence does not halt the node or trigger any recovery during C0. It is purely diagnostic. The canary gate (see below) aggregates these counters across the test fleet to determine whether the shadow implementation is ready for C1.

-
- -
-

Hash-neutral governance rotation — the critical canary scenario

-

A hash-neutral rotation is a governance event that changes the membership or key material of a scope without altering the application state hash. Examples include:

-
    -
  • Rotating a node's signing key while keeping its identity stable.
    • -
  • Adding or removing a passive observer that holds no voting weight.
    • -
  • Updating scope metadata fields that are excluded from the state Merkle tree.
    • -
-

Hash-neutral rotations are the hardest class of event for the shadow log to handle correctly because the legacy log's root hash does not change across the rotation, but the unified causal log's root must change — it encodes governance events explicitly in its DAG structure.

-

The canary gate therefore requires the test harness to inject at least one hash-neutral governance rotation per test run and assert that the shadow divergence counter fires on nodes that have not yet processed the rotation and clears once all nodes converge. A canary that never fires on a rotation indicates the shadow log is silently ignoring governance events, which would be a correctness defect.

-
- -
-

The e2e canary gate — pass conditions

-

The canary gate is an end-to-end test suite that must reach a steady-state pass before C1 is permitted to begin. The gate has two distinct pass conditions that must both be satisfied:

-

Condition 1 — zero false positives at steady state

-

After a fully-converged cluster has processed all pending events and all nodes agree on epoch boundaries, the scope_root_shadow_divergence counter must read zero across all nodes for a sustained observation window (currently defined as 30 consecutive epochs with no divergence events).

-
    -
  • Transient divergences during event processing are expected and do not fail this condition as long as they resolve before the window closes.
    • -
  • A persistent non-zero counter at steady state indicates a non-determinism bug in the shadow assembler and must be fixed before proceeding.
    • -
-

Condition 2 — fires on hash-neutral governance rotation

-

The test harness must confirm that injecting a hash-neutral governance rotation causes the divergence counter to increment on at least one node during the convergence window, and that the counter returns to zero after all nodes process the rotation.

-
    -
  • This condition validates that the shadow log is sensitive to governance events that the legacy hash does not capture.
    • -
  • A canary run in which no divergence fires during a rotation is treated as a gate failure, not a pass, because it indicates the event was silently dropped.
    • -
-

Both conditions must pass on three consecutive canary runs before C1 work is scheduled.

-
- -
-

Gate failure modes and remediation

-

The following table describes the failure modes the canary gate is designed to catch and the expected remediation path for each:

-
    -
  • Persistent divergence at steady state — indicates non-deterministic event ordering or a hash function mismatch between the local assembler and peers. Remediation: audit the event ingestion pipeline for ordering guarantees and verify the assembler's hashing matches the wire spec.
    • -
  • No divergence on hash-neutral rotation — indicates governance events are not being fed to the shadow assembler. Remediation: verify the governance event bus is wired to the shadow path behind the feature flag.
    • -
  • Divergence that never clears — indicates a convergence bug where the shadow DAG cannot reconcile after a fork. Remediation: inspect the merge logic in the causal-log assembler and check for missing causal links in the gossip path.
    • -
  • Counter fires on non-divergent epochs — indicates a clock or epoch-boundary mismatch causing nodes to compare roots from different epochs. Remediation: validate that epoch_seq in ShadowRootAnnouncement is derived from the same consensus clock on all nodes.
    • -
-
- -
-

C0 to C1 transition criteria — summary

-

A maintainer may schedule C1 work only when all of the following are true:

-
    -
  1. The e2e canary gate has passed three consecutive runs with zero false positives at steady state.
    2. -
  2. The canary has fired and cleared correctly on a hash-neutral governance rotation in each of those runs.
    3. -
  3. No open issues are tagged c0-blocker in the issue tracker.
    4. -
  4. The scope_root_shadow_divergence Prometheus counter has been reviewed by at least one maintainer who is not the author of the shadow assembler.
    5. -
  5. The design doc section for C1 has been written and reviewed, incorporating any lessons learned during C0.
    6. -
-

Do not begin C1 speculatively. The shadow plan exists precisely to surface correctness issues at zero operational risk. Cutting over before the gate passes converts a correctable bug into a potential data-integrity incident.

-
- -
-

Feature flag lifecycle

-

The unified_causal_log_shadow feature flag introduced in C0 follows the standard flag lifecycle for this codebase:

-
    -
  • C0 release — flag defaults to on; can be disabled per-node via config for performance triage.
    • -
  • C1 release — flag is removed; shadow path is promoted to the live path; the legacy path becomes the shadow for a brief overlap window (the C1 shadow plan will describe this separately).
    • -
  • Never ship C1 with the C0 flag still present — having two shadow flags simultaneously creates an untestable combination space.
    • -
-

Config key for the flag:

-
# calimero-node.toml
-[experimental]
-unified_causal_log_shadow = true   # default; set false to disable C0 shadow
-
- -
-

Relationship to the existing design doc

-

The canonical design document lives at docs/design/unified-causal-log-cutover-plan.md in the repository root. The C0 section added by this change is the authoritative specification; this architecture page is a derived summary intended for browsing alongside other architecture pages.

-

If this page and the design doc disagree, the design doc takes precedence. File a documentation bug to reconcile the two.

-
    -
  • docs/design/unified-causal-log-cutover-plan.md — authoritative cutover plan; C0 section is in-progress.
    • -
  • crates/causal_log/src/shadow/assembler.rs — shadow assembler implementation.
    • -
  • crates/causal_log/src/shadow/gossip.rs — ShadowRootAnnouncement broadcast logic.
    • -
  • crates/causal_log/src/shadow/divergence.rs — scope_root_shadow_divergence marker emission.
    • -
  • e2e/causal_log_canary/ — end-to-end canary test suite including the hash-neutral rotation scenario.
    • -
-
- -
-

C1 incremental landing plan — C1a and C1b

-

C1 is split into two slices to reduce the blast radius of the live-log cutover and to defer subsystems that carry their own staleness semantics.

-

C1a (current PR)

-

C1a lands the HashComparison end-of-session verdict: the unified causal log is promoted to the live path for the HashComparison comparison site, which produces a verdict at the end of each session. This is the lowest-risk cutover point because session boundaries are already natural quiesce points with no in-flight reads.

-
    -
  • The C0 canary gate (three consecutive passing runs, as described above) is the merge gate for C1a. C1a must not land until that gate clears.
    • -
  • The unified_causal_log_shadow feature flag is removed in C1a; the shadow path is promoted to live.
    • -
  • All other comparison sites remain on the legacy log after C1a lands.
    • -
-

C1b (completed 2026-06-24)

-

C1b updated the remaining comparison sites that were intentionally left on the legacy log in C1a. The LevelWise advisory handshake-root verdict was folded onto scope_root as planned. Two sites were deliberately excluded from the scope_root migration and remain entity-root-based by explicit design decision:

-
    -
  • Snapshot streaming-integrity boundary (snapshot.rs) — kept entity-root-based because the snapshot boundary check validates the integrity of streamed entity data at ingestion time. Threading scope_root through this path would cause the check to accept or reject a snapshot based on causal-log state that may not yet reflect the snapshot's own epoch, silently masking ingestion of a snapshot built against a stale causal root.
    • -
  • Protocol selector 'roots match → skip sync' arm — kept entity-root-based because this no-sync fast path compares entity roots as a proxy for whether a full sync can be skipped. Switching it to scope_root would conflate governance-event divergence (which the unified causal log captures) with data-state divergence (which drives the sync decision), potentially causing unnecessary syncs or, worse, skipping a sync when entity state has genuinely diverged.
    • -
-

These exclusions are permanent for C1b. If a future milestone determines that either site should be moved to scope_root, that work requires its own design pass and canary gate.

-

C1b also introduces the reversed-shadow arrangement: the legacy log becomes the shadow checked against the now-live unified log. The existing scope_root_shadow_divergence marker infrastructure was evaluated for polarity changes as part of the C1b design doc.

-
- -
-

Open questions for C1 planning

-

The following questions are deliberately left open until the C0 canary gate passes. They should be resolved as part of writing the C1 design doc section:

-
    -
  • Read cutover ordering — should reads switch to the unified log atomically at an epoch boundary, or is a gradual per-scope migration safer?
    • -
  • Legacy log retention — how long must the legacy log be retained after C1 for rollback purposes, and what is the storage cost on large scopes?
    • -
  • C1 shadow direction — during C1 the roles reverse: the unified log becomes live and the legacy log becomes the shadow. Does the existing divergence marker infrastructure need changes to support this reversed polarity?
    • -
  • Governance event backfill — nodes that were offline during C0 will have gaps in their shadow log. Does C1 require a backfill protocol, or is a fresh shadow-log start at C1 cutover acceptable?
    • -
  • C1b comparison-site ordering — resolved: LevelWise advisory verdict was migrated to scope_root; snapshot boundary verdict and protocol selector converged-decision were explicitly kept entity-root-based (see C1b section above).
    • -
-

None of these questions need to be answered before C0 is declared complete. They are recorded here so they are not forgotten when C1 planning begins.

-
- -
-
- - - \ No newline at end of file diff --git a/architecture/wire-protocol.html b/architecture/wire-protocol.html deleted file mode 100644 index 02a54fc8f6..0000000000 --- a/architecture/wire-protocol.html +++ /dev/null @@ -1,971 +0,0 @@ - - - - -Wire Protocol — Calimero Core Architecture - - - - -
-
-
Home/Wire Protocol
- -

Wire Protocol

-

Every message type with field-level documentation

- -
-
10
gossipsub variants
-
13
stream init types
-
24
GroupOp variants
-
- -
-
gossipsub pub/sub broadcast
-
stream point-to-point
-
req/resp request-response
-
internal actor messages
-
- - -
-

BroadcastMessage gossipsub 11 variants

-

Top-level enum for all messages published via gossipsub topics. Each variant is Borsh-serialized before being sent as a gossipsub message payload. Context topics use context/<hex>, group topics use group/<hex>, namespace topics use ns/<hex> (hex of the 32-byte namespace id, which is the root group id).

- -
- StateDelta — application state replication -
-

Carries a WASM execution result (state diff) to all peers in a context. Published on context gossipsub topics after a successful method execution and state commit.

-
struct StateDelta { - context_id: ContextId, // target context - author_id: PublicKey, // who executed the method - delta_id: [u8; 32], // content-addressed hash of this delta - parent_ids: Vec<[u8; 32]>, // DAG ancestry (current heads at produce time) - hlc: HybridTimestamp, // hybrid logical clock for causal ordering - root_hash: Hash, // Merkle root after applying this delta - artifact: Cow<[u8]>, // the actual state diff payload (opaque bytes) - nonce: Nonce, // deduplication / replay protection - events: Option<Cow<[u8]>>, // optional WASM-emitted events - governance_epoch: Vec<[u8; 32]>, // governance DAG heads at delta time (future: stale check) -}
-
Borsh serialization · delta_id = SHA-256(artifact + metadata) · parent_ids capped by DAG head limit
-
-
- -
- HashHeartbeat — context state comparison -
-

Periodic heartbeat published on context topics. Peers compare root_hash and dag_heads to detect divergence and trigger sync.

-
struct HashHeartbeat { - context_id: ContextId, // which context - root_hash: Hash, // current Merkle root of context state - dag_heads: Vec<[u8; 32]>, // current DAG head delta IDs -}
-
-
- -
- SpecializedNodeDiscovery — peer role announcement -
-
struct SpecializedNodeDiscovery { - nonce: Nonce, // unique per announcement (dedup) - node_type: NodeType, // e.g. Sequencer, Indexer, Relay -}
-
-
- -
- SpecializedNodeJoinConfirmation — role join ack -
-
struct SpecializedNodeJoinConfirmation { - nonce: Nonce, // matches the discovery nonce -}
-
-
- -
- TeeAttestationAnnounce — fleet-join attestation -
-

A TEE fleet node announces its TDX attestation to join a namespace as a ReadOnlyTee member. Published on the namespace topic ns/<hex(namespace_id)> and re-announced for up to ~30s until a verifier admits it. The verifier runs verify_attestation, then namespace-scoped admit_tee_node (policy check + quote-hash replay guard). See TEE Fleet HA and Sequence Diagrams.

-
struct TeeAttestationAnnounce { - quote_bytes: Vec<u8>, // TDX attestation quote bytes - public_key: PublicKey, // announcing node's identity public key - nonce: [u8; 32], // fresh random 32 bytes; report_data = nonce(32) || Sha256(pubkey) - node_type: SpecializedNodeType, // specialized node type (e.g. ReadOnlyTee) -}
-
Published on ns/<hex(namespace_id)> · drives fleet admission in crates/server/src/admin/handlers/tee/fleet_join.rs
-
-
- -
- GroupMutationNotification — group change signal -
-

Lightweight notification that a group was mutated. Does not contain the full op — peers use this to decide whether to fetch details or update caches.

-
struct GroupMutationNotification { - group_id: GroupId, // affected group - mutation_kind: GroupMutationKind, // what kind of mutation (see below) -}
-
-
- -
- SignedGroupOpV1 — governance operation (v1 wire) -
-

Carries a Borsh-serialized SignedGroupOp as an opaque byte vector. Published on group gossipsub topics. Receivers deserialize, verify signature, and apply.

-
struct SignedGroupOpV1 { - payload: Vec<u8>, // borsh(SignedGroupOp) — opaque wire format -}
-
Inner payload is borsh(SignedGroupOp) — see SignedGroupOp schema section below for full structure
-
-
- -
- GroupGovernanceDelta — governance DAG delta -
-

Alternative governance replication format used by the DAG sync layer. Carries the same data as SignedGroupOpV1 but with explicit DAG metadata for the governance DAG.

-
struct GroupGovernanceDelta { - group_id: GroupId, // target group - delta_id: [u8; 32], // content hash of this governance delta - parent_ids: Vec<[u8; 32]>, // governance DAG parents - payload: Vec<u8>, // borsh(SignedGroupOp) -}
-
-
- -
- GroupStateHeartbeat — group governance comparison -
-

Periodic heartbeat for governance DAG state. Published every 30s on group topics. Peers compare dag_heads to detect missing ops and trigger governance catch-up.

-
struct GroupStateHeartbeat { - group_id: GroupId, // which group - dag_heads: Vec<[u8; 32]>, // current governance DAG head op hashes - member_count: u64, // current member count (quick sanity check) -}
-
-
- -
- NamespaceGovernanceDelta — namespace governance op -
-

Carries a SignedNamespaceOp on namespace gossipsub topics. Published when a namespace governance operation is created (group creation, member join, key delivery, etc.). Receivers verify the signature, apply to the namespace DAG, and may trigger key delivery.

-
struct NamespaceGovernanceDelta { - namespace_id: [u8; 32], // target namespace (root group) - payload: Vec<u8>, // borsh(SignedNamespaceOp) -}
-
Payload size capped by MAX_SIGNED_GROUP_OP_PAYLOAD_BYTES · Published on ns/<hex> topic
-
-
- -
- NamespaceStateHeartbeat — namespace governance comparison -
-

Periodic heartbeat for namespace governance DAG state. Published every 30s on namespace topics. Peers compare dag_heads (capped at 256) to detect missing ops. Split logic: gossipsub for ops the peer needs, P2P stream for backfill of ops we need.

-
struct NamespaceStateHeartbeat { - namespace_id: [u8; 32], // which namespace - dag_heads: Vec<[u8; 32]>, // current namespace governance DAG heads -}
-
-
-
- - -
-

GroupMutationKind 14 variants

-

Enum carried inside GroupMutationNotification. Describes what kind of group mutation occurred, allowing receivers to react selectively without deserializing the full op.

- -
enum GroupMutationKind { - MembersAdded, // one or more members added - MembersRemoved, // one or more members removed (cascade) - Upgraded, // group upgrade policy or application changed - Deleted, // group deleted entirely - ContextDetached, // context unbound from group - SettingsUpdated, // general group settings changed - MemberRoleUpdated, // member role changed (admin/member) - ContextAttached, // new context bound to group - ContextAliasSet, // human-readable context alias changed - MemberCapabilitySet, // per-member capability grant/revoke - DefaultCapabilitiesSet, // default capabilities for new members changed - MemberAliasSet, // human-readable member alias changed - GroupAliasSet, // human-readable group alias changed - ContextRegistered, // new context registered to group -}
-
- - -
-

StreamMessage stream 4 variants

-

Framing protocol for all point-to-point stream communication. Each stream begins with an Init message, followed by zero or more Message frames, and can terminate with OpaqueError or — as of #2422 — the typed NotMaterialized variant for benign non-following peers.

- -
enum StreamMessage { - Init { - payload: InitPayload, // identifies the stream type and initial data - }, - Message { - payload: MessagePayload, // subsequent data frames - }, - OpaqueError, // unit variant — signals failure without leaking node state - NotMaterialized,// unit variant (#2422) — typed responder reply when - // dialer_verified=true but context not materialised within - // the join-race window. Initiator's SyncManager downcasts to - // PeerNotMaterialized and treats as benign — no failure_count - // increment, no exponential backoff. -}
-
Borsh serialization · length-prefixed framing over libp2p streams (QUIC/TCP)
-
- - -
-

InitPayload stream init 14 variants

-

Sent as the first message on a new stream. Determines the protocol/purpose of the stream and carries initial handshake data.

- -
- BlobShare — transfer application binary -
-
struct BlobShare { - blob_id: BlobId, // identifier of the WASM blob - blob: Cow<[u8]>, // the actual blob bytes -}
-
-
- -
- KeyShare — exchange encryption key -
-
struct KeyShare { - context_id: ContextId, // target context - holder_id: PublicKey, // who holds the key - encrypted_key: Cow<[u8]>, // encrypted store key material -}
-
-
- -
- DeltaRequest — fetch specific delta -
-
struct DeltaRequest { - context_id: ContextId, // which context - delta_id: [u8; 32], // content hash of the requested delta -}
-
-
- -
- DagHeadsRequest — get current heads -
-
struct DagHeadsRequest { - context_id: ContextId, // which context's DAG heads to return -}
-
-
- -
- SnapshotBoundaryRequest — negotiate snapshot range -
-
struct SnapshotBoundaryRequest { - context_id: ContextId, // target context - root_hash: Hash, // requester's current root (may be empty) -}
-
-
- -
- SnapshotStreamRequest — start full snapshot transfer -
-
struct SnapshotStreamRequest { - context_id: ContextId, // which context to snapshot -}
-
-
- -
- TreeNodeRequest — fetch Merkle tree node -
-
struct TreeNodeRequest { - context_id: ContextId, // target context - node_hash: Hash, // hash of the tree node to fetch -}
-
-
- -
- LevelWiseRequest — level-wise tree comparison -
-
struct LevelWiseRequest { - context_id: ContextId, // target context - level: u32, // which tree level - hashes: Vec<Hash>, // node hashes at this level -}
-
-
- -
- EntityPush — push entities to peer -
-
struct EntityPush { - context_id: ContextId, // target context - entities: Vec<u8>, // serialized entity data -}
-
-
- -
- GroupDeltaRequest — fetch governance op -
-

Requests a specific governance operation from the peer's op log. Used during governance catch-up when heartbeat reveals missing ops.

-
struct GroupDeltaRequest { - group_id: GroupId, // which group - delta_id: [u8; 32], // content hash of the requested governance op -}
-
-
- -
- NamespaceBackfillRequest — fetch namespace governance ops -
-

Requests one or more namespace governance operations by delta ID. Opened as a P2P stream when NamespaceStateHeartbeat comparison reveals ops we are missing. The peer responds with NamespaceBackfillResponse messages.

-
struct NamespaceBackfillRequest { - namespace_id: [u8; 32], // which namespace - delta_ids: Vec<[u8; 32]>, // content hashes of requested ops -}
-
-
- -
- NamespaceJoinRequest — join namespace via invitation -
-

Sent by a joiner to an existing namespace member after subscribing to the namespace topic and forming a mesh. The peer responds with the group key (wrapped via ECDH), a snapshot of governance ops, and a list of context IDs to auto-join.

-
struct NamespaceJoinRequest { - namespace_id: [u8; 32], // which namespace - invitation: SignedGroupOpenInvitation, // proof of valid invitation - joiner_pk: PublicKey, // joiner's namespace identity public key -}
-
-
- -
- GroupKeyRequest — pull-based group-key recovery -
-

Pull-based recovery for a group key the requester is already an admitted member of but does not yet hold locally — the durable replacement for the old on-DAG KeyDelivery push. The joiner sends this each sync round (via recover_missing_group_keys) until it has the key. The responder validates that requester_public_key is a current member of group_id, then ECDH-wraps the group key and replies with GroupKeyResponse.

-
struct GroupKeyRequest { - namespace_id: [u8; 32], // which namespace - group_id: [u8; 32], // the group whose key is being recovered - requester_public_key: PublicKey, // namespace identity (membership check + ECDH recipient) -}
-
Appended at the tail of InitPayload (existing discriminants unchanged) · responder in crates/node/src/sync/manager/namespace_sync.rs
-
-
-
- - -
-

MessagePayload stream data 15 variants

-

Subsequent data frames sent after the initial Init on a stream. The variant matches the stream type established by InitPayload.

- -
- DeltaResponse — delta fetch result -
-
struct DeltaResponse { - delta: Option<StateDelta>, // the requested delta, or None if not found -}
-
-
- -
- DagHeadsResponse — current DAG heads -
-
struct DagHeadsResponse { - heads: Vec<[u8; 32]>, // current DAG head delta IDs -}
-
-
- -
- SnapshotBoundaryResponse — snapshot range info -
-
struct SnapshotBoundaryResponse { - root_hash: Hash, // sender's current root - dag_heads: Vec<[u8; 32]>, // sender's current DAG heads - entity_count: u64, // estimated entity count for sizing -}
-
-
- -
- SnapshotStreamData — snapshot chunk -
-
struct SnapshotStreamData { - chunk: Cow<[u8]>, // serialized state chunk - is_last: bool, // true if this is the final chunk -}
-
-
- -
- SnapshotStreamEnd — snapshot complete -
-
struct SnapshotStreamEnd { - root_hash: Hash, // final root hash for verification - dag_heads: Vec<[u8; 32]>, // DAG heads at snapshot time -}
-
-
- -
- TreeNodeResponse — Merkle tree node data -
-
struct TreeNodeResponse { - node: Option<TreeNode>, // the tree node, or None if not found -}
-
-
- -
- LevelWiseResponse — level comparison result -
-
struct LevelWiseResponse { - differing: Vec<Hash>, // hashes that differ at this level - missing: Vec<Hash>, // hashes the responder doesn't have -}
-
-
- -
- EntityPushAck — entity push acknowledgment -
-
struct EntityPushAck { - accepted: u64, // number of entities accepted - rejected: u64, // number of entities rejected -}
-
-
- -
- GroupDeltaResponse — governance op data -
-
struct GroupDeltaResponse { - payload: Option<Vec<u8>>, // borsh(SignedGroupOp), or None if not found -}
-
-
- -
- NamespaceBackfillResponse — namespace governance ops -
-
struct NamespaceBackfillResponse { - deltas: Vec<Vec<u8>>, // borsh(SignedNamespaceOp) for each requested delta_id -}
-
-
- -
- BlobShareAck — blob received confirmation -
-
struct BlobShareAck { - blob_id: BlobId, // confirmed blob - accepted: bool, // true if stored, false if already had it -}
-
-
- -
- KeyShareAck — key share confirmation -
-
struct KeyShareAck { - accepted: bool, // true if key was stored -}
-
-
- -
- ContextStateCheckResponse — state check result -
-
struct ContextStateCheckResponse { - root_hash: Hash, // peer's current root hash - in_sync: bool, // whether peer considers itself in sync -}
-
-
- -
- StateSyncChunk — incremental state sync data -
-
struct StateSyncChunk { - chunk_id: u64, // sequential chunk index - data: Cow<[u8]>, // serialized entities - is_last: bool, // final chunk marker -}
-
-
- -
- StateSyncComplete — state sync finalization -
-
struct StateSyncComplete { - root_hash: Hash, // final root hash after sync - dag_heads: Vec<[u8; 32]>, // DAG heads after sync - total_entities: u64, // total entities transferred -}
-
-
- -
- GroupKeyResponse — pull-based group-key delivery -
-

Response to GroupKeyRequest. Carries the ECDH-wrapped group-key envelope; key_envelope_bytes is empty when the responder doesn't hold the key or the requester isn't a member (the joiner retries another peer next sync round). responder_identity is the trust anchor a keyless bootstrap joiner uses to seed the namespace admin when it receives the root-group key without an invitation (replaces the old KeyDelivery-signer anchor).

-
struct GroupKeyResponse { - key_envelope_bytes: Vec<u8>, // borsh(KeyEnvelope), ECDH-wrapped; empty ⇒ no key - responder_identity: PublicKey, // responder's namespace identity (the wrap sender) -}
-
Appended at the tail of MessagePayload (existing discriminants unchanged) · pairs with InitPayload::GroupKeyRequest
-
-
-
- - -
-

GroupOp gossipsub 20 variants

-

The actual governance operation inside a SignedGroupOp. Each variant represents a specific group mutation. Applied deterministically on all nodes for convergence.

- -
- Member Management — 9 variants -
-
MemberAdded { - member_id: PublicKey, // Ed25519 public key of new member - role: MemberRole, // Admin | Member - capabilities: Capabilities, // initial capability set -} - -MemberRemoved { - member_id: PublicKey, // triggers cascade removal from all contexts -} - -MemberRoleSet { - member_id: PublicKey, // target member - role: MemberRole, // new role -} - -MemberCapabilitySet { - member_id: PublicKey, // target member - capabilities: Capabilities, // updated capability bitmask -} - -DefaultCapabilitiesSet { - capabilities: Capabilities, // default for new members joining -} - -JoinWithInvitationClaim { - claim: InvitationClaim, // signed proof of valid invitation -} - -SubgroupCreated { - child_group_id: [u8; 32], // link a child group under this group -} - -SubgroupRemoved { - child_group_id: [u8; 32], // unlink a child group from its parent -} - -
-
-
- -
- Context Lifecycle — 3 variants -
-
ContextRegistered { - context_id: ContextId, // new context identifier - application_id: Option<ApplicationId>, // optional initial WASM app -} - -ContextDetached { - context_id: ContextId, // context to unbind from group -} - -// MemberJoinedContext — removed (context membership is now implicit from group membership) -// MemberLeftContext — removed (context membership is now implicit from group membership) - -TargetApplicationSet { - application_id: ApplicationId, // default app for new contexts in this group -}
-
-
- -
- Access Control — 2 variants -
-
ContextCapabilityGranted { - context_id: ContextId, // target context - member_id: PublicKey, // target member - capability: Capability, // specific capability granted -} - -ContextCapabilityRevoked { - context_id: ContextId, // target context - member_id: PublicKey, // target member - capability: Capability, // specific capability revoked -}
-
-
- -
- Aliases — 3 variants -
-
ContextAliasSet { - context_id: ContextId, // target context - alias: Option<String>, // human-readable name (None to clear) -} - -MemberAliasSet { - member_id: PublicKey, // target member - alias: Option<String>, // human-readable name (None to clear) -} - -GroupAliasSet { - alias: Option<String>, // human-readable group name (None to clear) -}
-
-
- -
- Group Lifecycle & Special — 4 variants -
-
GroupDelete { - // no fields — deletes the entire group and all bindings -} - -GroupMigrationSet { - migration: MigrationConfig, // migration parameters -} - -UpgradePolicySet { - policy: UpgradePolicy, // how application upgrades propagate -} - -Noop { - // no fields — merge sentinel, no state change - // used to reconcile divergent DAG heads - // typically has 2+ parent_op_hashes -}
-
-
-
- - -
-

SignedGroupOp Schema v3

-

The complete signed governance operation structure. Each op is content-addressed (SHA-256 of signable bytes), Ed25519 signed, and forms a DAG via parent hashes.

- -
-
-

Wire Layout

-
struct SignedGroupOp { - version: u8, // schema version (currently 3) - group_id: [u8; 32], // target group identifier - parent_op_hashes: Vec<[u8; 32]>, // DAG ancestry (current heads) - state_hash: [u8; 32], // optimistic lock on group state - signer: PublicKey, // Ed25519 public key of signer - nonce: u64, // per-signer monotonic counter - op: GroupOp, // the actual operation - signature: [u8; 64], // Ed25519 signature -}
-
-
-

Signable Subset

-
struct SignableGroupOp { - version: u8, - group_id: [u8; 32], - parent_op_hashes: Vec<[u8; 32]>, - state_hash: [u8; 32], - signer: PublicKey, - nonce: u64, - op: GroupOp, - // signature excluded from signing input -}
-
-
- -
-

Signing Process

-

1. Build signable: Construct SignableGroupOp (all fields except signature).

-

2. Domain prefix: Prepend domain separator b"calimero-signed-group-op-v3" to Borsh-serialized bytes.

-

3. Sign: signature = Ed25519::sign(private_key, domain_prefix ++ borsh(signable))

-

4. Content hash: op_hash = SHA-256(borsh(signable)) — used as the DAG delta ID.

-
- -
-
-

Validation Order

-
    -
  • Signature verification (Ed25519)
    • -
  • Schema version check (must be 3)
    • -
  • Nonce ≥ last seen nonce for signer
    • -
  • state_hash matches current group state (or all-zeros to bypass)
    • -
  • Signer has sufficient role/capabilities for the op
    • -
  • parent_op_hashes.len() ≤ 256
    • -
-
-
-

Key Invariants

-
    -
  • parent_op_hashes — capped at 256 entries
    • -
  • nonce — monotonically increasing per signer
    • -
  • state_hash — all-zeros means skip validation
    • -
  • max_dag_heads — capped at 64 per group
    • -
  • Content hash is deterministic: same signable bytes → same hash
    • -
-
-
-
- - -
-

SignedNamespaceOp Schema namespace governance

-

Namespace-level governance operations. Each namespace has a single DAG shared by all groups within it. Operations are either RootOps (cleartext, visible to all namespace members) or GroupOps (encrypted with the group key, only readable by group members).

- -
-
-

NamespaceOp — RootOp Variants

-

Cleartext operations visible to all namespace members. Used for structural changes and key distribution.

-
GroupCreated { - group_id: [u8; 32], // new group id - parent_id: [u8; 32], // REQUIRED — atomic create+nest (strict-tree invariant) - visibility: GroupVisibility, // Open | Restricted — carried at creation so the group is born-open in one atomic op, eliminating the Restricted-then-flip window and the transient direct ReadOnlyTee row (#2771, wire-breaking) -} - -GroupDeleted { - root_group_id: [u8; 32], // delete target - cascade_group_ids: Vec<[u8; 32]>, // descendants (children-first) - cascade_context_ids: Vec<[u8; 32]>, // all contexts in subtree; peers verify by re-enumeration -} - -GroupReparented { - child_group_id: [u8; 32], // subtree to move - new_parent_id: [u8; 32], // atomic edge swap — replaces old GroupNested + GroupUnnested pair -} - -MemberJoined { - member: PublicKey, // joiner's namespace identity - signed_invitation: SignedInvitation, // proof of valid invitation -} - -MemberJoinedAt { - member: PublicKey, // joiner's namespace identity - signed_invitation: SignedInvitation, // proof of valid invitation - joined_at: u64, // joiner-signed claim time; apply rejects if > invitation expiry. New joins emit this; legacy MemberJoined stays for decode -} - -KeyDelivery { - group_id: [u8; 32], // target group - envelope: KeyEnvelope, // ECDH-wrapped group key for joiner -} - -AdminChanged { new_admin: PublicKey } -PolicyUpdated { policy_bytes: Vec<u8> }
-
-
-

NamespaceOp — GroupOp Variant

-

Encrypted operations only readable by members of the target group. Non-members store an opaque skeleton (preserving causal structure).

-
Group { - group_id: [u8; 32], // target group - key_id: [u8; 32], // which group key was used - encrypted: Vec<u8>, // encrypted inner GroupOp - key_rotation: Option<..>, // optional key rotation -} - -// Inner GroupOp (after decryption): -// MemberAdded, MemberRemoved, MemberRoleSet, -// ContextRegistered, ContextDetached, -// UpgradePolicySet, TargetApplicationSet, -// ... (same variants as GroupOp above)
- -

Key Delivery Flow

-
-// 1. Joiner publishes RootOp::MemberJoined -// 2. Existing member sees MemberJoined on DAG -// 3. Member wraps group key via ECDH: -// shared = ECDH(sender_sk, joiner_pk) -// envelope = encrypt(shared, group_key) -// 4. Member publishes RootOp::KeyDelivery -// 5. Joiner unwraps: ECDH(joiner_sk, sender_pk) -// 6. Joiner can now decrypt GroupOp payloads
-
-
- -
-

Signing Process (Namespace Ops)

-

1. Build SignedNamespaceOp with: namespace_id, parent_hashes, state_hash, nonce, op.

-

2. Sign with the node's namespace identity private key (not the group signing key).

-

3. Content hash delta_id = SHA-256(signable_bytes) for DAG identity.

-
-
- - -
-

NetworkMessage internal 15 variants

-

The Actix message enum sent to the NetworkManager actor. These are internal to the node — never serialized on the wire. Each variant triggers a specific libp2p operation.

- -
enum NetworkMessage { - Publish { topic: TopicHash, data: Vec<u8> }, // gossipsub publish - Subscribe { topic: TopicHash }, // join gossipsub topic - Unsubscribe { topic: TopicHash }, // leave gossipsub topic - OpenStream { peer: PeerId, protocol: StreamProtocol }, // open direct stream - SendMessage { stream_id: StreamId, data: Vec<u8> }, // send on open stream - CloseStream { stream_id: StreamId }, // close stream - ProvideRecord { key: RecordKey, value: Vec<u8> }, // DHT PUT - GetRecord { key: RecordKey }, // DHT GET - GetProviders { key: RecordKey }, // DHT find providers - GetPeers, // list connected peers - GetPeerInfo { peer: PeerId }, // get specific peer metadata - Bootstrap, // trigger Kademlia bootstrap - DialPeer { peer: PeerId, addrs: Vec<Multiaddr> }, // explicit peer dial - GetListenAddrs, // get node's listen addresses - GetMeshPeers { topic: TopicHash }, // gossipsub mesh peers for topic -}
-
- - -
-

NetworkEvent internal 11 variants

-

Events emitted by the NetworkManager actor and handled by NodeManager. These represent observed network activity — gossip messages, peer changes, stream events.

- -
enum NetworkEvent { - GossipMessage { - topic: TopicHash, // which topic it arrived on - source: PeerId, // who sent it - data: Vec<u8>, // raw payload (deserialize as BroadcastMessage) - }, - StreamOpened { - stream_id: StreamId, // unique stream identifier - peer: PeerId, // remote peer - protocol: StreamProtocol, // negotiated protocol - }, - StreamMessage { - stream_id: StreamId, // which stream - data: Vec<u8>, // raw payload (deserialize as StreamMessage) - }, - StreamClosed { - stream_id: StreamId, // which stream was closed - }, - PeerConnected { - peer: PeerId, // newly connected peer - addrs: Vec<Multiaddr>, // peer's observed addresses - }, - PeerDisconnected { - peer: PeerId, // disconnected peer - }, - TopicSubscribed { - topic: TopicHash, // successfully subscribed - }, - TopicUnsubscribed { - topic: TopicHash, // successfully unsubscribed - }, - RecordFound { - key: RecordKey, // DHT key - value: Vec<u8>, // DHT value - }, - ProvidersFound { - key: RecordKey, // DHT key - providers: Vec<PeerId>, // peers providing this record - }, - ListenAddrExpired { - addr: Multiaddr, // address no longer valid - }, -}
- -
These are NOT serialized on the wire — they are Actix messages exchanged internally between NetworkManager and NodeManager actors via LazyRecipient<NodeMessage>
-
- -

Wire-Contract CI & PR Checklist

-

Path-filtered CI and a structured PR checklist enforce wire-contract discipline whenever DTO crates change.

- -

wire-contract.yml — GitHub Actions Workflow

-

The workflow triggers on pull requests that touch any of the following paths:

-
    -
  • crates/server/primitives/**
    • -
  • crates/primitives/**
    • -
  • .github/workflows/wire-contract.yml (self-trigger on workflow edits)
    • -
-

When triggered it runs the wire_fixtures test suite against the modified crates. The toolchain pin uses dtolnay/rust-toolchain@stable — not the archived actions-rs/toolchain@v1.

-
name: wire-contract
-
-on:
-  pull_request:
-    paths:
-      - 'crates/server/primitives/**'
-      - 'crates/primitives/**'
-      - '.github/workflows/wire-contract.yml'
-
-jobs:
-  wire-fixtures:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - run: cargo test -p <dto-crate> wire_fixtures
-

Because the paths filter is narrow the check is skipped on PRs that do not touch DTO crates, keeping overall CI time unchanged.

- -

PR Template — Wire Contract (SDK gate) Section

-

The PR template now includes a dedicated section that contributors must complete whenever the wire-contract.yml check runs:

-
    -
  • Regenerate fixtures — run the fixture-generation script and commit the updated golden files.
    • -
  • Update endpoints.json — reflect any added, removed, or renamed message variants in the endpoint registry.
    • -
  • Link paired SDK PR — paste the URL of the corresponding mero-js pull request so reviewers can verify the SDK side is ready before merge.
    • -
- -

sdk-ref Body Convention

-

To wire the live end-to-end suite to a paired SDK branch, add the following line anywhere in the PR body:

-
sdk-ref: <branch-name>
-

For example:

-
sdk-ref: feat/signed-group-op-v4
-

The CI pipeline reads sdk-ref from the PR body and checks out that branch of mero-js when running the live e2e job. If sdk-ref is absent the e2e job falls back to the SDK's main branch. This convention is required — not optional — whenever endpoints.json changes.

- -

Maintenance Obligations Summary

-
    -
  1. Change a DTO field or add/remove a message variant.
    2. -
  2. wire-contract.yml fires automatically via path filter.
    3. -
  3. Regenerate wire fixtures locally; commit updated golden files.
    4. -
  4. Update endpoints.json to match.
    5. -
  5. Open a paired PR in mero-js; add sdk-ref: <branch> to the server-side PR body.
    6. -
  6. Both PRs must be green before either is merged.
    7. -
-
-
-
- - - diff --git a/crates/client/src/client/group.rs b/crates/client/src/client/group.rs index c9b3b78c75..522ea46f0b 100644 --- a/crates/client/src/client/group.rs +++ b/crates/client/src/client/group.rs @@ -329,7 +329,6 @@ where /// Local-only opt-out from a single context. Stops sync and disarms /// auto-follow on this node only — peers do not observe the leave. /// Reversal: call [`Self::join_context`] which clears the marker. - /// See `architecture/membership-and-leave.html` § 4 for semantics. pub async fn leave_context(&self, context_id: &str) -> Result { let response = self .connection @@ -342,8 +341,7 @@ where /// `MemberLeft` op that all peers apply, deleting the leaver's /// direct membership row. Subject to apply-side validation: /// signer must be a direct member, must not be Owner, and must - /// not be the only admin (`LastAdmin` rejection). See - /// `architecture/membership-and-leave.html` § 5. + /// not be the only admin (`LastAdmin` rejection). pub async fn leave_group(&self, group_id: &str) -> Result { let response = self .connection @@ -356,7 +354,7 @@ where /// every descendant where the leaver has a direct row; multi-scope /// owner + last-admin checks run upfront. Rejects with /// `MustTransferOwnership` if the leaver owns any group in the - /// subtree. See `architecture/membership-and-leave.html` § 6. + /// subtree. pub async fn leave_namespace(&self, namespace_id: &str) -> Result { let response = self .connection diff --git a/crates/context/src/auto_follow.rs b/crates/context/src/auto_follow.rs index 1d9040cc00..ed68d95beb 100644 --- a/crates/context/src/auto_follow.rs +++ b/crates/context/src/auto_follow.rs @@ -5,8 +5,7 @@ //! join ops on behalf of this node — subject to the member having the //! relevant [`AutoFollowFlags`] set for the group in question. //! -//! See `architecture/auto-follow.html` for the full -//! architecture. This module implements the context side of Phase 3: +//! This module implements the context side of Phase 3: //! //! - `OpEvent::ContextRegistered { group, context }` — if this node is //! a member of `group` with `auto_follow.contexts = true`, emit a diff --git a/crates/context/src/handlers/leave_context.rs b/crates/context/src/handlers/leave_context.rs index 867cdc9c23..6ed8602268 100644 --- a/crates/context/src/handlers/leave_context.rs +++ b/crates/context/src/handlers/leave_context.rs @@ -1,7 +1,5 @@ //! Local-only opt-out from a single context. //! -//! See `architecture/membership-and-leave.html` § 4 for the design. -//! //! Mechanism: //! 1. Look up the context's local signing identity by scanning //! `ContextIdentity` rows for `(context_id, *)` and finding the one diff --git a/crates/context/src/handlers/leave_group.rs b/crates/context/src/handlers/leave_group.rs index f1563c361f..883ffa1f70 100644 --- a/crates/context/src/handlers/leave_group.rs +++ b/crates/context/src/handlers/leave_group.rs @@ -1,6 +1,5 @@ //! Self-leave from a single group. //! -//! See `architecture/membership-and-leave.html` § 5 for the full design. //! This handler publishes `GroupOp::MemberLeft { member: signer }` via //! the existing local-governance pipeline. All apply-side validation //! (direct-row, owner, last-admin) lives in @@ -23,8 +22,7 @@ //! listener drops the subgroup's local rows (signing keys included). //! The listener deliberately does NOT unsubscribe from the namespace //! gossipsub topic for a subgroup-only leave — other memberships -//! under the same namespace still need it. See ADR 0002 -//! (`docs/adr/0002-fleet-tee-leave-protocol.md`). +//! under the same namespace still need it. use std::sync::Arc; diff --git a/crates/context/src/handlers/leave_namespace.rs b/crates/context/src/handlers/leave_namespace.rs index a5ef5da05c..f1ff62c34d 100644 --- a/crates/context/src/handlers/leave_namespace.rs +++ b/crates/context/src/handlers/leave_namespace.rs @@ -1,6 +1,5 @@ //! Self-leave from a namespace (root group). //! -//! See `architecture/membership-and-leave.html` § 6 for the design. //! This handler is a thin wrapper over `leave_group`: it publishes //! `GroupOp::MemberLeft { member: signer }` at the namespace root. //! The apply path in `calimero_governance_store` detects "this group has no @@ -26,8 +25,7 @@ //! `node_client.unsubscribe_namespace` call below is still issued //! synchronously by this handler so the unsubscribe is ordered with //! the user-visible handler response; the listener also issues an -//! unsubscribe (idempotent) as part of its TEE-eviction cleanup. See -//! ADR 0002 (`docs/adr/0002-fleet-tee-leave-protocol.md`). +//! unsubscribe (idempotent) as part of its TEE-eviction cleanup. use std::sync::Arc; diff --git a/crates/context/src/lib.rs b/crates/context/src/lib.rs index 4ffdccc8c0..379f24f774 100644 --- a/crates/context/src/lib.rs +++ b/crates/context/src/lib.rs @@ -795,7 +795,7 @@ impl Actor for ContextManager { // the recovered context is reliably replicated rather than dropped by // the best-effort broadcast. Do not reorder these two spawns. auto_follow::spawn(self.datastore.clone(), self.context_client.clone()); - // Self-purge handler (see docs/adr/0002-fleet-tee-leave-protocol.md) — reacts + // Self-purge handler — reacts // to `OpEvent::TeeMemberRemoved` (paired follow-up emitted ONLY when the // removed member's prior role was `ReadOnlyTee`) for our own identity and // drops the local rows (signing keys, gov ops, namespace identity, diff --git a/crates/context/src/self_purge.rs b/crates/context/src/self_purge.rs index e128eaccac..f33a7b79ee 100644 --- a/crates/context/src/self_purge.rs +++ b/crates/context/src/self_purge.rs @@ -7,9 +7,6 @@ //! that signing-key material, gov-op log, namespace identity, and //! membership-side metadata do not linger after a TEE eviction. //! -//! See `docs/adr/0002-fleet-tee-leave-protocol.md` for the architectural -//! framing. -//! //! # Role-scoped: TEE removals only //! //! The listener intentionally gates on `OpEvent::TeeMemberRemoved`, diff --git a/crates/governance-store/src/op_events.rs b/crates/governance-store/src/op_events.rs index f6c689226d..e8e8f85b01 100644 --- a/crates/governance-store/src/op_events.rs +++ b/crates/governance-store/src/op_events.rs @@ -3,8 +3,7 @@ //! Downstream handlers subscribe to observe governance ops as they are //! applied to local state. Every emitted event corresponds to a //! successfully-applied op; this is the hook used by higher-level -//! features (notably auto-follow for group membership, see -//! `architecture/auto-follow.html`) to react to state +//! features (notably auto-follow for group membership) to react to state //! changes without reaching into the apply path. //! //! The channel is best-effort: slow subscribers that fall behind receive diff --git a/crates/governance-types/src/lib.rs b/crates/governance-types/src/lib.rs index d427b0dc9e..12622320ca 100644 --- a/crates/governance-types/src/lib.rs +++ b/crates/governance-types/src/lib.rs @@ -156,7 +156,6 @@ pub enum GroupOp { /// proper forward secrecy requires a follow-up two-phase rotation /// (planned as a follow-up). For full cryptographic leave today, /// pair with admin-initiated `MemberRemoved`. - /// See `architecture/membership-and-leave.html` § 5. /// /// Carries the same two convergence claims as `MemberRemoved`, /// signed by the leaver. The leaver is honest by definition for @@ -285,7 +284,6 @@ pub enum GroupOp { /// current Owner; `new_owner` must already be a member. Updates /// `GroupMetaValue.owner_identity`. The previous owner remains a /// regular admin (no automatic role change beyond the owner field). - /// See `architecture/membership-and-leave.html` § 7. TransferOwnership { new_owner: PublicKey }, /// Cascade variant of [`Self::TargetApplicationSet`]: update the /// target application on the signed group AND on every descendant diff --git a/crates/sdk/AGENTS.md b/crates/sdk/AGENTS.md index b2a9315cc1..9321380d5b 100644 --- a/crates/sdk/AGENTS.md +++ b/crates/sdk/AGENTS.md @@ -141,9 +141,9 @@ pub enum Event<'a> { ### `#[app::migrate]` — state-migration export > **App developers:** the user-facing guide is the Migrations page in the -> architecture docs (`architecture/migrations.html`) — when to migrate, -> `#[derive(Migrate)]`, the convergence rule, testing. This section is the -> contributor-facing internals. +> docs site () — +> when to migrate, `#[derive(Migrate)]`, the convergence rule, testing. This +> section is the contributor-facing internals. Marks a stand-alone function as the WASM export the node runtime calls during a migration upgrade. The node resolves which migrate to diff --git a/crates/storage/src/rotation_log.rs b/crates/storage/src/rotation_log.rs index 300688795e..50c2958eee 100644 --- a/crates/storage/src/rotation_log.rs +++ b/crates/storage/src/rotation_log.rs @@ -10,8 +10,7 @@ //! //! # Design //! -//! Per [ADR 0001](../../../docs/adr/0001-shared-storage-concurrent-rotation.md): -//! every accepted rotation appends an entry; the node-side reader compares +//! Every accepted rotation appends an entry; the node-side reader compares //! entries by **causal-first → HLC → signer-pubkey** ordering. //! //! The log is materialized as an `UnorderedMap<[u8; 32], RotationLogEntry>` diff --git a/crates/store/src/key/group/mod.rs b/crates/store/src/key/group/mod.rs index c09c492331..86e8f34c68 100644 --- a/crates/store/src/key/group/mod.rs +++ b/crates/store/src/key/group/mod.rs @@ -1234,7 +1234,6 @@ pub struct GroupMetaValue { /// pre-existing groups). The Owner has exclusive privileges no other /// admin can perform: `TransferOwnership`, `DeleteGroup`/ /// `DeleteNamespace`, and immunity from involuntary `MemberRemoved`. - /// See `architecture/membership-and-leave.html` § 7. /// /// Set to the signer of `CreateGroupRequest` on group creation. New /// groups have `owner_identity == admin_identity` initially. @@ -1253,8 +1252,6 @@ pub struct GroupMetaValue { /// is present, the handler emits a self-admission op in the child carrying /// the member's inherited role. /// -/// See `architecture/auto-follow.html` for the full design. -/// /// # Default /// /// `contexts = true`, `subgroups = false`. Joining a group implies wanting diff --git a/docs-site/README.md b/docs-site/README.md deleted file mode 100644 index e43e26eb7a..0000000000 --- a/docs-site/README.md +++ /dev/null @@ -1,32 +0,0 @@ -# Calimero Docs — Astro Starlight spike (Phase 0) - -Proof-of-concept for the documentation overhaul (see -[`../docs/documentation-roadmap.md`](../docs/documentation-roadmap.md)). - -It demonstrates, on real tooling: - -- **Astro Starlight** shell — sidebar, built-in search, dark/light, responsive. -- The **Calimero dark theme** ported into Starlight tokens (`src/styles/theme.css`). -- **MDX authoring** — content as Markdown, diffs cleanly in PRs. -- The **bespoke animated-SVG diagram engine** mounted as an Astro island - (`src/components/SeqDiagram.astro`), used on the Protocol Overview page. - -This site lives **outside `architecture/`** on purpose: the doc-update bot's -`static_docs_dirs` is `architecture/` only, so authored content here is fenced -from it by construction. - -## Run it - -```sh -cd docs-site -npm install -npm run dev # http://localhost:4321 -npm run build # static output in dist/ -``` - -## What's here vs. next - -This is the **walking skeleton**: the landing page (audience-routed) and one -ported Protocol chapter (Overview) with a live animated diagram. The full -content migration — porting the ten protocol chapters + appendix, then the -Build / Operate / Contribute tracks — happens in the later roadmap phases. diff --git a/docs-site/src/content/docs/contribute/adrs.mdx b/docs-site/src/content/docs/contribute/adrs.mdx deleted file mode 100644 index 172f05ddbf..0000000000 --- a/docs-site/src/content/docs/contribute/adrs.mdx +++ /dev/null @@ -1,39 +0,0 @@ ---- -title: Architecture Decision Records -description: The accepted ADRs that record why key design choices were made. -sidebar: - label: Decision Records - order: 5 ---- - -import { LinkCard, Aside } from '@astrojs/starlight/components'; - -**Architecture Decision Records (ADRs)** capture a significant design decision, -the context behind it, and its consequences. They live in -[`docs/adr/`](https://github.com/calimero-network/core/tree/master/docs/adr) and -are the authoritative record of _why_ the system is the way it is — read the -relevant ADR before changing the subsystem it covers. - - - -## Accepted ADRs - - - - -## Related - - - diff --git a/docs-site/.gitignore b/docs/.gitignore similarity index 100% rename from docs-site/.gitignore rename to docs/.gitignore diff --git a/docs-site/CODE-DOC-GAPS.md b/docs/CODE-DOC-GAPS.md similarity index 100% rename from docs-site/CODE-DOC-GAPS.md rename to docs/CODE-DOC-GAPS.md diff --git a/docs-site/IMPROVEMENTS.md b/docs/IMPROVEMENTS.md similarity index 99% rename from docs-site/IMPROVEMENTS.md rename to docs/IMPROVEMENTS.md index a3464fd6fd..c72f871a55 100644 --- a/docs-site/IMPROVEMENTS.md +++ b/docs/IMPROVEMENTS.md @@ -1,8 +1,7 @@ # Docs improvement backlog (100) A living checklist for taking the Calimero docs from "complete first draft" to -polished product. Grouped by theme; check items off as they land. See -`../docs/documentation-roadmap.md` for the overall plan. +polished product. Grouped by theme; check items off as they land. ## A · Visual design & theme - [x] 1. Add a Calimero logo in the header + a favicon. @@ -96,7 +95,7 @@ polished product. Grouped by theme; check items off as they land. See - [ ] 73. "Where to start / good first issue" guide. - [ ] 74. Testing strategy page (unit/integration/e2e/sync-sim). - [ ] 75. "Add a new X" cookbook (config option, collection, host fn). -- [x] 76. ADR index page that lists `docs/adr`. +- [ ] 76. ~~ADR index page~~ (dropped — ADRs removed from the repo). - [ ] 77. Debugging guide (merodb, tracing, state inspection). - [ ] 78. Document the actor message contracts. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000..94349454c6 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,26 @@ +# Calimero Docs + +The Calimero Core documentation site, built with [Astro Starlight](https://starlight.astro.build/) +and published to . + +- **Astro Starlight** shell — sidebar, built-in search, dark/light, responsive. +- The **Calimero dark theme** ported into Starlight tokens (`src/styles/theme.css`). +- **MDX authoring** — content as Markdown, diffs cleanly in PRs. +- A **bespoke animated-SVG diagram engine** mounted as an Astro island + (`src/components/SeqDiagram.astro`), used on the Protocol Overview page. + +## Run it + +```sh +cd docs +npm install +npm run dev # http://localhost:4321/core/ +npm run build # static output in dist/ +npm run check # astro build + internal link check (what CI runs) +``` + +## Layout + +Pages live in `src/content/docs/`, split by audience track — **Build**, **Operate**, +**Protocol Reference**, and **Contribute**. See `src/content/docs/contribute/docs.mdx` +for the authoring guide. diff --git a/docs/adr/0001-shared-storage-concurrent-rotation.md b/docs/adr/0001-shared-storage-concurrent-rotation.md deleted file mode 100644 index 468abeb820..0000000000 --- a/docs/adr/0001-shared-storage-concurrent-rotation.md +++ /dev/null @@ -1,195 +0,0 @@ -# ADR 0001 — Concurrent rotation semantics for `SharedStorage` - -| | | -|---|---| -| **Status** | Accepted | -| **Date** | 2026-04-24 | -| **Deciders** | Calimero core team | -| **Context** | [#2233](https://github.com/calimero-network/core/issues/2233) — DAG-causal Shared verification, phase **P4 (design)** | -| **Constrains** | [#2233](https://github.com/calimero-network/core/issues/2233) phase **P2** (rotation history schema) | - -> First ADR in this repo. Convention: `docs/adr/NNNN-kebab-title.md`, monotonically numbered, "Proposed → Accepted → Superseded" lifecycle. - -## Context - -`SharedStorage` is group-writable storage with a mutable writer set. Any current writer can rotate the writer set via `rotate_writers(new_writers)`. v2 (#2230) ships with monotonic-nonce + last-write-wins on `writers_nonce`; #2233 replaces that with DAG-causal verification. - -When two current writers issue rotations *concurrently* (siblings in the DAG, neither in the other's causal history, both signed by valid pre-rotation writers), the merge has to pick a deterministic outcome. v2 picks LWW by `writers_nonce`; that loses information. Before P2 designs the rotation-history schema, we need to know what merge rule we're storing for, because the rule constrains the schema. - -This ADR captures the rule and the rejected alternatives. - -### Concurrent vs. causal: definitions used in this ADR - -For two rotation actions `R₁` and `R₂` on the same entity, contained in causal deltas `D₁` and `D₂`: - -- **`R₁ happens-before R₂`** iff `D₁.id` is in the transitive `parents` set of `D₂` (the DAG-causal "knows about" relation). -- **Concurrent** iff neither `R₁ happens-before R₂` nor `R₂ happens-before R₁`. - -These are the standard CRDT terms; we use them in the strict DAG-structural sense, not wall-clock time. - -## Decision - -**Causal-first LWW, with HLC tiebreak, with signer-pubkey tiebreak.** - -Given two rotations `R₁` (in delta `D₁`) and `R₂` (in delta `D₂`) on the same entity: - -1. **Causal precedence wins, unconditionally.** If `R₁ happens-before R₂`, then `R₂` wins. The author of `R₂` saw `R₁`'s reality and chose to rotate from that state — overriding `R₂` would discard a writer's informed decision. -2. **For truly concurrent rotations**, the rotation with the **larger `D.hlc`** wins. `HybridTimestamp` is `Ord` and embeds both NTP64 wall-time and a node `ID`, giving a deterministic total order across nodes. -3. **HLC tiebreak (vanishingly rare)** by lexicographic comparison of the signing writer's pubkey bytes — smaller bytes win. Spelled out for spec completeness; practically unreachable because HLC's embedded node ID already disambiguates same-instant events from different nodes. - -The same rule resolves writes (`insert`) by analogy: causal-first, then HLC, then signer-hash. Rotations and value writes use the same merge ordering. - -### Worked examples - -> Notation: `Wₐ`, `W_b` are writers. `(setₓ)` is the post-rotation writer set. Arrows `→` denote DAG edges (parent → child). Times in brackets are HLCs. - -**Example A — sequential, no race.** -``` -[t=10] D₁ by Wₐ: rotate → (Wₐ, W_b) - ↓ -[t=20] D₂ by W_b: rotate → (W_b, W_c) -``` -`D₁ happens-before D₂`. Result: `(W_b, W_c)`. (No change vs. v2.) - -**Example B — concurrent siblings, different intent.** -``` - [t=10] D_root: writers = (Wₐ, W_b) - ↓ ↓ -[t=20, Wₐ] D₁: → (Wₐ) [t=21, W_b] D₂: → (W_b) - (Wₐ rotates W_b out) (W_b rotates Wₐ out) -``` -Concurrent. HLC: `D₂` (21) > `D₁` (20). Result: `(W_b)`. `Wₐ` is rotated out; `Wₐ`'s rotation that removed `W_b` is discarded. - -This is the *expected information loss* of any non-union rule — and is why option 2 (union-with-re-election) exists. We accept the loss; see "Consequences" below. - -**Example C — concurrent siblings, same HLC (exotic).** -``` - D_root: writers = (Wₐ, W_b) - ↓ ↓ -[t=20, Wₐ] D₁: → (Wₐ, W_c) [t=20, W_b] D₂: → (W_b, W_d) -``` -HLC tie. Tiebreak by signer pubkey bytes: assume `bytes(Wₐ) < bytes(W_b)` lexicographically → `D₁` wins → result `(Wₐ, W_c)`. - -In practice, `HybridTimestamp` includes a node ID, so two distinct nodes producing identical HLCs at the same nanosecond requires the IDs to also collide — astronomically unlikely. The tiebreak is for spec completeness, not real-world frequency. - -**Example D — concurrent rotation vs. concurrent value write.** -``` - D_root: value = "hello", writers = (Wₐ, W_b) - ↓ ↓ -[t=20, Wₐ] D₁: rotate → (Wₐ) [t=21, W_b] D₂: insert("world") -``` -The value-write action (D₂'s insert) and the rotation action (D₁) target the same entity but are different action kinds. Resolved independently: -- **Writer set**: only `D₁` rotates → `(Wₐ)`. -- **Value**: `D₂` writes "world" with HLC 21; later HLC wins → "world". - -Result: writers `(Wₐ)`, value `"world"`. `W_b`'s write is *kept* even though `W_b` is no longer a writer post-merge — `W_b` was authoritatively a writer at the time `D₂` was authored (siblings of `D_root`), and the verifier validates against `writers_at(entity, D₂.parents)`, not against the post-merge writer set. (See [#2233 epic constraints](https://github.com/calimero-network/core/issues/2233).) - -This is the central guarantee that makes concurrent operation safe: a write signed by the writer-set-as-of-the-author's-causal-view never gets retroactively rejected by a later rotation. - -## Alternatives considered - -### A1 — LWW by `writers_nonce` *(what v2 ships)* - -| Pro | Con | -|---|---| -| Simple, already implemented | Loses one rotation's intent on every concurrent rotation | -| No DAG infra needed | `writers_nonce` collisions on concurrent siblings — falls back to BTreeSet sort, fully arbitrary | -| | Doesn't address the four #2197 partition scenarios | - -**Rejected** because it doesn't deliver the correctness improvement that motivates #2233. If the answer is "LWW is fine," the entire epic is unnecessary. - -### A2 — Writer-set union with re-election - -Apply both rotations, take the union, mark the entity as "needs re-election." A current writer must issue a signed re-election to reduce the set. - -| Pro | Con | -|---|---| -| No information loss — both rotations preserved | UX disaster: every concurrent rotation requires a follow-up coordination step | -| Theoretically the most "CRDT-pure" choice | Forces every app to handle a "pending re-election" UI state | -| | Storage cost: P2 must carry intermediate union state | -| | If no re-election ever happens, the union persists indefinitely — *worse* than LWW because a malicious writer added by an attacker stays in until manually removed | -| | Double-spend / safety windows: between rotation and re-election, the union has more writers than either author intended | - -**Rejected.** The intent-preservation upside doesn't justify the operational complexity. Apps that need this can implement it on top of the chosen rule (commit-then-re-elect via two `rotate_writers` calls) without the storage layer mandating it. - -### A3 — Signer-pubkey hash tiebreak (alone) - -| Pro | Con | -|---|---| -| Deterministic | Same intent loss as LWW | -| | Doesn't use causal information at all | - -**Rejected as the primary rule.** Adopted as the *final* tiebreak in our chosen rule (after HLC) because it's a clean deterministic floor. - -### A4 — First-causal-parent-wins *(closest to chosen rule)* - -The original framing in the #2233 epic: "the rotation whose nearest common DAG ancestor is causally earliest wins." - -| Pro | Con | -|---|---| -| Uses DAG infra naturally | The "nearest common ancestor" is the same delta for any two siblings — degenerates to needing a separate tiebreak anyway | -| | Underspecified for the sibling case (which is exactly the hard case) | - -**Refined into the chosen rule.** The chosen rule keeps the spirit ("causal first") and replaces the underspecified ancestor comparison with concrete HLC + pubkey tiebreaks for true siblings. - -### A5 — Reject concurrent rotations - -The entity refuses to apply a rotation if its current writer set has already moved past the rotation's claimed view. The losing writer must re-fetch state and re-issue. - -| Pro | Con | -|---|---| -| Strong intent preservation (winning rotation always reflects its author's full intent) | Breaks the "eventually converges without app intervention" CRDT guarantee | -| | Two co-admins rotating at the same time → both rotations *could* fail in pathological reorderings, requiring app retry logic | -| | Adversarial writers can DoS rotation by spamming concurrent attempts | -| | Nodes processing the rejection have to surface "rotation rejected, please retry" to userland — every SharedStorage user has to handle this | - -**Rejected.** Calimero's model is "sync converges silently, no app retries needed for ordinary CRDT merges." Rotation should not be a special case where convergence requires application-level retry. - -## Consequences - -### What P2 needs to store - -The chosen rule is **stateless beyond a per-entity rotation log** sorted by causal order. No intermediate "pending re-election" state, no union accumulation. P2 schema (option a in the epic) is sufficient: - -```text -RotationLogEntry { - delta_id: [u8; 32], // CausalDelta.id where the rotation lives - delta_hlc: HybridTimestamp, // for sibling tiebreak - signer: PublicKey, // for HLC-tie tiebreak - new_writers: BTreeSet, - writers_nonce: u64, // kept for v2 compat / debugging only -} -``` - -`writers_at(entity_id, causal_parents)` walks the log filtered to entries whose `delta_id` is in `causal_parents`' transitive ancestor set, picks the *latest* by the rule above, and returns its `new_writers`. - -If `causal_parents` is empty (P1 codepath, snapshot leaf push, local apply), `writers_at` returns the most recently appended entry — matches v2 LWW behavior, preserves backward compatibility. - -### What P3 (verifier) does with it - -The verifier validates a Shared action's signature against `writers_at(action.entity_id, ctx.causal_parents)` — *not* against the currently stored writer set. This is what makes Example D safe. - -The v2 `signer: Option` hint on `SignatureData` (already shipped, dormant) is validated against the same `writers_at(...)` set — not current. Spelled out in the #2233 epic; reaffirmed here. - -### What about pure value writes (insert)? - -The same rule extends without change: causal-first, HLC second, signer-hash third. P2 doesn't need a separate value-write history; the existing CRDT-merge layer (LWW per slot for `LwwRegister`, etc.) already handles this — only the *verifier* needs to know `writers_at(entity, causal_parents)` to validate signatures, not to merge values. - -### What we accept - -- **One rotation's intent can be lost** in the truly-concurrent case (Example B). This is unavoidable without union-with-re-election (A2), and we judged A2's complexity worse than the loss. -- **Out-of-order delivery is fine.** Two nodes seeing `D₁` then `D₂` vs. `D₂` then `D₁` converge to the same result because the rule depends only on (causal relation, HLC, signer hash) — all immutable per-delta. -- **HLC clock skew matters.** A writer with a fast clock can win all concurrent races. This is the standard LWW concern; mitigated by the existing HLC + drift-tolerance check (`DRIFT_TOLERANCE_NANOS`). If clock-skew abuse becomes a real threat, A2 (union-with-re-election) is the escape hatch — schema can be extended without a wire-format break since the rotation log is local-only. - -### What we explicitly do NOT decide here - -- **Compaction policy.** The epic specifies "1000-entry sliding window + snapshot." That number belongs in P6 measurements; the *shape* (sliding window with a snapshot of the writer set at the boundary) is locked in here so P2 can lay out the index schema. -- **Re-election as an opt-in feature.** Apps that want union-with-re-election can build it: `rotate_writers(union)` followed by a coordinated `rotate_writers(reduced)`. The storage layer doesn't need to know. -- **Bootstrap race.** Listed as item 4 in #2197 / #2233. Not addressed by this ADR — bootstrap concurrency happens *before* there's a shared causal ancestor, so causal-first doesn't apply. P5/P6 is the right place; tracked separately. - -## References - -- [#2197](https://github.com/calimero-network/core/issues/2197) — original SharedStorage spec, partition scenarios -- [#2230](https://github.com/calimero-network/core/issues/2230) — v2 follow-ups (signer hint, rotate-self-out) -- [#2233](https://github.com/calimero-network/core/issues/2233) — DAG-causal verification epic (this ADR is phase P4-design) -- [#2249](https://github.com/calimero-network/core/pull/2249) — v2 PR (signer hint plumbing referenced above) diff --git a/docs/adr/0002-fleet-tee-leave-protocol.md b/docs/adr/0002-fleet-tee-leave-protocol.md deleted file mode 100644 index 30f032d75f..0000000000 --- a/docs/adr/0002-fleet-tee-leave-protocol.md +++ /dev/null @@ -1,133 +0,0 @@ -# ADR 0002 — Fleet TEE leave hygiene (post-eviction cleanup) - -| | | -|---|---| -| **Status** | Accepted — implemented (see §Implementation as shipped) | -| **Date** | 2026-06-02 (proposed); 2026-06-08 (updated to match implementation) | -| **Deciders** | Calimero core team, mero-tee KMS team | -| **Context** | [mdma#155](https://github.com/calimero-network/mdma/issues/155) — HA disable doesn't propagate to merod; `[[project-ha-no-convergence-loop]]` — parent reconcile-loop gap | -| **Constrains** | Local soft-vs-hard cleanup choice for evicted members; the self-purge listener + reconcile in `crates/context/src/self_purge.rs` | -| **Implemented by** | [#2680](https://github.com/calimero-network/core/pull/2680) (role-scoped purge), [#2724](https://github.com/calimero-network/core/pull/2724) (failure-class gating + metric), [#2725](https://github.com/calimero-network/core/pull/2725) (marker-based reconcile) | - -> Earlier drafts of this ADR scoped a full pull-based leave protocol (sidecar polls steady-state, fleet-leave path, KMS forget). That over-reached: `MemberRemoved` *already* does cryptographic eviction via the existing key-rotation pipeline. The user-visible "I disabled HA but still see the fleet node" complaint is solved client-side by tauri-app issuing `remove_group_members` on disable. What remains is the *hygiene* tail: local data purge on the evicted node, stale sidecar cache, and the broader sidecar-reconcile-loop story. -> -> **Update (2026-06-08):** the original draft concluded "core: NO CHANGE — defer the local hard purge." That was revisited during implementation. Forward secrecy is provided by key rotation regardless, but a **`ReadOnlyTee`** member has no rejoin pathway (re-admission derives a *fresh* attestation pubkey), so leaving its on-disk signing-key material around buys nothing and is a forward-secrecy hygiene risk. So core *does* now hard-purge — but **only for the TEE role**, leaving the soft-leave path intact for everyone else. §Decision(a) and §Implementation record what shipped. - -## Context - -When the owner publishes `MemberRemoved` for a `ReadOnlyTee` member (the path that fires when the desktop client calls `remove_group_members` after a successful cloud `disable-ha`), the existing key-rotation pipeline in `calimero-governance-store` already handles cryptographic eviction: it generates a fresh group key wrapped for everyone **except** the removed member, so the evicted node can decrypt nothing written after eviction. - -Effects on the evicted fleet node, *before* this ADR's work: - -- **Membership row removed** on the fleet node's merod when it applies the gossiped op; entry added to the deny-list to prevent state-delta replay-in. -- **Forward secrecy on new writes**: the fleet node never receives the rotated key. Cannot decrypt anything written after eviction. -- **Local key material + identity on the fleet's disk**: *untouched* under the old "soft leave" default. The node's keyring still held the old group key and its `NamespaceIdentity`. - -The open hygiene tail, narrowed to what was actually missing: - -1. **Soft-vs-hard local cleanup on the evicted node.** Historically "soft" (rows removed, keys retained). For a regular member this is *desirable* — it's what `kick-and-rejoin-keyshare` / inheritance-rejoin reuse. For a `ReadOnlyTee` node — which cannot rejoin under the same identity — retained key material is pure hygiene debt. -2. **Fleet sidecar reconcile loop** (mero-tee). Cosmetic-but-real: the sidecar's cache going stale for namespaces it's no longer in. Composes with the broader `[[project-ha-no-convergence-loop]]` rework. -3. **KMS forgetting** (mero-tee). Whether the KMS should release attestation-bound key material on eviction. Cross-team flag, independent of core. - -## Decision - -**Treat eviction hygiene as independent follow-ups, each scoped narrowly, none blocking the user-visible feature.** - -### (a) Local cleanup — role-scoped hard purge (implemented) - -The soft-vs-hard choice is resolved **by role**, not globally: - -- **Non-TEE removals stay soft.** `Admin` / `Member` / `Observer` removals keep their local rows (identity, signing keys, contexts) so `kick-and-readd` / `rejoin-via-keyshare` / inheritance-rejoin can reuse them. Hard-purging these would regress the `apps/scaffolding-e2e/workflows/group-{kick,leave}-*` workflows. This is the soft-leave invariant. -- **`ReadOnlyTee` removals hard-purge.** A TEE node admitted via `MemberJoinedViaTeeAttestation` re-derives its identity from a *fresh* attestation on any future admission, so there is no rejoin path that could reuse the old material. Leaving it on disk buys nothing and is forward-secrecy hygiene debt. So a TEE eviction purges the local signing-key material, `NamespaceIdentity`, gov-op log, and (for a namespace-root eviction) unsubscribes from the namespace gossipsub topic. - -This is **not** a new forward-secrecy mechanism — FS on future writes is provided by the key-rotation pipeline, independent of the purge. The purge deletes the now-orphaned old material the rotation already made useless. - -### (b) Fleet sidecar reconcile — small, in mero-tee - -Sidecar drops stale cache entries when a namespace disappears from `/should-join`. No protocol or merod changes. Sidecar-internal, in `mero-tee`. (Tracked/landed separately; not core's scope.) - -### (c) KMS forget — cross-team flag - -On eviction the fleet node's KMS could invalidate the attestation-bound key it holds for the namespace. Requires a KMS-side invalidation API, a trigger path, and a clear threat model (the TEE could have copied plaintext while admitted). Filed as a cross-team flag, owned by the mero-tee team. Not a core decision. - -## Implementation (as shipped) - -All in `crates/context/src/self_purge.rs` unless noted. The handler is a detached listener spawned per node, mirroring the `auto_follow` architectural split (self-detection — "did this op evict *me*?" — is per-node state, not part of the node-agnostic apply contract). - -### Role-scoped event listener (#2680) - -The listener gates on `OpEvent::TeeMemberRemoved`, **not** the generic `OpEvent::MemberRemoved`. Both are emitted by the apply path; `TeeMemberRemoved` fires only when the removed member's stored role was `ReadOnlyTee`. This is what keeps the soft-leave path intact for non-TEE removals (§Decision(a)). - -- **Subgroup-only eviction** → purge that subgroup's local rows; keep the `NamespaceIdentity` + gossipsub subscription (other memberships under the namespace still need them). -- **Namespace-root eviction** → cascade-purge the whole subtree, then drop namespace-level state, then unsubscribe. - -### Failure-class gating (#2724) - -The namespace cascade tracks two failure classes: - -- `signing_key_purge_failed` — the security-critical `delete_group_local_rows` (signing-key material) failed. Load-bearing. -- `context_cleanup_failed` — a best-effort dead-pointer cleanup (context-index unregister, tree-edge delete) failed. Non-security. - -Dropping the `NamespaceIdentity` and unsubscribing are gated on the **signing-key purge only**. A best-effort cleanup failure no longer blocks the security-critical finalize. A `self_purge_failures_total` Prometheus counter (labeled by branch × class) records every failure (#2686). - -### Marker-based reconcile (#2725) - -`TeeMemberRemoved` fires once per eviction, and an already-evicted identity receives no further events — so a *missed* or *partially-failed* purge has no event-driven retry. Recovery is a **startup reconcile**, made role-safe by a durable **pending-self-purge marker** (`PendingSelfPurge` store key): - -- The marker is written **only** in the listener's namespace-purge dispatch — reachable only for a confirmed `TeeMemberRemoved` matching this node's identity (node-aware *and* role-aware). Cleared on full purge success. -- On startup the reconcile enumerates **markers only** and purges a namespace **iff (marker present) AND (still no surviving membership)**. Identity-gone or re-admitted clears the stale marker without purging; a read error skips (never purge on uncertainty). - -The marker is essential: post-eviction the role row is erased, so a role-blind scan of `NamespaceIdentity` rows could not distinguish evicted-TEE residue from a *pending join* (identity written before the membership row materializes) or *non-TEE soft-leave residue* (identical shape, but must be kept for rejoin) — and would false-purge both. The marker is the role/intent gate; the still-evicted check is the safety gate; both must hold. - -## Alternatives considered - -### Full pull-based leave protocol (the original ADR draft) - -Redefine `/should-join` to set-state, add a `fleet_leave` path in core, wire the sidecar to issue `meroctl tee fleet-leave`. **Rejected**: `fleet_leave` would invoke `leave_group` self-leave, whose key rotation is the deferred two-phase design — *worse* forward secrecy than the owner-side `MemberRemoved` path, which already rotates correctly. A fleet-initiated leave would be redundant or actively wrong. - -### Role-blind reconcile scan (first cut of #2725) - -Scan every `NamespaceIdentity` and purge any with no surviving membership. **Rejected** in review: false-purges pending joins and non-TEE soft-leave residue (see §Implementation → marker-based reconcile). Replaced by the marker gate. - -### Cloud-to-fleet push notification - -mdma webhook to fleet nodes on disable. **Rejected**: server surface on fleet sidecars, addressability on mdma, solves nothing the tauri-driven `remove_group_members` doesn't. - -## Consequences - -### What we lock in - -- The user-visible HA-disable flow stays client-driven: tauri-app calls `remove_group_members` on the owner's local merod after a successful cloud disable. (Separate tauri-app issue.) -- **Role-scoped cleanup**: soft-leave for non-TEE (rejoin material retained), hard-purge for `ReadOnlyTee` (no rejoin path). -- Recovery for missed/partial TEE purges is a **marker-gated startup reconcile**, not an event retry. -- Fleet sidecar reconcile + KMS forget remain mero-tee-owned. - -### What we deliberately DON'T do - -- Add a `fleet_leave` path in core. Owner-side `remove_group_members` is the correct eviction primitive. -- Hard-purge non-TEE removals. Would regress the soft-leave rejoin workflows. -- A continuous/periodic reconcile. Startup-only for now (see §Failure modes). - -### Failure modes we accept - -- **Historical-blob retention on an evicted TEE before purge / on the soft-leave path.** No forward-secrecy delta — FS is provided by key rotation, not the purge. The TEE hard-purge deletes the node's *own* now-orphaned key material as hygiene. -- **Pure lagged-drop of a `TeeMemberRemoved` event.** If the broadcast channel lags (requires >1024 events between recv calls — rare) the listener never dispatches, so no marker is written and the reconcile cannot recover it. Residue persists until a future eviction event or a periodic reconcile (not yet built). Bounded and rare; not an FS hole. -- **Partial cascade failure (`signing_key_purge_failed`).** The `NamespaceIdentity` + signing-key residue is left on disk *with the marker*, and the startup reconcile completes it on the next restart. No event-driven retry within a session. -- **Subgroup-only purge-failure residue.** The reconcile is namespace-level: a node kicked from a single subgroup (still a namespace member) whose `purge_subgroup_for_self` partially failed keeps subgroup-level residue the sweep won't catch (its still-evicted gate correctly keeps a live namespace member). The **subgroup-reconcile follow-up** is tracked in [#2726](https://github.com/calimero-network/core/issues/2726). -- **Stale sidecar cache for ~1 poll cycle.** Bounded by the `should-join` poll interval (mero-tee). -- **KMS holding unused key material** until KMS-side forget lands. Acceptable given the threat model. - -## Open questions - -1. **Periodic reconcile** — startup-only leaves a mid-session lagged-drop uncovered until the next restart. Worth a periodic pass if that window matters. -2. **Subgroup-level reconcile** — [#2726](https://github.com/calimero-network/core/issues/2726); pairs with giving `purge_subgroup_for_self` a `Result`/retry surface ([#2692](https://github.com/calimero-network/core/issues/2692)). -3. **KMS forget semantics** — owned by the mero-tee team. - -## Related - -- `[[project-ha-no-convergence-loop]]` — parent architectural gap (sidecar reconcile-loop story). -- `[[project-ha-fleet-join-working-recipe]]` — sister recipe; leave hygiene must compose without regressing join. -- [#2680](https://github.com/calimero-network/core/pull/2680), [#2724](https://github.com/calimero-network/core/pull/2724), [#2725](https://github.com/calimero-network/core/pull/2725) — the implementing PRs. -- [#2686](https://github.com/calimero-network/core/issues/2686), [#2692](https://github.com/calimero-network/core/issues/2692), [#2721](https://github.com/calimero-network/core/issues/2721), [#2726](https://github.com/calimero-network/core/issues/2726) — tracked follow-ups. -- mdma#155 — the issue that surfaced the gap. -- `crates/context/src/self_purge.rs` — the listener, cascade, failure-class gating, and reconcile. diff --git a/docs-site/astro.config.mjs b/docs/astro.config.mjs similarity index 98% rename from docs-site/astro.config.mjs rename to docs/astro.config.mjs index 3655b96593..c6f992666a 100644 --- a/docs-site/astro.config.mjs +++ b/docs/astro.config.mjs @@ -34,7 +34,7 @@ export default defineConfig({ }, lastUpdated: true, editLink: { - baseUrl: 'https://github.com/calimero-network/core/edit/master/docs-site/', + baseUrl: 'https://github.com/calimero-network/core/edit/master/docs/', }, head: [ { tag: 'meta', attrs: { name: 'theme-color', content: '#0d1117' } }, @@ -214,7 +214,6 @@ export default defineConfig({ label: 'Working on core', items: ['contribute/development', 'contribute/testing', 'contribute/docs'], }, - { label: 'Reference', items: ['contribute/adrs'] }, ], }, ], diff --git a/docs/design/2230-shared-wrapper-verification.md b/docs/design/2230-shared-wrapper-verification.md deleted file mode 100644 index bc8cf47ec5..0000000000 --- a/docs/design/2230-shared-wrapper-verification.md +++ /dev/null @@ -1,196 +0,0 @@ -# Design — Authenticate the `SharedStorage` writer-set rotation (#2230 item 1) - -| | | -|---|---| -| **Status** | Implemented (item 1) — PR #2601, merged 2026-06-02 | -| **Date** | 2026-06-02 | -| **Owner** | Storage / Context / Node sync | -| **Issue** | #2230 (SharedStorage v2: live per-entity verification) — **item 1 only** | -| **Builds on** | #2588 (collection guarding — child entries verified at merge), #2266/#2233 (DAG-causal `writers_at`), the `stored ∪ claimed` signing fix | - -> Illustrative; not final signatures. - -## 1. What is already done (scope reduction) - -#2230 listed four items. Three are effectively closed: - -- **Item 2 — rotate-self-out signing.** Done: `save_raw` stamps the signing - placeholder on `stored ∪ claimed` writers (`interface.rs:2660`), so a writer - rotating itself out (executor ∈ stored, ∉ claimed) still signs; - `sign_authorized_actions` signs whenever the placeholder is present - (`execute/mod.rs:1956`). -- **Item 4 — signer-pubkey hint.** Done: `SignatureData.signer` + the fast-path - single-verify (`action.rs:295`, `interface.rs:308`). -- **Item 1 (data writes) — covered by #2588.** A guarded collection's **child - entries** now sync as their own entities and are verified at merge against - `effective_writers` (proven by the `shared-storage` adversarial e2e: a - non-writer's forged map entry is rejected). So *writes to the data* are - authenticated. - -**What remains: authenticating the writer-set *rotation* itself** (and item 3, -populating `signature_data`, which rides on it). - -## 2. The remaining gap - -The `SharedStorage` **wrapper's** own state — `value` (for a scalar), `writers`, -`writers_nonce` — propagates via the **root-state borsh sync path**: the wrapper -is an inline field of the enclosing `#[app::state]` struct, so its bytes ride in -the root state, and writer-set convergence is `SharedStorage::merge` doing **LWW -on `writers_nonce`** (`shared.rs`). That merge does **not verify who rotated** — -it just takes the higher nonce. - -`rotate_writers` checks `executor ∈ writers` *locally* (`shared.rs:283`), but that -is an API-layer check. A malicious context **member** (not a current writer) can -hand-craft a root-state delta that bumps `writers_nonce` and swaps `writers` to -themselves — bypassing the local check — and the LWW merge on honest peers -accepts it. **Writer-set rotation is currently unauthenticated at merge.** - -(Data writes are safe — they're per-entity verified — but whoever controls the -writer set controls all future writes, so an unauthenticated rotation defeats the -whole guarantee.) - -## 3. Why the obvious fix traps: the dual-write hazard - -The wired-but-disabled approach was: on `insert`/`rotate_writers`, also emit a -per-entity `Update` action for the wrapper so the merge-time `Shared` verifier -runs (`shared.rs:256`, `:297`). It is disabled because the wrapper **also** -propagates inline via root-state borsh, so the data flows **twice** → - -- the receiver computes a **different root hash** than the sender (permanent - divergence — "every rotation produces a permanent divergence"), and -- a **WASM trap during `__calimero_sync_next`**. - -So per-entity verification can't be *added on top of* root-state borsh. The -duplication has to be removed. - -## 4. The fork - -- **(a) Dual-write** (root-state borsh + per-entity action). Rejected — it is the - disabled path above (divergence + trap). Making the root hash ignore the - wrapper's double contribution is fragile and exactly what bit v2. -- **(b) Suppress the root-state duplication.** The wrapper's mutable state lives - **only in its own storage entity**, synced via per-entity `Update` actions - (verified against the writer set, like #2588's child entries). The root state - carries only a **reference** (the wrapper's entity id), not the inline bytes — - exactly how a **collection** already behaves (an `UnorderedMap`'s root-state - borsh is just its id; its data is in child entities). **Recommended.** - -Why (b) is now the natural choice: #2588 already makes the wrapper a child of -root (`add_child_to(*ROOT_ID)`) and makes per-entity `Shared` verification a live, -tested path. (b) finishes the job — it stops the wrapper from *also* serializing -its state into the root blob, so the single source of truth is the verified -per-entity entity. - -## 5. Mechanism (option b) - -1. **Wrapper borsh = ref, not inline.** `SharedStorage` must serialize into the - root state as its entity id only (+ the `_adaptor` marker), with `value`, - `writers`, `writers_frozen`, `writers_nonce`, `signature_data` persisted on its - **own** entity. Two ways: - - a custom `BorshSerialize/Deserialize` that writes the id and lazy-loads the - body from storage on access (like `Collection`), or - - have `#[app::state]` treat `SharedStorage` fields like collection fields - (it already special-cases them in the deterministic-id pass — same predicate). -2. **Mutation emits a verified per-entity Update.** `insert`/`rotate_writers` - `Interface::save(self)` (or `add_child_to`) so each mutation produces an - `Update` action carrying the wrapper's bytes + `StorageType::Shared` metadata, - signed by the executor (gated by the existing `stored ∪ claimed` rule, which - already handles rotate-self-out). No root-state duplication ⇒ no divergence. -3. **Merge verifies the rotation.** On apply, the node resolves `effective_writers` - for the wrapper entity (from its rotation log via `writers_at`) and - `apply_action` validates the signature — so a forged rotation from a non-writer - is rejected, exactly like a forged data write today. -4. **Item 3 (populate `signature_data`)** falls out: after signing, write the - `SignatureData` from the action metadata back onto the wrapper's field (the - wrapper is now a real entity in the action stream). - -The wrapper's own rotation log already exists conceptually (it's the anchor #2590 -will also point children at); this makes it the *authenticated* source of truth -for the writer set. - -## 5a. Implementation recipe (the template exists: `Root`) - -`Root` (`collections/root.rs`) is *exactly* the handle shape to copy: - -```rust -pub struct Root { - inner: Collection, // value stored as ONE entry at a fixed id - #[borsh(skip)] value: RefCell>, // lazy cache, loaded via inner.get(id) - #[borsh(skip)] dirty: bool, -} -// borsh(Root) == borsh(inner Collection) == just its Element (id + metadata). -// The value T is NOT in the struct's borsh — it's a separate entity (the entry). -``` - -So `SharedStorage` becomes: - -```rust -pub struct SharedStorage { - inner: Collection, // holds the single value entry, Shared-stamped - #[borsh(skip)] value: RefCell>, - #[borsh(skip)] frozen: bool, - // NO writers / writers_nonce fields — see below. -} -``` - -- `get`/`get_mut` lazy-load the value entry (Root's `get()` pattern); `insert` - writes it back; the entry is `Shared{writers}` so writes are verified (#2588). -- `borsh(SharedStorage) == borsh(inner) == its Element` ⇒ root state carries only - the id, no body. **No double-write.** - -### Where the writer set lives (the load-bearing detail) - -Do **not** keep `writers` inline, and do **not** rely on it riding in the inner -`Collection`'s Element metadata — that metadata still travels in root state and is -unverified, so a forged root-state delta could still swap it. Instead the writer -set is **resolved from the wrapper entity's rotation log** via -`rotation_log_reader::writers_at` (the verified, per-entity source of truth that -#2588's child-entry verification already consults). `writers()` reads the log; -`rotate_writers` appends a signed entry to the log (a verified per-entity action). -This is what lets `writers_nonce` + the LWW merge be **deleted** — convergence is -the log + ADR 0001, not nonce-LWW. - -## 6. Risks - -- **Borsh/wire change** for how `SharedStorage` serializes into root state - (ref vs inline) — needs a migration/version story for existing `SharedStorage` - state (the `scenario-shared-storage` fixtures, `kv-store-with-shared-storage`). -- **Convergence under concurrent rotation** — once rotations are per-entity - actions, concurrent rotations converge by the ADR 0001 rule via the wrapper's - rotation log (the machinery exists); must be tested, it is the split-brain-prone - part. -- **Migration of in-flight root-state SharedStorage** to the ref form on first - load. - -## 7. Plan - -1. **P1 — wrapper-as-entity (suppress root-state inline).** Custom borsh / macro - special-case so the wrapper serializes as a ref; persist its body on its own - entity. No behaviour change to verification yet; assert root hashes match - sender/receiver (no divergence) on a 2-node e2e. -2. **P2 — emit verified per-entity Update on mutation.** Re-enable the disabled - `Interface::save(self)` in `insert`/`rotate_writers`; drop the root-state LWW - merge for the wrapper. Negative e2e: a **non-writer member's forged rotation is - rejected** (extend `kv-store-with-shared-storage`'s adversarial workflow). -3. **P3 — populate `signature_data`** writeback (item 3) + signer-hint already - present. -4. **P4 — concurrent-rotation convergence tests** on the #2552 harness + - migration of existing root-state SharedStorage to the ref form. - -## 8. Test plan - -- **Forged-rotation rejected (the proof):** a non-writer context member crafts a - writer-set rotation; honest node rejects it, writer set unchanged. Mirror of - the #2588 forged-data-write e2e. -- **No divergence:** legit rotation propagates, sender/receiver root hashes match - (this is the thing v2's dual-write broke). -- **Concurrent rotations converge** by ADR 0001. -- **Rotate-self-out** still works (regression — item 2 already landed). - -## 9. Relationship to #2590 - -#2590 (retroactive rotation revocation for guarded *collections*) needs the -wrapper's rotation log to be the authenticated anchor. This issue makes that log -authenticated and live; #2590 then points child entities at it so a rotation -revokes the whole subtree at the causal cut. So **#2230 item 1 is the prerequisite -for #2590** — do it first. diff --git a/docs/design/calimero-components.md b/docs/design/calimero-components.md deleted file mode 100644 index c567e41954..0000000000 --- a/docs/design/calimero-components.md +++ /dev/null @@ -1,575 +0,0 @@ -# calimero-components — CRDT-aware access-control components - -Status: **Partially Implemented** (tracked in #2687; supersedes #2557; op-aware ACL slice of -#2541). Targets the merged enforcement substrate: -#2588 (collection guarding), #2230/#2601/#2612 (authenticated rotation), -#2655 (SharedMember anchor / retroactive revocation), #2665 (concurrent-rotation -convergence). - -### Implementation status (as of 0.11.0-rc.2) - -| Phase | Status | PR / commit | Notes | -|---|---|---|---| -| P0 — field-id registry | ✅ shipped | #2544 | `TypeId AssignFieldId` registry | -| P1 — `Ownable` + `PermissionedStorage` + `Authorizer` seam | ✅ shipped | #2700 (`c444e8ef`) | `WriterSetAcl`, `OwnerAcl`; `only_owner`/`transfer`/`renounce` | -| P2 — `AccessControl` roles | ✅ shipped | #2700 (`c444e8ef`) | `grant`/`revoke`; `AccessControl::project_onto` (#2741) | -| OpMask (§8) — WRITE/DELETE/ADMIN bits | ✅ shipped | #2735/#2736/#2741 (`bbcc9f3a`, `99fe48b4`, `744eaeb1`) | `OpMask` in writer map; `ProtocolAuthorizer` + `grant_capability` | -| OpMask — INSERT vs UPDATE distinction | ⚠️ deferred | — | `v2_upsert` tag split; existence-at-cut check (§8.9); see scorecard | -| P3 — `Pausable` (advisory) | 🔲 not started | — | | -| P4 — `#[component]` macro sugar | 🔲 not started | — | | - -## 1. Motivation - -App authors keep hand-rolling the same authorization patterns: an owner key, a -set of admins, a pause switch. Today they write them by hand against -`SharedStorage`/`UserStorage`, and the easy way to write them is **wrong**: - -```rust -pub fn delete_entry(&mut self, k: String) -> app::Result<()> { - require!(env::executor_id() == self.owner, "not owner"); // advisory only - self.entries.remove(&k); - Ok(()) -} -``` - -That `require!` is **not a security boundary**. A malicious context member never -calls `delete_entry`; they run a patched node, craft a signed delta containing a -`DeleteRef` against the entry's entity, and gossip it. It reaches honest nodes -through the **sync/merge path**, which never runs the app's wasm. The check is -bypassed. - -`calimero-components` exists to make the *correct* pattern the *easy* pattern: -ship `Ownable`, `AccessControl`, and `Pausable` as thin facades over -writer-set-guarded storage, so protected data physically lives inside an entity -that is signature-verified at merge — and app authors can't accidentally store -it somewhere world-writable. - -## 2. The non-negotiable principle - -> **Enforcement lives at merge, never in the method.** - -A call-site guard (`only_owner()?`) is UX sugar: it fails fast with a good error -and saves a round-trip. The actual boundary is `Interface::verify_*_signature` -(`crates/storage/src/interface.rs:300`): for any `User` / `Shared` / -`SharedMember` entity, an unsigned action is `InvalidSignature`, and a signed -action must `ed25519_verify` against a key in the entity's writer set -(`interface.rs:368`). Deletes are guarded identically (`interface.rs:1481`). -`Public`/`Frozen` skip verification because there is nothing to guard. - -Therefore: a component is secure **iff the data it guards is stored inside a -`Shared`/`SharedMember`/`User` entity whose writer set encodes the policy.** The -component's whole job is to guarantee that placement and keep the API predicate -and the merge predicate in sync. - -## 3. Primitives recap (what we build on) - -| Storage type | Shape | Owner determined by | Transfer / rotation | Verified against | -|---|---|---|---|---| -| `User { owner }` | map of per-member self-owned slots | the writer (`executor_id`) | ❌ none | the one fixed owner | -| `Shared { writers }` | one guarded cell/collection | chosen explicitly | ✅ `rotate_writers` (signed, ADR 0001) | the writer set (any member) | -| `SharedMember { anchor }` | entries under a `Shared` anchor | inherited from anchor | ✅ rotate the anchor (retroactive) | anchor's writers, resolved at the causal cut | -| `Frozen` | immutable after freeze | n/a | ❌ | n/a (read-only) | - -Components are built on **`Shared`/`SharedMember`** (not `User`) because they need -**transferable ownership** and the path to **roles** — both of which only exist on -the writer-set side. `UserStorage` stays the right primitive for genuinely -per-member private data (each member owns their own slot, no transfer). - -## 4. Architecture - -### 4.1 The `Authorizer` seam - -One predicate, evaluated at two sites (API + merge), so they cannot drift. - -```rust -/// What an action wants to do, for op-granular policies (#2541). -pub enum Op { Read, Write, Delete, Admin } - -pub trait Authorizer { - /// Is `who` permitted to perform `op` on the guarded resource, given its - /// current writer set? Pure function of (who, op, writers) — no I/O, so the - /// same call is valid at the API call-site and conceptually mirrors what - /// merge enforces. An associated function (no `&self`): policies are - /// zero-sized markers carried as a type parameter, not stateful values. - fn authorize(who: &PublicKey, op: Op, writers: &BTreeSet) -> bool; -} - -/// Default: membership in the writer set authorizes any op. This is exactly -/// what merge-time `verify_*_signature` already enforces, so API and merge agree -/// by construction. -pub struct WriterSetAcl; -impl Authorizer for WriterSetAcl { - fn authorize(who: &PublicKey, _op: Op, writers: &BTreeSet) -> bool { - writers.contains(who) - } -} -``` - -As of #2736 (`ProtocolAuthorizer` + `grant_capability`) and #2735 (`OpMask` -merge-time enforcement), merge enforces **membership + op mask** (WRITE, DELETE, -ADMIN bits). `ProtocolAuthorizer` has shipped and resolves the causal op mask -from the anchor log at merge time — `WRITE` vs `DELETE` distinctions are now -merge-enforced. INSERT vs UPDATE (§8.9) remains advisory (deferred) until the -`exists_at_cut` query is implemented. - -### 4.2 `Ownable` — single transferable owner - -A newtype over `SharedStorage` constrained to a one-key writer set, with -ergonomic guards. - -```rust -pub struct Ownable { - inner: SharedStorage, -} - -impl Ownable { - /// New resource owned by `owner`. - pub fn new(owner: PublicKey, value: T) -> Self { /* SharedStorage::new({owner}) + insert */ } - - /// New resource owned by the installer (common case). - pub fn new_owned_by_caller(value: T) -> Self { - Self::new(env::executor_id().into(), value) - } - - pub fn owner(&self) -> Option { self.inner.writers().into_iter().next() } - - /// Fail-fast API guard. NOT the security boundary — merge is (see §2). - pub fn only_owner(&self) -> app::Result<()> { - let me: PublicKey = env::executor_id().into(); - if self.owner() == Some(me) { Ok(()) } else { Err(NotOwner.into()) } - } - - /// Read (anyone) / mutate (guarded at merge). - pub fn get(&self) -> app::Result<&T> { Ok(self.inner.get()?) } - pub fn get_mut(&mut self) -> app::Result<&mut T> { Ok(self.inner.get_mut()?) } - - /// Transfer ownership — signed, DAG-causal rotation. The thing `User` can't do. - pub fn transfer_ownership(&mut self, new_owner: PublicKey) -> app::Result<()> { - self.inner.rotate_writers(BTreeSet::from([new_owner]))?; - Ok(()) - } - - /// One-way: freeze ownership so the resource becomes immutable. - pub fn renounce_ownership(&mut self) -> app::Result<()> { /* freeze */ } -} -``` - -Security note: `transfer_ownership` is enforced at merge — a non-owner's forged -rotation delta is rejected (`forged_shared_rotation_rejected_at_merge`). -`only_owner()` is just the courtesy fail-fast. - -### 4.3 `AccessControl` — roles as writer sets - -A role *is* a `SharedStorage`-backed writer set; `grant`/`revoke` are rotations. -An admin role gates membership changes to the other roles. - -```rust -pub struct AccessControl { - /// role-name → guarded member set. Each role is its own anchor, so revoking - /// a member is one anchor rotation and is retroactive (#2655). - roles: UnorderedMap>>, - admin_role: String, -} - -impl AccessControl { - pub fn has_role(&self, role: &str, who: &PublicKey) -> bool { /* ... */ } - - /// Fail-fast guard; merge is the boundary. - pub fn only_role(&self, role: &str) -> app::Result<()> { /* has_role(role, executor) */ } - - /// Grant/revoke = rotate the role's writer set. Caller must hold admin_role; - /// enforced at merge because the role set is itself a guarded entity whose - /// writers are the admins. - pub fn grant(&mut self, role: &str, who: PublicKey) -> app::Result<()> { /* rotate += who */ } - pub fn revoke(&mut self, role: &str, who: PublicKey) -> app::Result<()> { /* rotate -= who */ } -} -``` - -Revoke is **remove-wins / retroactive** thanks to the `SharedMember` anchor model: -rotate the role's anchor and a revoked member's older signatures no longer verify -against the new causal writer set (#2655). This is the concrete win over a -hand-rolled `UnorderedSet` guarded only by an app method. - -**Do not** let `AccessControl` grow into a parallel permission system (#2541 -forbids it). Keep it a thin map of writer sets so it composes with the protocol -authorizer later. - -### 4.4 `Pausable` — advisory, honestly labeled - -```rust -pub struct Pausable { - paused_epoch: LwwRegister, // pause-dominant - unpaused_through: LwwRegister, -} -impl Pausable { - pub fn is_paused(&self) -> bool { /* paused_epoch > unpaused_through */ } - pub fn when_not_paused(&self) -> app::Result<()> { /* require !is_paused */ } -} -``` - -**Caveat that must ship in the docs:** there is no merge-time "reject because -paused" today. A patched node can still gossip a write while the group considers -itself paused. `when_not_paused()` is an **advisory** gate on the honest -execution path only. Merge-enforced pause requires freeze-rotation semantics and -is out of scope for v1. Pause-dominant epoch merge (a stale `unpause` can't lift a -newer `pause`) is the one real guarantee here, and it's a *convergence* property, -not an *authorization* one. - -## 5. The `#[component]` macro & the field-id prerequisite (#2544) - -### Problem -`#[app::state]` assigns each collection field a deterministic id by **string- -matching the field's type** (`is_collection_type` in -`crates/sdk/macros/src/state.rs`). A field typed `Ownable` or -`SharedStorage>` does **not** string-match → the inner storage -keeps a random id → it never merges with the same field on a peer → **silent -split-brain**. This is a P0 blocker for components wrapping collections. - -### Fix (build before components ship) -Replace the string-match with a **TypeId registry**, mirroring -`crates/storage/src/collections/rekey.rs`: - -- a trait `AssignFieldId` that types self-register in their constructor; -- the `#[app::state]` pass emits `assign_field_id_dyn(&mut self.field, "field")` - for **every** field (no-op if the type isn't registered); -- `SharedStorage`/`Ownable`/`AccessControl` register themselves and propagate the - field name to their inner collection. - -Consequence: `#[component]` becomes **optional sugar** (opt-in compile-time -`T: Component` armor + generated `only_*` helpers), not a correctness requirement. -This is shared infrastructure with #2544 — do it once, there. - -## 6. Examples - -### 6.1 Ownable config cell - -```rust -#[app::state] -pub struct Treasury { - config: Ownable>, // owner-gated settings blob - balances: UserStorage, // each member's own balance slot -} - -#[app::logic] -impl Treasury { - #[app::init] - pub fn init() -> Self { - Self { - config: Ownable::new_owned_by_caller(LwwRegister::new("{}".into())), - balances: UserStorage::new(), - } - } - - /// Owner-only. The require is fail-fast; the real gate is that `config` is a - /// Shared entity whose sole writer is the owner — a forged delta is rejected - /// at merge. - pub fn set_config(&mut self, json: String) -> app::Result<()> { - self.config.only_owner()?; // fail-fast UX - *self.config.get_mut()?.get_mut() = json; // enforced at merge - Ok(()) - } - - pub fn get_config(&self) -> app::Result { - Ok(self.config.get()?.get().clone()) // anyone may read - } - - /// Hand the treasury to a new owner. Signed rotation; a non-owner cannot. - pub fn transfer(&mut self, new_owner: PublicKey) -> app::Result<()> { - self.config.transfer_ownership(new_owner) - } - - /// Each member writes only their own balance — UserStorage stamps owner = - /// executor automatically, so nobody can forge another member's balance. - pub fn set_my_balance(&mut self, amount: u64) -> app::Result<()> { - self.balances.insert(amount)?; - Ok(()) - } -} -``` - -This example deliberately contrasts the two primitives: `config` is a single -**transferable** owned resource (`Ownable`); `balances` is **per-member -self-owned** data (`UserStorage`) that is never transferred. - -### 6.2 Role-gated collection - -```rust -#[app::state] -pub struct Forum { - acl: AccessControl, - // Writers of `posts` are the "editor" role's members; an editor's delete is - // enforced at merge, a non-editor's forged DeleteRef is rejected. - posts: SharedStorage>>, -} - -#[app::logic] -impl Forum { - #[app::init] - pub fn init() -> Self { - let me: PublicKey = env::executor_id().into(); - let acl = AccessControl::new(/*admin*/ me, &["editor"]); - Self { posts: SharedStorage::new(BTreeSet::from([me]), false), acl } - } - - pub fn add_editor(&mut self, who: PublicKey) -> app::Result<()> { - self.acl.grant("editor", who) // admin-gated rotation - } - - pub fn remove_editor(&mut self, who: PublicKey) -> app::Result<()> { - self.acl.revoke("editor", who) // retroactive — old writes lose validity - } - - pub fn delete_post(&mut self, id: u64) -> app::Result<()> { - self.acl.only_role("editor")?; // fail-fast - self.posts.get_mut()?.remove(&id)?; // merge rejects a non-editor's delete - Ok(()) - } -} -``` - -The `delete_post` case is the exact attack the issue raised: even if an attacker -skips `delete_post` and crafts a `DeleteRef` delta directly, honest nodes reject -it because the `posts` entries are `Shared`/`SharedMember` and the delete must be -signed by a current editor (`interface.rs:1481`). - -### 6.3 Test with `TestHost` - -```rust -#[cfg(test)] -mod tests { - use calimero_sdk::testing::TestHost; - use super::*; - - #[test] - fn only_owner_can_set_config() { - let mut app = TestHost::new(Treasury::init); // caller is owner - app.call(|s| s.set_config("{\"fee\":1}".into())).unwrap(); - assert_eq!(app.view(|s| s.get_config()).unwrap(), "{\"fee\":1}"); - } - - #[test] - fn transfer_moves_ownership() { - let mut app = TestHost::new(Treasury::init); - let alice: PublicKey = app.executor_id().into(); - app.call(|s| s.transfer(alice)).unwrap(); - // post-transfer the original owner's fail-fast guard now refuses - } -} -``` - -For the security property (forged delta rejected at merge), unit tests aren't -enough — it needs a 2-node merobox adversarial e2e in the shape of -`kv-store-with-shared-storage`'s workflow: a non-writer's forged write/delete is -applied locally, the honest node never converges to it. - -## 7. Implementation plan (phased, each its own PR) - -- **P0 — field-id registry (#2544).** ✅ **Shipped.** TypeId `AssignFieldId` registry replacing - the macro string-match; components self-register. -- **P1 — `Ownable` / `PermissionedStorage` + `Authorizer`/`WriterSetAcl` seam.** ✅ **Shipped (#2700).** Facade over - `WriterSetCell`; `only_owner`/`transfer`/`renounce`; `WriterSetAcl`/`OwnerAcl`. Unit tests - via `TestHost` + adversarial e2e. -- **P2 — `AccessControl`.** ✅ **Shipped (#2700, #2741).** Roles as `SharedStorage`-backed sets; admin-gated - `grant`/`revoke`; `AccessControl::project_onto` for role-based op-mask projection. -- **P3 — `Pausable` (advisory).** 🔲 Not started. Pause-dominant epoch merge; advisory caveat. -- **P4 — `#[component]` macro sugar + app migrations.** 🔲 Not started. -- **P5 — `ProtocolAuthorizer` (#2541).** ✅ **Shipped (#2736, #2735, #2741).** Op-granular bounds (WRITE/DELETE/ADMIN) enforced at merge via `OpMask` in the anchor writer map; `grant_capability` for per-principal mask assignment. INSERT vs UPDATE remains deferred (§8.9). - -## 8. Protocol-level extension (#2541): `OpMask` - -Components ship enforcing *membership* ("a writer may do anything"). #2541 is the -protocol-level extension that makes authorization **operation-aware** — enforced -at merge, bound into the signed delta — by attaching a per-principal `OpMask` to -the writer set and checking it after signature verification. Almost all the -substrate is already merged; the only genuinely new machinery is the op check. - -### 8.1 Which ops are enforceable at merge - -Merge only ever sees signed deltas, and a delta is one of three signed shapes -(`action.rs:164`): - -| Signed payload tag | Action | Enforceable at merge? | -|---|---|---| -| `v2_upsert` | `Add` / `Update` | ✅ | -| `v2_delete` | `DeleteRef` | ✅ | -| `v2_compare` | `Compare` | n/a (unsigned reconciliation) | - -`Read` is **not** merge-enforceable: in a replicated CRDT every node already holds -the bytes, so there is no apply step to reject. Real read-control needs encryption -(TEE / per-recipient keys), a separate mechanism. An `OpMask` `READ` bit may exist -as an advisory API gate or an "this scope is encrypted" marker, but it MUST NOT be -advertised as protocol-enforced. Enforceable ops: **Insert, Update, Delete, Admin.** - -### 8.2 The mask - -```rust -bitflags! { - pub struct OpMask: u8 { - const INSERT = 0b0000_0001; // create a new entity (Add on a fresh id) - const UPDATE = 0b0000_0010; // modify an existing entity (Update) - const DELETE = 0b0000_0100; // DeleteRef - const ADMIN = 0b0000_1000; // rotate writers / grant / revoke / change masks - - const APPEND = Self::INSERT.bits() | Self::UPDATE.bits(); // write, no delete - const WRITE = Self::INSERT.bits() | Self::UPDATE.bits() | Self::DELETE.bits(); - const FULL = Self::WRITE.bits() | Self::ADMIN.bits(); - } -} -``` - -The #2541 headline ("write but not delete") is `OpMask::APPEND`. - -### 8.3 Where masks live + `capability_at` - -The writer set gains a mask per key: - -```rust -// today -Shared { writers: BTreeSet, .. } -// extended -Shared { writers: BTreeMap, .. } -``` - -`capability_at(key, anchor_log, parents)` is `writers_at` with `OpMask` as the -resolved value — the same ADR-0001 causal fold over the anchor's rotation log -(merged, #2665). Because of the `SharedMember` anchor model, collection *entries* -commit only the anchor id (`action.rs` `hash_authorization_for_payload`), not -writers — so **masks live only at the anchor; entries are byte-unchanged.** Only -the anchor's representation changes. - -### 8.4 The merge check (the one new thing) - -```rust -// in verify_*_signature / apply_action, AFTER ed25519 verify succeeds: -let required = match verified_tag { - Tag::Upsert if entity_exists_at(cut) => OpMask::UPDATE, - Tag::Upsert => OpMask::INSERT, - Tag::Delete => OpMask::DELETE, -}; -let granted = capability_at(signer, anchor_log, delta.parents); -if !granted.contains(required) { - return Err(StorageError::Unauthorized); // "not for THIS op", not just "not a writer" -} -``` - -This *replaces* today's binary `writers.contains(signer)`. - -### 8.5 Grant / revoke = `ADMIN` op, retroactive by construction - -A grant is an `ADMIN`-op action appended to the anchor log, authorized iff the -caller holds `ADMIN` at the cut (verified at merge like any rotation). `revoke = -grant(who, OpMask::empty())`. Because `capability_at` folds at the operation's -causal position (#2655 semantics): writes causally *before* a revoke stay valid; -writes *after* it are checked against the reduced mask → rejected. Retroactive -revocation falls out of the representation — no history rewrite. - -### 8.6 Concurrent conflicting grants → intersection (least privilege) - -Two concurrent entries setting different masks for one key merge by bitwise AND: - -```rust -fn merge_concurrent(a: OpMask, b: OpMask) -> OpMask { a & b } // revoke-wins, fail-safe -``` - -Deterministic, order-independent, default-deny — matches #2541's conflict rule. -Must resolve the *causal* set (reuse `writers_at` discipline), never insertion -order (the #2673 trap). - -### 8.7 Back-compat + wire - -- A key with no explicit mask resolves to `OpMask::FULL` → today's "in the set ⇒ - anything" preserved exactly. -- The **anchor** `Shared` writer set is committed into its signed payload, so - `BTreeSet→BTreeMap<_,OpMask>` changes that hash → a wire/format change **at - anchors only** (version tag in `entities.rs`); entries unaffected. - -### 8.8 Status scorecard - -| OpMask bit | Status | -|---|---| -| `DELETE` vs `WRITE` | ✅ **shipped** (#2735/#2736) — merge-time enforcement via `ProtocolAuthorizer`; op mask in anchor writer map | -| `ADMIN` | ✅ **shipped** (#2736) — `grant_capability` appends to anchor log as `ADMIN`-op action | -| `INSERT` vs `UPDATE` | ⚠️ deferred — needs `exists_at_cut` query; `v2_upsert` tag split optional hardening (§8.9) | -| `READ` | ❌ not merge-enforceable — needs encryption; advisory only | -| wire change | anchors only (writer map: `BTreeSet → BTreeMap`); entries unchanged | - -### 8.9 What `INSERT` vs `UPDATE` specifically needs - -`WRITE` vs `DELETE` is free because `v2_upsert` and `v2_delete` are already -distinct signed payloads. `INSERT` vs `UPDATE` is *not* — both ride `v2_upsert` -(`action.rs:164`) — so to require `INSERT` for creates and `UPDATE` for edits the -verifier must decide, for an incoming upsert, **whether the target already -exists**, and decide it identically on every node. - -**The load-bearing mechanism: existence at the causal cut.** Every delta carries -its DAG parents, which pin an immutable causal position. Define *exists-at-cut* = -"an `Add` for this id is reachable from those parents and not superseded by a -tombstone." Because the parents are fixed, this is a **pure function of the -delta** — order-independent, identical on every node, no race. (A naive "does it -exist *now*" check against the local head is the wrong thing: it differs by apply -order → two nodes reach different authorization verdicts → split-brain.) Then: - -```rust -let required = if exists_at_cut(id, delta.parents) { OpMask::UPDATE } else { OpMask::INSERT }; -``` - -This is the same family of causal query as `writers_at` / `membership_status_at`: -fold the entity's lifecycle (`Add`/`Delete`) along the DAG up to the cut. The new -cost is one causal reachability query per upsert verify (amortizable via the -per-entity index). - -**Concurrency falls out for free.** Two concurrent `Add`s of the same id each see -"not present at *my* cut" → both are INSERTs (both need `INSERT` in the mask), -then merge per CRDT. An edit whose creating `Add` is concurrent (not reachable -from its cut) is likewise an INSERT at its cut. No special-case rule, because each -delta is judged at its own fixed parents. - -**The `v2_upsert` tag split is optional hardening, not a requirement.** Splitting -the tag into `v2_insert` / `v2_update` lets the signer *declare* intent, but the -tag alone can't be trusted — an `INSERT`-only signer could label an edit as an -insert — so it still has to be validated against `exists_at_cut`. The existence -check is what actually enforces the boundary; the tag is defense-in-depth (and a -signed-payload format change, so version-gated) worth adding only if we want the -intent committed for auditing/replay-hardening. - -Cost/benefit: this buys "append-only / create-but-not-edit" (audit logs, -claim-once registries). Until a concrete use case wants it, ship `APPEND` as -`INSERT|UPDATE` (no existence query, no split) and keep `WRITE`/`DELETE`/`ADMIN` — -which cover the #2541 headline gap — as the v1 granularity. - -## 9. Open decisions - -1. **`Ownable` representation** — single-writer `SharedStorage` (recommended, for - rotation/transfer) vs `User{owner}` (no transfer). Leaning `Shared`. -2. **`revoke` semantics** — remove-wins tombstone (recommended, retroactive via - #2655) vs add-wins. Lock with the #2541 owner so `AccessControl` and the - protocol authorizer agree. -3. **`Op` surface** — final enum shape; must match #2541's signed-action op bound - so the seam is a no-op swap. -4. **`Pausable` scope** — ship advisory-only in v1, or block on freeze-rotation - merge enforcement? Recommended: advisory v1, labeled. - -## 10. Known constraints / footguns - -- **Don't enforce in the method.** Every guard is fail-fast only; the data must - live in a guarded entity or there is no security. -- **`SharedStorage>` requires `V: Mergeable`** — `String` is - not; use `LwwRegister`. -- **Collection mutation goes through `get_mut`**, which re-stamps the writer - domain on the value entry each call (sidesteps the persistence round-trip) — - components must route mutation there, never a back-door `set`. -- **Local-causal resolution under concurrent rotation (#2673)** is a liveness/UX - wrinkle, not a security gap — merge-time `writers_at` is the boundary. Components - inherit this; don't paper over it with extra local checks. -- **One `SharedStorage` field = one independent writer set.** A component holding - several must rotate each explicitly; name such methods per the field they touch. - -## 11. References - -- Enforcement: `crates/storage/src/interface.rs:300` (verify), `:1481` (delete). -- `SharedStorage` API: `crates/storage/src/collections/shared.rs`. -- `UserStorage`: `crates/storage/src/collections/user.rs`. -- Example app: `apps/kv-store-with-shared-storage/src/lib.rs`. -- Re-key registry to mirror for #2544: `crates/storage/src/collections/rekey.rs`. -- Related issues: #2544 (field-id), #2541 (fine-grained perms), #2230/#2655/#2665 - (rotation substrate, merged), #2673 (local causal resolution). diff --git a/docs/design/shared-collection-guarding.md b/docs/design/shared-collection-guarding.md deleted file mode 100644 index e850ff3e5c..0000000000 --- a/docs/design/shared-collection-guarding.md +++ /dev/null @@ -1,131 +0,0 @@ -# Design — `SharedStorage` guards the whole subtree (with rotation) - -| | | -|---|---| -| **Status** | Partially implemented (static writers) — see *Current implementation status* below | -| **Date** | 2026-06-01 | -| **Owner** | Storage / Node sync | -| **Problem** | `SharedStorage` compiles but only gates its own wrapper entity; the collection's child entries default to `StorageType::Public` and are world-writable. A wrapped collection looks protected and isn't. | -| **Decision** | Keep one type (`SharedStorage`); make it propagate the writer domain into a collection value's children, **with rotation**. | -| **Reuses** | per-entity rotation log + `writers_at` causal resolution (ADR 0001 / #2266/#2267), `effective_writers` apply hook (`interface.rs`). | - -> Illustrative code; not final signatures. - -## Current implementation status - -This doc describes the **full target design, including rotation**. What is actually -**shipped today** (PR #2588) is the **static-writer** subset: - -- **Implemented:** propagation by *inline writer-set copy* — every child entry - inherits the collection's own element domain on insert (`Collection::insert` + - the per-map inserts), and `SharedStorage::get_mut` re-establishes that domain - before in-place mutation (re-derived from the persisted `writers` on every call, - so it survives reload). A guarded collection's entries are stamped - `Shared{writers}` and verified at merge. -- **NOT implemented (target design, not in this PR):** the `domain_anchor` field - on `StorageType::Shared` (§3), the per-child anchor registry (§4), and - node-layer anchor resolution (§5) — i.e. **anchor inheritance**. The shipped - code uses the inline copy this doc labels "rejected" for the *rotation* case; - inline copy is correct and sufficient for **static** writers, which is the - shipped guarantee. Anchor inheritance is required only for **retroactive - rotation revocation**, tracked in **#2590** (rotation is forward-only until then; - see the `get_mut` rotation note). -- **Phases:** §7 P1's "child entry is `Shared`" test is present, **and** the - adversarial proof — a 2-node e2e in `kv-store-with-shared-storage` where the - writer's guarded-map entry converges but a non-writer's entry is rejected at - merge (the writer node never sees it). Enforcement is live on the delta apply - path: `delta_store` resolves `effective_writers` for every `Shared` entity and - `apply_action` validates the signature, so a guarded collection's child entries - (which sync as their own entities, unlike the root-state-borsh wrapper) are - verified against the writer set. - -## 1. Goal - -```rust -doc: SharedStorage> -``` -must mean: **every entry of the map (and nested descendants) is writable only by the current writer set, verified at merge** — and `rotate_writers` re-resolves who can write, convergently across nodes. - -Today `SharedStorage` is a *guarded cell*: it stamps `Shared{writers}` on its own entity and stores `value: T` inline, so a scalar `T` (e.g. `LwwRegister`) is fully covered. But a collection `T`'s entries are **separate entities** created `Public` by `UnorderedMap::insert` (`unordered_map.rs:288`) — the wrapper never touches them. This closes that gap. - -## 2. The machinery we build on (don't reinvent) - -- **Rotation log, per entity.** `calimero_storage::rotation_log` stores a per-`entity_id` log of writer-set rotations. The node loads it via `load_rotation_log_direct(ctx, entity_id)` (`delta_store.rs:648`). -- **Causal resolution.** `rotation_log_reader::writers_at(log, causal_parents, happens_before)` returns the writer set as of an op's causal cut, implementing ADR 0001's full merge rule (reachable-latest, HLC tiebreak, signer-bytes tiebreak). **Pure, already tested.** -- **Apply hook.** `interface.rs` verification takes `effective_writers: Option`: `Some` = node pre-resolved via `writers_at`; `None` = fall back to the entity's inline `storage_type.writers` (snapshot path). Verification is `ed25519_verify(sig, writer.digest(), payload)` against that set. - -The merge rule for "who can write" already exists and is correct. The only thing missing is **routing a collection's children to the right rotation log.** - -## 3. Core idea: domain anchoring (not per-child writer copies) - -Two ways to guard children: - -- **Inline copy (rejected):** stamp every child `Shared{writers: }`. Rotation must re-stamp **all N children** → O(N) writes and N independent concurrent-rotation merges. Fragile, exactly the split-brain trap. -- **Anchor inheritance (chosen):** each child is stamped `Shared{ domain_anchor: Some(A), writers: }` where `A` = the `SharedStorage` entity's id. The **one** rotation log lives at `A`. At verification, a child with `domain_anchor = Some(A)` resolves its writers from **A's** log via `writers_at(A_log, child_op.parents, …)`. Rotation appends one entry to **A's** log — **O(1), children untouched.** - -Anchor inheritance reuses `writers_at` verbatim (same ADR 0001 rule), so concurrent insert-vs-rotate and rotate-vs-rotate inherit the existing, tested semantics — no new merge math. - -### Representation change -`StorageType::Shared` gains an optional anchor: -```rust -Shared { - writers: BTreeSet, // the anchor's own set; on a child, a fallback cache - domain_anchor: Option, // None = this entity owns its domain (today's behaviour) - signature_data: Option, -} -``` -- `domain_anchor: None` → unchanged from today (the `SharedStorage` cell, and the anchor entity itself). -- `domain_anchor: Some(A)` → "I'm a member of A's writer domain; resolve writers from A's rotation log." - -Backward-compatible: existing Shared entities deserialize with `domain_anchor: None` (borsh field add → needs a versioned/`Option` default; see §6). - -## 4. Propagation: how `SharedStorage` stamps children - -`SharedStorage` is generic; it can't call "stamp your children" only when `T` is a collection without specialization. Reuse the **`TypeId` registry pattern from `rekey.rs`**: collections register a "apply writer domain" thunk in their constructor; `SharedStorage` calls `apply_domain_dyn::(&mut value, anchor, writers)` which **no-ops for non-collection `T`** (scalars are inline in the anchor → already covered). - -- A domained collection carries `(anchor, writers)` and passes `StorageType::Shared{ writers, domain_anchor: Some(anchor), .. }` into `insert_with_storage_type` instead of `Public`. -- **Recursion:** a nested collection value re-keyed under a domained parent inherits the same anchor (extend the existing `rekey` walk, which already descends nested collections relative to a parent id). - -This is the same shape as the deterministic-ID and rekey registries — consistent, source-compatible, no trait bound on `T`. - -## 5. Verification & rotation flow - -**Write a child** (insert): child entity stamped `Shared{ writers: current, domain_anchor: Some(A) }`, signed by the executor (must be a current writer). - -**Apply/merge a child delta:** node resolves `effective_writers`: -``` -if child.storage_type.domain_anchor == Some(A): - log = load_rotation_log(A) // anchor's log, not child's id - effective = writers_at(log, child_op.parents, happens_before) -else: - (today's path: child's own id / inline writers) -``` -then `interface.rs` verifies the signature against `effective`. **One line of routing change** in the node's effective-writers resolution. - -**Rotate:** `shared.rotate_writers(new)` (existing path) appends to **A's** log, signed by a current writer. Children are not rewritten; their next write resolves the new set automatically. Concurrent rotations converge by ADR 0001 (already implemented in `writers_at`). - -## 6. Risks / must-handle - -- **Wire format.** Adding `domain_anchor` to `StorageType::Shared` changes borsh layout → version the metadata or use a trailing `Option` with a default-on-missing decoder; verify ABI/snapshot compatibility. -- **Anchor not yet synced.** A child delta may arrive before the anchor's rotation log → reuse the existing orphan/queued-delta handling (interface.rs already stores orphaned children); resolution retries once the anchor lands. -- **Anchor id stability.** The anchor is the `SharedStorage` entity id, which is deterministic (field-name derived) — good. Children must capture it at insert. -- **Snapshot path** (`effective_writers: None`): child falls back to its inline `writers` cache — must be kept ~current, or snapshots resolve via the embedded rotation-log snapshot. - -## 7. Phased plan - -1. **P1 — representation + propagation (fixed writers).** Add `domain_anchor` to `StorageType::Shared` (default `None`, back-compat). Add the `apply_domain_dyn` registry; `UnorderedMap` (first) stamps children with the anchor. `SharedStorage` sets the domain. **No rotation yet.** Tests: a child entry is `Shared`, and an **adversarial non-writer delta to a child is rejected at apply** (the proof this guards the subtree). -2. **P2 — rotation.** Route child verification to the anchor's rotation log in the node layer; wire `rotate_writers` on the domained collection. Convergence tests: concurrent insert-vs-rotate, rotate-vs-rotate (ADR 0001), across 2–3 nodes (merobox). -3. **P3 — all collections + nesting.** `UnorderedSet`/`Vector`/`RGA`/`Sorted*` register; nested collections inherit the anchor through the rekey walk. Map-of-maps test. -4. **P4 — sync/snapshot/back-compat hardening.** Snapshot leaf push, orphan-before-anchor, wire-version migration, ABI guard. - -## 8. Test matrix (the part that prevents split-brain) - -- **Adversarial:** non-writer-signed delta to any child → rejected on every node (P1). -- **Convergence:** concurrent `insert` (writer A) ∥ `rotate` (writer B) → all nodes converge, op evaluated at its cut (P2). -- **Rotate ∥ rotate:** ADR 0001 outcome identical across nodes (P2). -- **Nested:** map-of-maps, deep child guarded + converges (P3). -- **Snapshot / orphan-before-anchor:** child arriving before its anchor resolves correctly (P4). - -## 9. Relationship to #2541 - -This *is* a slice of #2541's "collection-scope" capabilities, implemented concretely on the existing Shared machinery: a collection-level writer ACL evaluated at the causal cut and enforced at merge. It stays write-granular (all-or-nothing per writer); #2541's op-granularity (`Insert` vs `Delete`) layers on top later via the same anchor. Build it here so it composes with `calimero-components` (`Owned`/`AccessControl` become typed facades over a *genuinely* guarded collection), not as a parallel system. diff --git a/docs/design/unified-causal-log-cutover-plan.md b/docs/design/unified-causal-log-cutover-plan.md deleted file mode 100644 index b5a3d2edbe..0000000000 --- a/docs/design/unified-causal-log-cutover-plan.md +++ /dev/null @@ -1,305 +0,0 @@ -# Unified causal log — cutover plan - -Status: **blueprint**. The additive foundation is merged in #2775 (the -`op`/`authz`/`projection`/`op-adapter` crates, the scope-isolation property -harness, op coverage, and both halves of the data-write decision proven -fold-equivalent — see `unified-causal-log-p5-decisions.md`). This document -sequences the **cutover**: replacing the three separate per-context op stores -(data DAG, governance DAGs, rotation log) and the `state_hash`/`root_hash` -convergence signal with one op-log + `ScopeState` projection + `scope_root`, -then deleting the old folds. - -> **Note on references.** This plan names files by **path + stable symbol** -> (struct/function), not line numbers — the codebase is actively refactored -> between slices, so line numbers would be stale by the time each slice lands. -> Verify the symbol against `HEAD` when starting a slice. - -## Ground rules - -- **Flag-day, no backwards compat.** A coordinated redeploy; nodes re-bootstrap - and re-sync. No migration tool, no re-projection. The on-disk format already - broke in #2745, so re-bootstrap is already required. The operator-facing "wipe - + re-sync" path reuses the existing `resync_context` admin endpoint - (`crates/server/src/admin/handlers/context/resync_context.rs`, exposed via the - client in `crates/context/primitives/src/client/mod.rs`) rather than a bespoke - one. (This is the §9.7/§9.8 decision in `unified-causal-log-p5-decisions.md`, - recorded 2026-06-12.) -- **Each behavioral slice is its own PR, gated on a divergence-zero e2e run** - (merobox / scaffolding-e2e / Sync-regression), run by the maintainer. -- **Author + unit-test in this repo; the maintainer runs CI/e2e.** Slices are - ordered so each compiles, passes unit tests, and is independently reviewable. - (This division of labor for the cutover PR series was agreed 2026-06-17; it is - about *who runs what*, separate from the design decisions above.) - -## Why the order is "augment the signal → unify the store → cut the decision → delete" - -The security property (a hash-neutral ACL/membership rotation can't be hidden) -is delivered the moment the convergence signal folds ACL + membership in — -**before** the store is unified. So C1 (the `scope_root` signal) ships the -security win early and low-risk. The store unification (C2/C3) is mechanical -plumbing once the projection is authoritative for the signal. The decision cut -(C4) and deletion (C5) are last, when nothing else reads the old folds. - ---- - -## C0 — `scope_root` shadow (pre-C1 de-risk, observe-only) - -**Status: in progress (2026-06-24).** A de-risking precursor to C1, chosen over -flipping straight to the new signal — same shadow→flip discipline that carried -F4b/F5. C0 changes **no sync decision**: it computes `scope_root` alongside the -existing `root_hash`, exchanges it, and **logs** when the two signals would -*disagree about convergence* — i.e. `entities_root` (the current `root_hash`) -agrees between two peers but `scope_root` differs. That disagreement is exactly -the hash-neutral ACL/membership rotation the current signal is blind to (the -root cause behind the rotation split-brain family: stale-heads, clear() -tombstone-blindness, #2607 verified-but-divergent). C0 proves the new signal -catches it in real e2e *before* C1 makes it load-bearing. - -**Where.** Reuse the #2607 end-of-session convergence re-query in -`crates/node/src/sync/hash_comparison_protocol.rs` (`query_peer_current_root` → -`DagHeadsResponse`). The responder additionally computes and returns its -`scope_root`; the initiator computes its own and compares. **`root_hash` / -`root_hash_verified` keep driving every decision** — `scope_root` is logged, never -acted on. - -**Computing a context's `scope_root`.** `entities_root` = the existing storage -Merkle `root_hash` for the context (unchanged, `get_local_root_hash_for_context` -/ `DagHeadsResponse.root_hash`). The ACL + governance come from the **maintained -projection**: resolve `context → group → namespace` (`get_group_for_context` + -`NamespaceRepository::resolve`) to the namespace `ScopeId`, then -`ScopeProjections::scope_root_for(scope, entities_root)` (new, thin) returns -`states.get(scope).map(|s| s.scope_root_with_entities(entities_root))`. `None` -(scope not folded yet) ⇒ skip the shadow comparison for that session — never a -false divergence on a cold projection. - -**Wire.** Add `scope_root: Option` to `DagHeadsResponse` -(`crates/node/primitives/src/sync/wire.rs`). `Option` so a node that can't -resolve/fold the scope sends `None` and the initiator skips — no false signal. -Safe under the flag-day rule (all nodes redeploy together; e2e runs one build), -and it is the **same field C1 promotes** to the authoritative compare, so the -throwaway is one `Option` unwrap. - -**Marker.** Log `scope_root_shadow_divergence` (gate marker, like the F5 planes) -at WARN when `local_entities_root == peer_entities_root && local_scope_root != -peer_scope_root` (the blind spot), and a quieter `debug!` for the inverse -(`entities` differ — already caught by `root_hash`, expected mid-sync). The e2e -divergence gate greps the WARN marker; a hit on a *converged-entities* scenario -is the proof C0 is looking for, and on a *should-converge* scenario it's a real -bug the old signal hid. - -**e2e gate.** A **hash-neutral-rotation canary**: two nodes reach an identical -`entities_root`, one rotates its writer set; assert the shadow fires -`scope_root_shadow_divergence` until the rotation propagates and `acl_hash` -agrees, then stops. Plus: existing concurrent-rotation/governance scenarios show -**zero** shadow divergence at steady state (no false positives). C1 promotes the -signal only once this canary is green. - -## C1 — `scope_root`: fold ACL + groups into the convergence signal - -**Landing incrementally (merge-gated on C0's canary):** -- **C1a (drafted 2026-06-24)** — the **HashComparison end-of-session verdict** - (`hash_comparison_protocol.rs`, the #2607 re-query path C0 instrumented). When - both peers resolve a `scope_root` it alone decides `root_hash_verified`; cold - projection / non-group context falls back to the bare entity compare (no - context drops below the pre-C1 check). This is the kernel security win on the - general sync path. -- **C1b (drafted 2026-06-24)** — **LevelWise** brought to HashComparison parity: - it now does the same end-of-session peer re-query (the #2407 comment said it - *needed* "a second handshake round-trip" to tell real divergence from drift — - this is that round-trip, reusing `query_peer_current_root`) and decides on - `scope_root` with the same entity fallback. Closes the blind spot on the - wide-shallow-tree path. -- **NOT folded into `scope_root`, deliberately:** - - **Snapshot boundary** (`snapshot.rs`) stays entity-root-based. The snapshot - streams the *data plane*; its boundary check is streaming-integrity ("did the - responder's entity state drift mid-stream"). Folding governance in would abort - a perfectly valid entity snapshot whenever governance moves independently - (over-conservative). Post-snapshot convergence — including authorization — is - verified by the next HC/LevelWise session's `scope_root` verdict (C1a/C1b). - - **Protocol selector's `None` arm** ("roots match ⇒ no sync") stays entity-root - based. Governance reconciles on its *own* independent tick (the - `namespace_sync` beacon), so a selector that skips state-sync on equal entity - roots never strands a governance divergence — it's pulled by governance sync, - and the following state-sync's `scope_root` verdict confirms convergence. - Folding `scope_root` here would only add redundant state-sync work. - -**Goal.** Replace the bare entity `root_hash` on the wire and in comparison with -`scope_root`, where - -``` -scope_root = SHA-256( entities_root ‖ acl_hash ‖ governance_hash ) -``` - -— where each input is a fixed **32-byte** value (`entities_root` is the storage -Merkle root; `acl_hash` and `governance_hash` are SHA-256 outputs). The plain -concatenation is collision-free **only because all three are fixed-width**; this -invariant is load-bearing (see the risk register). Implemented by -`calimero_op::scope_root` and `ScopeState::scope_root_with_entities` (already on -master from #2775). -`entities_root` stays the **existing, proven storage Merkle `root_hash`** — we -do not re-hash entities. `acl_hash` and `governance_hash` (membership + admin + -policy + live subgroups) come from the projection. - -This is the kernel security win: a divergent writer set or membership becomes a -divergent root, so sync can never declare "done" while authorization disagrees. - -**Touch points (verify symbols against HEAD):** -- Compute: `ScopeState::scope_root_with_entities` is the combiner (already - landed). C1 makes the projection's `acl_hash`/`governance_hash` available at - the sync sites — either from a live `ScopeState` (if C2 lands first) or, as - interim glue, derived on demand from the existing governance store + rotation - logs. -- Ship: the `root_hash` field on `DagHeadsResponse` carries `scope_root` - (`crates/node/primitives/src/sync/wire.rs`); likewise the snapshot boundary - hashes (`SnapshotStreamRequest` / `SnapshotBoundaryResponse`). -- Compare: every local-vs-peer root comparison switches to `scope_root` — the - protocol selector, level-wise sync, hash-comparison protocol, and the - post-sync reconciler (all under `crates/node/src/sync/`). -- The HC entity tree-walk itself is unchanged (it still reconciles entities by - the storage Merkle); only the *top-level convergence decision* uses - `scope_root`. Divergence localized to acl/groups (equal `entities_root`, - different `scope_root`) routes to a governance/ACL pull, not an entity walk. - -**e2e gate.** Concurrent-rotation + governance scenarios still converge, **and** a -new **hash-neutral-rotation canary** proves a divergent ACL can no longer hide: -add (or extend) a `scaffolding-e2e` scenario where two nodes reach an identical -`entities_root` but a node performs a writer-set rotation, and assert sync stays -active (does not declare "converged") until the rotation has propagated and the -`acl_hash` agrees. This subsumes the #2607 "verified-but-divergent" guard for the -ACL/membership dimension. - -## C2 — one `DagStore` per context (the unified store) - -**Goal.** Collapse `DagStore>` + `DagStore` + -`DagStore` + the rotation log into a single `DagStore` per -context, applied by one `UnifiedApplier` that folds each op into `ScopeState` -(data → storage apply as today; acl/membership/admin → projection state). `Op` is -the `calimero-op` envelope; the `op-adapter` encoders become the *construction* -path (local ops) and the wire decode path (remote ops). - -**Touch points (symbols, in `crates/node/src/delta_store.rs` and -`crates/context/src/governance_dag.rs`):** -- New `UnifiedApplier: DeltaApplier` replacing `ContextStorageApplier` - (delta_store), `GroupGovernanceApplier` + `NamespaceGovernanceApplier` - (governance_dag). Data ops still drive `__calimero_sync_next`; acl/membership/ - admin ops drive `ScopeState::apply` plus the storage rotation / membership - writes that remain authoritative until C5. -- Persistence: one keyspace for `Op` rows keyed `[ContextId ‖ OpId]` (repurpose - `Column::Delta`). `dag_heads` (in `Column::Config`) now tracks the unified DAG. - From this slice on, the old governance-op and rotation keyspaces are **no - longer written** *and no longer read* (C2+ code reads only the unified op-log); - their on-disk **column removal** is deferred to C5 (with the code that - references them). So stale old-format rows left by pre-C2 code are **inert** — - never read, just dead bytes until C5 drops the column. A node that missed the - flag-day redeploy (e.g. offline during it) has *no* unified op-log and must - run `resync_context` to rebuild it before it can participate; its stale old - columns are ignored, not interpreted. There is no path where C2+ code mixes - old-column data with the unified op-log. -- `load_persisted_deltas` / `persist_cascaded_deltas_and_update_heads` - (delta_store) operate on `Op`. - -**e2e gate.** Full lifecycle (create context, add/remove members, rotate writers, -concurrent writes) converges; partial-replication isolation holds (a non-member -never receives a scope's ops). - -## C3 — projection-authoritative reads - -**Goal.** Reads that today hit the three stores (writer resolution, membership -status, admin/policy) now read `ScopeState::acl_view_at(parents)`. The old -resolvers (`rotation_log::resolve_local`, governance `acl_view_at`, -`membership_status_at`) become thin shims over the projection, then are removed -in C5. Fold-equivalence for both halves is already proven, so this is a -mechanical swap behind the same call signatures. - -**e2e gate.** This slice introduces no new *behavior*, so its gate is a -**regression check** that the swap is transparent — but it must exercise a read -path C2's scenarios don't: add a scenario where membership/writers are resolved -on a **snapshot-replay or DAG-catchup** path (not just live gossip apply), on a -2-node cluster after a concurrent rotation, and assert the projection-served -result matches what the old resolver returned pre-swap. Without that, an -identical-to-C2 gate couldn't distinguish a C3 regression from a passing C2. - -## C4 — `authorize` is the decision (the cut) - -**Goal.** Replace `authorize_delta_at_edge` -(`crates/node/src/handlers/state_delta/verify.rs`) + `writers_at_authenticated` -(storage) with one `authorize(op, ScopeState::acl_view_at(op.parents))` at every -apply site. This is the single security decision; the two-layer split (membership -gate + per-object writer gate) collapses into one fold. - -**Coexistence with the #2763 pull-side gate.** The pull-side membership gate -added on master in #2763 remains the **live** authorization through C1–C3 -(C1–C3 change the *signal* and the *store*, not the decision). C4 replaces both -it and `authorize_delta_at_edge` **atomically within this one slice** — there is -never a window where two gates run concurrently with potentially different -outcomes. If C4 is delayed, the #2763 gate simply stays live; nothing regresses. - -**e2e gate (the big one):** divergence==0 across concurrent-rotation, governance -add/remove, the snapshot/HC/level paths, **and** an explicit group-remove -scenario that makes the #19 closure *verifiable*: after a group-remove op is -applied, assert that a subsequent op authored in that group's plane is **rejected -by `authorize`** (no authorless plane survives). **C4 does not merge until this -is green.** - -## C5 — delete the old folds (~3,500 LOC) - -Once nothing reads them, in dependency order: - -1. **Persistence layer first** — drop the old on-disk keyspaces no longer written - since C2: the `Column::Group` op-log rows and the rotation-log keyspace, plus - the `state_hash` field on `SignedGroupOp`/`SignedNamespaceOp` and - `compute_group_state_hash` / `snapshot_context_state_hashes`. -2. **Resolver / apply code** — `crates/storage/src/rotation_log.rs`, - `crates/node/src/sync/rotation_log_reader.rs`, - `crates/context/src/governance_dag.rs`, `apply_local_signed_group_op`, - `apply_signed_namespace_op`, `membership_status_at`-as-fold. -3. **The `op-adapter` crate**: delete `crates/op-adapter`, **and** remove it from - the workspace `members` list in the root `Cargo.toml` and from `deny.toml` - (otherwise `cargo build` fails on a missing member). - -**On losing the equivalence proofs.** The fold-equivalence tests are inherently -*transitional*: they assert "the new projection resolves the same writer set / -membership as the **old resolver**." Once C5 deletes the old resolvers -(`resolve_local`, `membership_status_at`), there is nothing left to compare -against, so those tests retire *with* the code they compare to — keeping them -would not even compile. The durable post-cutover safety net is **not** these -proofs but the **convergence + scope-isolation property harness** in -`calimero-projection` (`testing` feature), which is independent of the old -resolvers and is **not** deleted. Before deleting `op-adapter`, confirm that -harness still covers the properties the equivalence tests were guarding (it -does today: per-scope convergence + non-member isolation); if any unique case is -only in an `op-adapter` test, port it into the projection harness first. - -group-remove (#19) closes here structurally. - -## P6 (separate epic, after C5) - -Collapse `HashComparison` / `Snapshot` / `LevelWise` / `protocol_selector` / -governance catch-up / `rotation_log_reader` into one per-scope sync engine -(head-accumulator → pull-by-ancestry → re-project; Merkle-diff + checkpoint as -strategies), per-shard + membership-gated. This surface grew with the migrations -work (chained catch-up, parent-pull short-circuit, the peer-auth gate) — re-survey -before starting. - -## Risk register - -- **`entities_root` ≠ projection entities hash.** Resolved by *keeping* the - storage Merkle as `entities_root` and only folding acl/governance into - `scope_root` (C1). The projection does not re-hash entity state. (Corollary: - `scope_root_with_entities` must be called with the storage Merkle root, **not** - `ScopeState`'s own entity hash — the type system can't distinguish two - `[u8; 32]`s, so this is a documented caller contract.) -- **`scope_root` concatenation assumes fixed-width inputs.** `SHA-256(a ‖ b ‖ c)` - is only collision-free because `a`/`b`/`c` are each exactly 32 bytes — there is - no length prefix or separator. Any future change that makes a component - variable-length (a truncated/extended hash, a different digest) silently breaks - collision-resistance. Enforce the widths at the `calimero_op::scope_root` - boundary (it takes `[u8; 32]` arguments — keep it that way) and re-check this if - the digest ever changes. -- **Concurrent equal-HLC rotation tiebreak.** `ScopeState` uses `op_id` - (content-addressed → identical on all nodes → deterministic convergence, proven - by the harness). The old `resolve_local` signer-digest tiebreak dies with it in - C5; moot under flag-day (no mixed-version window). -- **No e2e in authoring env.** Every behavioral slice (C1–C4) carries an explicit - e2e gate run by the maintainer; unit + property tests are the authoring-time - signal. diff --git a/docs/design/unified-causal-log-flip-plan.md b/docs/design/unified-causal-log-flip-plan.md deleted file mode 100644 index d65aea71a3..0000000000 --- a/docs/design/unified-causal-log-flip-plan.md +++ /dev/null @@ -1,118 +0,0 @@ -# Unified causal log — the cutover flip (do NOT merge until e2e-green) - -This branch carries the **decision flip**: making the unified-op projection the -authoritative answer to "is this author permitted to write at its governance -cut", then deleting the old gates. It is the security boundary, it is flag-day, -and per the cutover plan it **must not merge until a divergence-zero e2e run is -green** (a merobox/scaffolding run the maintainer executes — it cannot be run in -the authoring environment). - -The safe, additive groundwork is already on master / in #2783: -- the `op`/`authz`/`projection`/`op-adapter` crates + isolation harness (#2775); -- the live per-scope projection fed from governance + raw rotations, the - feed-verification shadow-compare, and **op-log retention + causal-cut - `acl_view_at`** (#2783). - -What remains here, in order, each its own commit and the last two gated: - -## F1 — bridge: share the projection node↔context (additive) -The projection lives in `ContextManager` (context crate); the data-write -decision is orchestrated in `authorize_delta_at_edge` (`node`/verify.rs). Share -one `Arc>`: -- `run.rs` creates it, passes it to `ContextManager::with_scope_projections(..)` - (a new builder that replaces the internally-created registry) **and** stores - it on `NodeState`. -- The governance/ACL feeds then populate the shared registry; the node decision - site can read it. -Pure plumbing, nothing decides against it yet — zero behavioral risk. - -## F2 — persistence + startup backfill (prerequisite, additive) -The projection is in-memory and only fills from feeds since process start, so a -just-restarted node has an empty projection. Authorizing against an empty -projection = deny = broken. Before the flip, the op-log must be **persisted** -and **rebuilt on startup** (replay the retained ops, like `load_persisted_deltas` -seeds the data DAG). Until F2, the projection can only be a *shadow* (act on the -live decision), never authoritative. - -## F3 — decision-site shadow-compare (additive, acts on live) -At `authorize_delta_at_edge`, also compute the projection's verdict via -`ScopeProjections::acl_view_at(scope, governance_dag_heads)` + -`calimero_authz::authorize`, and compare to the live `acl_view_at`/`writers_at` -decision behind the `unified_projection_divergence` marker. **Still act on the -live decision.** This is the literal authorize-vs-live compare; its e2e output -(zero divergence) is the gate for F4. - -## F3.5 — namespace-wide governance resolution (prerequisite for F4) -The shadow-compare surfaced TWO problems, the second deeper than it first looked. - -**1. The encrypted `GroupOp` plane wasn't folded.** Admin-push `MemberAdded` / -`MemberRemoved` / `MemberRoleSet` / `MemberJoinedViaTeeAttestation` ride the -namespace DAG encrypted, so the projection couldn't see admin-added members while -the live decision (which decrypts via the keyring) authorized them. - -**2. The bigger one — the per-scope log fragmented the namespace-wide cut.** The -live system keeps ONE governance DAG per namespace, interleaving namespace-root -ops and every group's membership ops on a single parent chain, and a data write -cites namespace-wide `governance_dag_heads`. The projection keyed its op-log -per-group, so `acl_view_at(group_scope, heads)` truncated the ancestry walk at -the first node that wasn't in that group's log (a namespace-root op, another -group's op, or an unmodeled op) — orphaning every membership op behind it. Even -a member couldn't see *itself* once any later cross-scope op became the head. - -The fix addresses both at the root: -- **namespace-scoped governance** — all of a namespace's governance ops fold into - one log keyed by `ScopeId::from(namespace_id)`; membership for a group is read - from the folded view's `groups[group]`. `member_at_cut` resolves the namespace - and walks the whole namespace ancestry (mirroring the live resolver's - `prefix_walk_membership`), so the cut never truncates. -- **graph-complete ancestry** — EVERY namespace op becomes a node, even ones the - projection doesn't model (out-of-model Root ops, undecryptable group ops): they - fold as `OpPayload::Noop` but keep the parent chain unbroken so the walk can - pass through them. `op_from_namespace_op` always returns a node. -- **encrypted plane folded** — the namespace apply handler and the backfill walk - both decrypt an applied/persisted `NamespaceOp::Group` (read-only, via - `decrypt_group_op` — no re-run of the mutation) and fold its membership variant - at the carrying namespace delta's id/parents. Backfill decrypts the ops for - groups this node belongs to, so cold-start and the op-emitting originator are - covered once the namespace is (re)walked. - -Residual (narrow): an originator that emits a *new* group op AFTER its namespace -was already backfilled won't live-fold it (its own ops don't pass through the -namespace apply handler, and backfill runs once). Re-walk-on-new-head or a -post-publish feed closes it; tracked here. Until divergence is zero across e2e, -the divergence step stays **informational**; it flips back to a hard gate as the -last step here. - -## F4 — the flip (gated: do not merge until divergence==0 e2e) -Once divergence reached zero across e2e (the hard gate), the flip lands in two -sub-steps so the security boundary moves under continuous validation rather than -in one leap. - -**F4a — projection as a load-bearing co-authorizer (DONE).** At the data-write -edge's `Authorized` arm, the projection's `member_at_cut` must now CONCUR: a -`Some(false)` DENIES the write (rejects) instead of merely logging the divergence -marker. This only ANDs with the live authorize, so it can never grant a write -live rejected (the still-unvalidated permissive direction) — it cannot -over-authorize. It can only additionally reject, and with forward divergence at -zero a wrong denial would both fail an e2e convergence scenario and trip the hard -gate. `None` (no projection answer) defers to live. RwLock makes this a cheap -concurrent read; the prior `Mutex` + an `if let`-scrutinee lock self-deadlocked -(fixed). - -**F4b — sole authority (next).** Validate the inverse/permissive direction -(projection authorizing where live rejects) with a reject-arm cross-check — now -safe as a refresh-free RwLock read — and, once zero, replace -`authorize_delta_at_edge` + `writers_at_authenticated` outright with -`authorize(op, projection.acl_view_at(parents))` as the single decision. Subsumes -the #2763 pull-side membership gate. - -## F5 — delete the old folds (~3,500 LOC), after F4 soaks -`rotation_log.rs`/`rotation_log_reader.rs`, `governance_dag.rs`, -`apply_local_signed_group_op`/`apply_signed_namespace_op`-as-fold, -`membership_status_at`, the `state_hash` field + `compute_group_state_hash`, and -the `op-adapter` crate (its job — proving equivalence — is done). Persistence -columns retired. group-remove (#19) closes here structurally. The durable -post-cutover safety net is the projection's convergence + isolation property -harness, not the (now-deleted) equivalence proofs. - -See `unified-causal-log-cutover-plan.md` for the full rationale. diff --git a/docs/design/unified-causal-log-p5-decisions.md b/docs/design/unified-causal-log-p5-decisions.md deleted file mode 100644 index 3697a6f734..0000000000 --- a/docs/design/unified-causal-log-p5-decisions.md +++ /dev/null @@ -1,261 +0,0 @@ -# Unified Causal Log — Phase 5/6 open decisions (§9) - -Status: **DRAFT for ratification.** The Phase-5 migration (route storage + -governance through one `ScopeState::apply`, wire one `authorize`, delete -`state_hash`) cannot be written correctly until these are settled — each shapes -`OpPayload`, `authorize`, `ScopeState`, the key/scope tree, or the cutover. - -For each: the question, the options, and a **recommended default** grounded in -how the system behaves *today* and in what the P5 scaffold (`crates/op`, -`crates/authz`, `crates/projection`) already assumes. Ratify or override. - -The scaffold + the convergence/scope-isolation harness already exist -(`feat/unified-op-p5`); these decisions unblock turning them into the live -migration. - ---- - -## §9.1 Concurrent-revoke semantics — ✅ DECIDED: **causal-honor** - -When a member/writer is revoked *concurrently* with an op they authored, does -that op survive? **Causal-honor** (chosen 2026-06-12): an op authored before -the revocation *in causal order* stays valid, regardless of the order a -receiver observes the revocation. - -- Matches today's `writers_at(parents)` / forward-only `acl_view_at` walk — - lowest behavioral risk, no convergence-semantics change. -- Already implemented in the scaffold: `ScopeState::acl_view_at(log, parents)` - folds only the ancestry of an op's parents. -- Rejected: *revoke-wins* (retroactive invalidation — harder to make - deterministic across peers) and *quarantine* (new held state + resolution - policy — most work). - ---- - -## §9.2 Who may rotate an object's writers? Is "owner" distinct from "writer"? - -**Recommendation: owner = the `OpMask::ADMIN`-capability holder on the object — -NOT a separate owner identity.** A `SetWriters` op is authorized iff its author -holds `ADMIN` on that object in the ACL view at its causal cut. - -- Matches today's rule: a writer-set rotation is accepted only when its signer - held `ADMIN` in the prior set (`writers_at_authenticated`'s ADMIN-chain). -- Already in the scaffold: `AclView::is_owner(author, object) = - may(author, object, ADMIN)`; `authorize(SetWriters) → is_owner`. -- Bootstrap: the object's creator seeds the initial writer set with `ADMIN` - (the genesis `SetWriters`, self-authorizing — see §9.3 note). -- Alternatives rejected: *any current writer may rotate* (too permissive — any - WRITE holder could lock others out); *a separate owner identity distinct from - the capability set* (extra state, no current need). - -**Open sub-point to confirm:** may a non-writer **group admin** rotate an -object's writers (admin override), or only an object `ADMIN` holder? Recommend -**object-`ADMIN` only** for data objects (least authority); group/root admins -act through the membership/admin planes, not by reaching into object ACLs. - ---- - -## §9.3 OpMask lattice — required-bit map, grant/revoke authority, monotonicity - -**Recommendation: keep today's 3-bit lattice and the scaffold's mapping.** - -- Bits: `WRITE` (0b001), `DELETE` (0b010), `ADMIN` (0b100); `FULL` = all; - `NONE` = 0. `contains` = superset test. (Unchanged from - `calimero_storage::entities::OpMask`.) -- Op → required capability (already in `authz::required_mask_for` + `authorize`): - `Put → WRITE`, `Delete → DELETE`, `SetWriters → ADMIN` (owner), - `Member* → group-admin`, `Admin*/Policy*/SubgroupCreated → root-admin`. -- **Grant/revoke:** an `ADMIN` holder sets the entire writer/cap map for an - object via `SetWriters` (grant = add an entry, revoke = drop/lower it). -- **Monotone?** **No.** `SetWriters` replaces the set, so capabilities can be - revoked; safety comes from causal-honor (§9.1) + per-entry authentication, - not from monotonicity. (A monotone-only lattice would forbid revocation, - which the product needs.) -- Future-compat: bits are a `u8`; new capabilities (e.g. `GRANT` distinct from - `ADMIN`) are an additive bit, not a wire break. - ---- - -## §9.4 Scope tree & key boundaries — which levels are separate key domains? - -**Recommendation: a scope is a key domain iff it *restricts* membership.** - -- **Root governance scope (`gov-N-root`)** — one per namespace, members = all - namespace participants. Always shared between any two members; carries the - owner/admin + root policy + the *non-restricted* group structure. Its key is - the namespace-wide key. -- **Restricted subgroup → its own scope + its own key.** Matches today's - per-group `GroupKeyring`: a restricted subgroup's data ops *and* the - governance ops about it live only in its member-only scope. -- **Open subgroup (inherited membership via `CAN_JOIN_OPEN_SUBGROUPS`) → - inherits the parent scope/key** (no separate restriction, so no separate key - domain). Membership is derived by the ancestor walk, not a separate key. -- **A context** is a scope under its owning group, keyed by that group's key. - -Net: separate key domain ⇔ restricted membership. This preserves the existing -key model and Invariant 0 (a non-member of a restricted scope never holds its -key, never receives its ops, never computes its root). - ---- - -## §9.5 Existence- vs content-hiding for restricted subgroups (drives §3.4) - -**Recommendation: existence-hiding.** A restricted subgroup's existence, -membership, and root must not be derivable by a non-member. - -- A restricted child scope's `scope_root` **never** appears inside a visible - parent's root (no upward leak, §3.4) — a non-member can't even tell the - subgroup exists. -- Matches today's posture (restricted-subgroup governance lives in the - member-only scope, not the visible namespace root). -- Rejected: *content-only hiding* (existence visible, payloads encrypted) — - leaks the subgroup's existence + membership-set size, a weaker privacy - guarantee than the product currently offers. -- Consequence for the migration: `gov-N-root` reflects only owner/root-policy + - non-restricted structure; the sync `shared` set is computed from - *stream-authenticated* membership so a restricted scope is never offered to a - non-member (it's never named on the wire). - ---- - -## §9.6 Encryption granularity - -**Recommendation: confirm — `Op.payload` is ciphertext at rest under the -scope's symmetric key** for the data and membership/admin arms; op metadata -(id, scope, parents, author, hlc, signature, expected_scope_root) stays -cleartext so non-content sync/causality works. - -- Matches today's group-key encryption of state deltas + governance ops. -- The scaffold currently stores cleartext payloads (noted in `crates/op`); the - migration wraps payloads with the scope key at the boundary, leaving the op - envelope + projection logic unchanged. - ---- - -## §9.7 Ambition — ✅ DECIDED: **full P5 + P6** - -Proceed through the merge-core (P5) and sync-core (P6) unification — not -P2-only or through-P4-only. (Chosen 2026-06-12.) - ---- - -## §9.8 Upgrade / cutover — RESOLVED main, **one sub-question open** - -**Resolved (design §0.1):** flag-day, no versioned coexistence — a coordinated -redeploy, no mixed-version cluster. (The P2–P4 wire breaks already landed on -this basis in #2745.) - -**✅ RESOLVED 2026-06-12: wipe + re-sync — NO re-projection, no backwards -compatibility.** ("I don't care about backwards compatibility.") The P5 -migration ships as a clean flag-day: nodes re-bootstrap on the new engine; no -data-preservation/re-projection step is built. This matches the norm already -in effect — #2745 itself changed the on-disk format (rotation-log side-store → -`UnorderedMap` children; `GovernancePosition` borsh layout; Shared-anchor -hashing), so a re-bootstrap across this line of work is already required. - ---- - -## What unblocks once ratified - -§9.2/§9.3 → finalize `authorize` + the `OpMask` mapping (mostly confirming the -scaffold). §9.4/§9.5 → define `ScopeId` derivation + the key-domain boundary + -the visibility rule for cross-scope parent edges. §9.6 → the encryption wrap -boundary. §9.8-sub → whether the migration ships a re-projection step or a -wipe. With these settled, the P5 migration (task #16) can begin on a fresh PR -off master, validated against the scope-isolation harness + e2e at each step. - ---- - -# Slice 2 design — wiring `authorize` into the live path - -Wiring the one `authorize` in surfaces three semantic questions the per-plane -folds never had to answer together. Recommended resolutions below; ratify, then -the implementation is a **shadow → cutover** (run new alongside old, assert -agreement on live traffic, then flip — never a blind swap of the security -boundary). - -## S2.1 — granularity: per-delta membership gate vs per-op capability - -Today two *different* checks run: a **coarse** per-delta gate ("is the author a -member of the context's group at the cut?", `authorize_delta_at_edge`) **and** a -per-action signature/writer check (`writers_at_authenticated`). The unified -`authorize` asks one **finer** question per op: `Put → may(author, entity, -WRITE)`. These don't trivially match — a member who isn't an entity writer would -pass the coarse gate but fail `may`. - -- **Recommendation: default-write = membership.** A non-restricted context - grants every member `WRITE` on its entities; the ACL is *seeded from the - member set*. A restricted object carries an explicit `SetWriters`. Then - `may(author, entity, WRITE)` reproduces today's "members can write" for - default contexts and is strictly finer for restricted ones. The migration - seeds each context's ACL from its membership at re-projection (or, under the - §9.8 wipe, from genesis membership as contexts re-bootstrap). - -## S2.2 — where the `AclView` comes from during shadow - -`authorize(op, acl_view_at(parents))` needs an `AclView` at the cut, which -ultimately comes from a live `ScopeState` (slice 3). But slice 2 can run -*before* `ScopeState` is the live projection. - -- **Recommendation: shadow off the current resolvers.** In slice 2 build the - `AclView` from the *existing* resolution (`writers_at_authenticated` → `acl`, - `acl_view_at`/membership → `groups`, governance meta → `root_admin`), run - `authorize` against it, and **compare** to the current decision behind a - divergence metric — acting on the current decision. This validates - `authorize ≡ the old folds` on live traffic with zero behavioral risk, and - decouples slice 2 from slice 3. Slice 3 then swaps the `AclView` source to - `ScopeState::acl_view_at` (same interface). - -## S2.3 — OpPayload coverage for the live op stream *(COMPLETE)* - -Coverage is now closed in `calimero-op-adapter`. Every op the live stream can -carry either maps to an `OpPayload` arm or is **out-of-model by design** with a -documented rationale. The dividing line is the unified `authorize` decision, -which is **role**-based (`is_group_admin` ⇔ `role == Admin`, `is_owner` ⇔ root -admin) — exactly like today's cross-DAG `membership_status_at` check. An op is -in-model iff it moves that decision (or the data/ACL/scope-tree state under it). - -**In-model — mapped:** -- data: `Action::Add/Update` → `Put`, `DeleteRef` → `Delete` (`Compare` is a - sync hint, not a state change → `None`). -- ACL: `RotationLogEntry` → `SetWriters`. -- membership: `GroupOp::MemberAdded/MemberRoleSet` → `MemberAdded`; - `MemberRemoved/MemberLeft` → `MemberRemoved`; - `MemberJoinedViaTeeAttestation` → `MemberAdded` (attestation evidence is the - admission gate's, not the projection's); `RootOp::MemberJoined` → `MemberAdded` - (group_id + role decoded off the admin-signed invitation, no escalation); - `RootOp::MemberJoinedOpen` → `MemberAdded` (Member). -- ownership: `GroupOp::TransferOwnership` → `AdminChanged` (owner ⇔ ADMIN, §9.2; - authored in the group's scope, so it sets that scope's root admin). -- admin/policy: `RootOp::AdminChanged` → `AdminChanged`, `PolicyUpdated` → - `PolicyUpdated`. -- scope tree: `RootOp::GroupCreated` → `SubgroupCreated`, `GroupReparented` → - `SubgroupReparented`, `GroupDeleted` → `SubgroupDeleted` (one per cascaded - scope; `restricted` resolved from policy at the live call site). - -**Out-of-model by design (`None`) — never enters the auth decision:** -- key transport: `KeyDelivery` (§9.6 — payloads are scope-key-encrypted; the - key delivery rides its own channel), `Noop` (padding). -- capability refinement: `MemberCapabilitySet`, `DefaultCapabilitiesSet`, - `ContextCapabilityGranted/Revoked` — a separate, deferred permission layer - (§9.3 keeps the simple lattice). Capabilities never gate `authorize`. -- app/upgrade/migration config: `UpgradePolicySet`, `TargetApplicationSet`, - `GroupMigrationSet`, the `Cascade*` ops — owned by app-version / migrations-v2. -- metadata + policy knobs: `GroupMetadataSet`, `MemberMetadataSet`, - `ContextMetadataSet`, `SubgroupVisibilitySet`, `TeeAdmissionPolicySet`, - `MemberSetAutoFollow`. -- context↔group binding: `ContextRegistered/ContextDetached`, `GroupDelete` — - `authorize` derives a context's group from that binding at auth time (the P4 - context→group lookup), so it lives in that index, not a scope's `ScopeState`. - -The adapter uses `_ => None` catch-alls (not exhaustive matches) so it keeps -compiling as master grows the governance enums; any genuinely auth-relevant new -variant must be armed explicitly, and slice-3's coverage/divergence tests would -surface one that slipped through. - -## Slice 2 implementation order (once ratified) -1. `AclView`-from-current-resolvers builder + `authorize` shadow at the - gossip-receive site, divergence metric, act-on-current. (Additive-ish.) -2. Extend shadow to the other auth sites (snapshot-replay, drain, DAG-catchup, - manager, writer-plane resolve). e2e: divergence metric stays zero. -3. Cut over: `authorize` becomes the decision; old folds deleted in slice 5. diff --git a/docs/documentation-roadmap.md b/docs/documentation-roadmap.md deleted file mode 100644 index 4f4d52d0f3..0000000000 --- a/docs/documentation-roadmap.md +++ /dev/null @@ -1,337 +0,0 @@ -# Calimero Documentation Roadmap — "Next Level" - -Status: **proposal / for discussion** · Owner: docs · Last updated: 2026-06-26 - -This is the plan to take Calimero's documentation from "good content, dated and -inconsistent presentation" to a best-in-class developer documentation product. -It is grounded in a full audit of the existing docs, a design/UX audit, a -coverage-gap analysis against the whole core, and a benchmark of how Stripe, -libp2p/IPFS, Tailscale, Linear, Matrix and the Diátaxis framework set the bar. - ---- - -## 1. Where we are (honest assessment) - -We are not starting from zero. The audit found **~59 documents, roughly two-thirds -genuinely strong on content** — the crate deep-dives, `membership-and-leave`, -the new protocol reference, the host-functions and storage references. The -problem is **not the content; it is everything around it.** - -Three structural problems drag the whole thing down: - -1. **No information architecture.** 47+ pages accreted by audience-blind growth - and a per-PR auto-update bot. There is no audience routing, no Diátaxis - separation (tutorials vs how-to vs reference vs explanation are mixed on the - same pages), and overlapping pages that contradict each other - (`tee-mode` vs `tee-fleet-ha` vs `auto-follow` vs `protocol/governance`; - `local-governance` vs `protocol/governance`; `dag.html` superseded by - `protocol/operations`). - -2. **The presentation looks dated ("ugly").** The design audit is blunt: failing - contrast (lime `#a5ff11` and `--text-dim` fail WCAG AA in several places), - body text too small (14px), monospace overused in UI chrome, card-on-card - visual fatigue with no whitespace to rest the eye, no real type scale, an - inconsistent diagram palette (hand-SVG vs the new engine vs d3, 6+ colour - sets), a half-baked light theme, no code-copy buttons, no syntax - highlighting, tables that overflow on mobile, and a search that builds its - index on first keystroke. - -3. **The authoring & freshness model is fragile.** Content is hand-written HTML - (high friction, no PR-friendly diffs), and an external bot regenerates pages - per-PR with no editorial pass — which is exactly why nav labels were literal - PR titles. There is no signal of what is *stable* vs *draft* vs *blueprint*. - -Coverage gaps compound it: **no SDK API reference, no CLI reference -(merod/meroctl have ~60 routes), no admin-API reference (~60 endpoints), no -install/quickstart for operators, no operational runbooks**, and example apps -documented conceptually with no annotated code. - -The goal of this roadmap is to fix all three at once: **re-architect, re-skin, -and re-tool — without throwing away the strong content or the bespoke animated -diagrams that make us distinctive.** - ---- - -## 2. Vision - -> One documentation product, four clearly-routed audiences, every page in a -> single Diátaxis mode, on a fast modern platform, with one consistent visual -> language and a small set of unforgettable interactive diagrams. - -Concretely, a reader should: - -- land on a page that asks **"who are you / what do you want to do"** and routes - them (Tailscale/Vercel pattern), not a table of contents; -- never confuse a tutorial with a reference — each page declares its mode; -- read the **protocol spec** as a normative, versioned document distinct from the - friendly **concepts** explainers (libp2p/Matrix pattern); -- find anything via an instant **⌘K search**; -- see **one diagram language** throughout, with a handful of step-controllable - animated showpieces; -- always know whether what they're reading is **Stable / Draft / Planned** and - **which version** it pins to. - ---- - -## 3. Target information architecture - -Split by **audience track first**, then apply Diátaxis modes within each track. -This kills the "everyone fights over Getting Started" failure. - -| Track | Reader | Diátaxis emphasis | Sections | -|---|---|---|---| -| **Build** | App developers writing WASM apps | Tutorials + How-to + Reference | Quickstart → How-to guides → **SDK & API reference** → examples | -| **Operate** | Node operators & deployers | How-to + Reference | Install/run → **config reference** → **CLI reference** → networking/NAT → observability & troubleshooting/runbooks | -| **Protocol** | Reimplementers, auditors | Explanation + **normative Reference** | **Concepts** (explainers) → **Spec** (normative, versioned) → message/state/storage reference | -| **Contribute** | Core developers | Explanation + How-to | Architecture explainers → crate map → dev setup → testing → RFC/release process | - -Rules we enforce in review: - -- **Concepts ≠ Spec.** A friendly Concepts explainer for CRDTs, causal auth, - gossip/sync; a separate normative Spec an independent implementer can build - from. Concepts link *into* the spec, never duplicate it. (The current - `protocol/` reference becomes the spine of the Protocol track; the explanatory - `concepts.html`/`system-overview.html` become Concepts.) -- **Each page declares one mode** (a badge): Tutorial / How-to / Reference / - Explanation, plus a status (Stable / Draft / Planned) and, for the spec, a - version. -- **One canonical "Start here" per track**, surfaced by the routed landing page. - -### Page disposition (keep / refactor / write / retire) - -- **Keep & re-skin:** crate deep-dives, `membership-and-leave`, `system-overview`, - `concepts`, `migrations`, `glossary`, host-functions & storage references, the - new `protocol/*` chapters. -- **Refactor / merge:** consolidate the four TEE pages into one "TEE & Fleet" - reference; fold `local-governance` implementation detail into the Contribute - track and point the protocol story at `protocol/governance`; refresh stale - `app-lifecycle` (multi-service bundles) and `dependency-explorer`. -- **Write (high-value gaps):** SDK API reference; collections reference; admin-API - reference (from the ~60 routes); merod/meroctl CLI reference; operator install - + quickstart + runbooks; app-upgrade guide; xcall & blob guides; contributor - getting-started. -- **Retire / absorb:** `dag.html` (superseded), thin `tee-mode.html`, duplicate - entry points; standardise the 12 app READMEs against one template. - ---- - -## 4. The pivotal decision — platform & tooling - -This forks the entire execution plan, so it is called out first. - -The benchmark is unambiguous that our real liabilities are *authoring in HTML* -(no Markdown diffs, high friction) and *maintaining bespoke nav/search/versioning*. -Two viable paths: - -### Option A — Migrate content to **Astro Starlight**, keep our identity (recommended) -- Content becomes **Markdown/MDX** → PR-friendly diffs, contributors can write. -- **Built-in Pagefind search, dark mode, responsive nav, ⌘K, perfect-Lighthouse - baseline** replace our bespoke JS — less code to own, faster site. -- **Astro islands let us mount the existing animated-SVG engine** as components on - otherwise-static pages — we keep the diagrams, we don't rewrite them. -- We port our dark theme + accent into a Starlight custom theme so it still looks - like *us*, just fixed (contrast, type scale, spacing). -- **Cost / gap:** versioning isn't native (community plugin or Git-branch - strategy); a migration project to move ~50 pages of HTML → MDX. - -### Option B — Stay bespoke, do a deep in-place redesign -- Keep `architecture/*.html` + `nav.js` + `seq-diagram.js`; invest only in a CSS - design-system rewrite and IA reshuffle. -- **Pro:** no migration; we already control everything. -- **Con:** we keep paying the HTML-authoring and bespoke-search/nav/versioning - tax forever; "next level" findability (⌘K, pre-built index) and Markdown - authoring remain DIY. - -**Recommendation: Option A (Starlight, SVG engine as islands)** unless -cross-version spec switching is a day-one hard requirement — in which case -Docusaurus (native versioning, heavier JS) is the alternative. This supersedes -the earlier "keep bespoke HTML" call now that the bar is "next level," but it is -the owner's decision; everything below is written to work under either option. - ---- - -## 5. Design system overhaul (the "ugly" fixes) - -Independent of platform, the visual language gets a single coherent spec. -Highest-leverage moves, in order: - -1. **Reading & contrast.** Body text 16px / line-height ~1.7, measure capped at - ~70ch. Lighten `--text-dim` (`#8b949e`→`~#a8b1ba`) and reserve the lime accent - for active/focus/links only — everything else in a tight neutral gray scale. - Audit every pair to WCAG AA. -2. **Type scale.** Adopt a real modular scale (~1.2×) and apply it consistently; - lighter, larger headings (weight 300–500, drop the −1px tracking). Monospace - only for code/commands, never for nav, breadcrumbs, or subtitles. -3. **Whitespace over borders.** Replace card-on-card chrome with section dividers - + negative space; one dominant corner radius (~12–16px); barely-there - elevation on hover instead of border-flashing. -4. **One diagram palette.** Extract diagram colours to CSS variables; the bespoke - engine and any Mermaid theme share one palette, one arrow/lifeline vocabulary, - one label type scale. -5. **Code presentation.** Syntax highlighting (Rust/JSON/TOML), copy-to-clipboard - on every block, per-heading anchor links ("copy link"), horizontally - scrollable tables on mobile. -6. **Callouts for the four voices.** A small fixed component set: `Note` - (explanation), `Warning` (footgun), and a distinct **`Normative`/MUST** style so - spec implementers can scan obligations (Matrix convention). -7. **Navigation feel.** ⌘K palette + pre-built search index; sticky sidebar with - scroll-spy; real breadcrumbs; prev/next within a track; an on-page outline for - long spec pages; respect `prefers-reduced-motion`. -8. **Complete the light theme** and ship both to AA. - -Target "feel": Linear/Tailscale restraint — one accent as rare punctuation, -generous whitespace, authority through typography not ornament. - ---- - -## 6. Diagram strategy — two tiers - -Keep the bespoke engine as an asset; stop hand-drawing routine figures. - -- **Tier 1 — Mermaid (diagrams-as-code) for ~90% of figures.** Sequence diagrams - (handshakes, sync rounds), state machines (peer/protocol states), simple - topology. Lives in the repo, diffs in PRs, never goes stale, any contributor - edits it. (D2 is the upgrade path if we outgrow Mermaid's layout.) -- **Tier 2 — bespoke animated SVG for 5–10 hero showpieces:** the gossip/sync - walkthrough, the CRDT merge convergence, the causal-auth / state-divergence - flow, the life-of-an-operation. Step-controllable (play/pause/scrub), static - fallback under reduced-motion. -- **One visual language across both tiers** (palette, arrows, labels). - -This reconciles the earlier "all animated hand-SVG" preference with -maintainability: hero diagrams stay hand-built and animated; the long tail moves -to Mermaid so it can't rot. - ---- - -## 7. Content plan by track (what to write) - -**Build** — Quickstart (15-min app); How-tos (persist with a CRDT map; handle -peer events; emit & handle events; xcall another context; blobs); **SDK reference** -(all `#[app::*]` macros, the collection types `UnorderedMap`/`Vector`/`Authored*`/ -`Frozen`/`Shared`/… with semantics, the `env` host functions); annotated example -walkthroughs replacing the thin example pages. - -**Operate** — Install (binaries vs source vs Docker); Quickstart (init→run→join); -**config reference** (refresh, all sections); **CLI reference** (merod + meroctl, -every subcommand/flag with examples); **admin-API reference** (the ~60 endpoints, -ideally OpenAPI-generated); networking/NAT/relays/bootstrap; observability -(metrics + dashboards); **runbooks** (upgrade a node, rotate keys, recover from -divergence, back up/restore); troubleshooting. - -**Protocol** — Concepts explainers (scopes, CRDTs, causal auth, gossip/sync); -the normative **Spec** = the current `protocol/*` chapters promoted to versioned, -normative status with MUST/SHOULD language and the byte-level appendices; fill -the spec gaps the audit found: **blob protocol, TEE attestation protocol, xcall -message format, migration/upgrade protocol, identity/key rotation, sync error & -divergence recovery, capability inheritance edge cases.** - -**Contribute** — Contributor getting-started (build, test, debug with merodb); -crate-map narrative; actor-communication contracts; testing strategy (unit/ -integration/e2e/sync-sim); ABI stability rules; RFC/ADR + release process. - ---- - -## 8. Auto-update pipeline integration - -The per-PR doc bot is useful for keeping *generated* crate/reference pages fresh, -but it must never touch *authored* narrative. Policy: - -- **Authored zone** (tracks, concepts, spec, design system) — hand-written, - fenced **out** of the bot (paths-ignore), reviewed like code. -- **Generated zone** (crate deep-dives, API/CLI/endpoint references) — keep - auto-generating, but from structured sources (rustdoc, the CLI's own `--help`, - the server's route table / OpenAPI) rather than free-form LLM HTML, and render - into the design system. -- The bot's job becomes "regenerate the reference tables," not "write prose." -- **Immediate action regardless of platform:** fence `architecture/protocol/**` - out of `doc-update.yaml` before more authored work lands on `master`. - ---- - -## 9. Phased execution - -**Phase 0 — Decide & de-risk (this week).** Pick the platform (§4). Fence the -authored zone from the bot. Lock the design tokens (type scale, colour, spacing) -and the diagram palette. Stand up a spike of the chosen platform with our theme + -one ported page + one mounted animated diagram. - -**Phase 1 — Skeleton & shell.** Build the four-track IA, the routed landing page, -the page-mode/status/version badges, ⌘K search, and the design system. Migrate -(or re-skin) the strong existing pages into the shell. Outcome: the site *looks* -next-level and is correctly organised, even where content is still thin. - -**Phase 2 — Close the blocking gaps.** SDK reference, CLI reference, admin-API -reference, operator install/quickstart. These unblock real users today. - -**Phase 3 — Protocol spec hardening.** Promote `protocol/*` to a versioned -normative spec; add the missing protocol chapters (blobs, TEE attestation, xcall, -migration, rotation, divergence recovery); convert routine figures to Mermaid; -build the 5–10 hero animated diagrams. - -**Phase 4 — Operate depth & polish.** Runbooks, troubleshooting, observability, -deployment; app-README standardisation; example walkthroughs; light-theme AA; -performance pass. - -**Phase 5 — Sustain.** Generated-zone pipeline from rustdoc/CLI/OpenAPI; docs CI -(link-check, contrast-check, build); versioning workflow; a short docs style -guide so quality holds. - ---- - -## 10. Success criteria - -- A new app developer ships a working app from the Quickstart in **< 30 min**. -- An operator stands up and joins a node from docs alone, **no source reading**. -- An independent implementer can build a conformant node from the **Protocol spec** - without reading Rust. -- Every page: one Diátaxis mode, a status badge, AA contrast, ⌘K-findable. -- Lighthouse ≥ 95 across the board; search is instant; both themes pass AA. -- One diagram language; no contradictory/duplicate pages; no stale-by-months pages. - ---- - -## Appendix — source analyses - -This roadmap synthesises four audits (doc inventory & critique; design/UX audit; -coverage-gap matrix vs the codebase; best-in-class benchmark). Key exemplars -referenced: Diátaxis (mode separation), Stripe (three-pane reference, single-page -deep links), libp2p/IPFS (Concepts vs Spec), Matrix/Noise (normative spec style), -Tailscale/Linear/Vercel (visual restraint), Astro Starlight (tooling). - ---- - -## Execution status (as built in `docs-site/`) - -- **Phase 0 — Platform** ✅ Astro Starlight with the Calimero dark theme; the - bespoke animated-SVG diagram engine mounted as an island. -- **Phase 1 — Protocol track** ✅ All 10 core chapters + storage appendix ported - to MDX; structural SVGs are `.astro` diagram components, sequences are islands. -- **Phase 2 — Build / Operate / Contribute** ✅ Four-track IA; SDK, CLI - (merod/meroctl), config, and admin-API references grounded in code. -- **Phase 3 — Spec hardening** ✅ Added Blob Transport, TEE Attestation, - Cross-Context Calls, Application Upgrades; RFC 2119 Normative callouts; schema - version pins; conformance statement. -- **Phase 4 — Accuracy & depth** ✅ Verified the flagged cautions against source - (fixed real flag/endpoint/macro errors); added observability, troubleshooting, - runbooks, and example walkthroughs; completed the light theme. -- **Phase 5 — Sustain** ◑ Docs CI (`docs-ci.yml`: build + internal-link check on - every docs PR), a zero-dependency link checker (`npm run check`), and a - contributor guide (`/contribute/docs/`). Remaining below. - -### Sustain decisions - -- **Versioning — deferred to 1.0.** Starlight has no native versioning and the - protocol is pre-1.0 (breaking freely). Pinning schema versions in the spec - text is sufficient for now; adopt Git-branch snapshots or a plugin when the - first stable wire version ships. -- **Generated references — incremental.** The CLI / admin-API / config tables are - currently hand-curated and verified. The durable fix is to generate them from - the sources of truth (clap `--help`, the axum route table / an OpenAPI export, - and rustdoc) so they can't drift; this is the main outstanding sustain task. -- **Legacy cutover.** `docs-site.yml` (manual) publishes the Starlight site and - preserves the old `architecture/` site under `/architecture-legacy/`. Cutover: - (1) merge; (2) run the workflow to verify the live deploy; (3) retire - `pages.yml` and switch `docs-site.yml` to deploy on push; (4) eventually delete - `architecture/` once nothing links to it (and retarget the doc-update bot, or - repoint it to regenerate the *generated* reference pages only). diff --git a/docs-site/package-lock.json b/docs/package-lock.json similarity index 100% rename from docs-site/package-lock.json rename to docs/package-lock.json diff --git a/docs-site/package.json b/docs/package.json similarity index 82% rename from docs-site/package.json rename to docs/package.json index bfde08e670..6143d4a101 100644 --- a/docs-site/package.json +++ b/docs/package.json @@ -3,7 +3,7 @@ "version": "0.0.0", "private": true, "type": "module", - "description": "Calimero Core documentation site (Astro Starlight). Phase 0 spike — lives outside architecture/ so the doc-update bot never touches it.", + "description": "Calimero Core documentation site (Astro Starlight).", "scripts": { "dev": "astro dev", "start": "astro dev", diff --git a/docs-site/public/favicon.svg b/docs/public/favicon.svg similarity index 100% rename from docs-site/public/favicon.svg rename to docs/public/favicon.svg diff --git a/docs-site/public/robots.txt b/docs/public/robots.txt similarity index 100% rename from docs-site/public/robots.txt rename to docs/public/robots.txt diff --git a/docs-site/scripts/check-links.mjs b/docs/scripts/check-links.mjs similarity index 100% rename from docs-site/scripts/check-links.mjs rename to docs/scripts/check-links.mjs diff --git a/docs-site/src/assets/logo.svg b/docs/src/assets/logo.svg similarity index 100% rename from docs-site/src/assets/logo.svg rename to docs/src/assets/logo.svg diff --git a/docs-site/src/components/ArchMap.astro b/docs/src/components/ArchMap.astro similarity index 100% rename from docs-site/src/components/ArchMap.astro rename to docs/src/components/ArchMap.astro diff --git a/docs-site/src/components/Figure.astro b/docs/src/components/Figure.astro similarity index 100% rename from docs-site/src/components/Figure.astro rename to docs/src/components/Figure.astro diff --git a/docs-site/src/components/SeqDiagram.astro b/docs/src/components/SeqDiagram.astro similarity index 100% rename from docs-site/src/components/SeqDiagram.astro rename to docs/src/components/SeqDiagram.astro diff --git a/docs-site/src/components/diagrams/AbiBoundary.astro b/docs/src/components/diagrams/AbiBoundary.astro similarity index 100% rename from docs-site/src/components/diagrams/AbiBoundary.astro rename to docs/src/components/diagrams/AbiBoundary.astro diff --git a/docs-site/src/components/diagrams/ActorModel.astro b/docs/src/components/diagrams/ActorModel.astro similarity index 100% rename from docs-site/src/components/diagrams/ActorModel.astro rename to docs/src/components/diagrams/ActorModel.astro diff --git a/docs-site/src/components/diagrams/CausalDag.astro b/docs/src/components/diagrams/CausalDag.astro similarity index 100% rename from docs-site/src/components/diagrams/CausalDag.astro rename to docs/src/components/diagrams/CausalDag.astro diff --git a/docs-site/src/components/diagrams/ContextInMemory.astro b/docs/src/components/diagrams/ContextInMemory.astro similarity index 100% rename from docs-site/src/components/diagrams/ContextInMemory.astro rename to docs/src/components/diagrams/ContextInMemory.astro diff --git a/docs-site/src/components/diagrams/CrateStack.astro b/docs/src/components/diagrams/CrateStack.astro similarity index 100% rename from docs-site/src/components/diagrams/CrateStack.astro rename to docs/src/components/diagrams/CrateStack.astro diff --git a/docs-site/src/components/diagrams/DagVsState.astro b/docs/src/components/diagrams/DagVsState.astro similarity index 100% rename from docs-site/src/components/diagrams/DagVsState.astro rename to docs/src/components/diagrams/DagVsState.astro diff --git a/docs-site/src/components/diagrams/EntityStorage.astro b/docs/src/components/diagrams/EntityStorage.astro similarity index 100% rename from docs-site/src/components/diagrams/EntityStorage.astro rename to docs/src/components/diagrams/EntityStorage.astro diff --git a/docs-site/src/components/diagrams/FoldPipeline.astro b/docs/src/components/diagrams/FoldPipeline.astro similarity index 100% rename from docs-site/src/components/diagrams/FoldPipeline.astro rename to docs/src/components/diagrams/FoldPipeline.astro diff --git a/docs-site/src/components/diagrams/KeyLayers.astro b/docs/src/components/diagrams/KeyLayers.astro similarity index 100% rename from docs-site/src/components/diagrams/KeyLayers.astro rename to docs/src/components/diagrams/KeyLayers.astro diff --git a/docs-site/src/components/diagrams/Libp2pStack.astro b/docs/src/components/diagrams/Libp2pStack.astro similarity index 100% rename from docs-site/src/components/diagrams/Libp2pStack.astro rename to docs/src/components/diagrams/Libp2pStack.astro diff --git a/docs-site/src/components/diagrams/MerkleTreeMemory.astro b/docs/src/components/diagrams/MerkleTreeMemory.astro similarity index 100% rename from docs-site/src/components/diagrams/MerkleTreeMemory.astro rename to docs/src/components/diagrams/MerkleTreeMemory.astro diff --git a/docs-site/src/components/diagrams/ScopeHierarchy.astro b/docs/src/components/diagrams/ScopeHierarchy.astro similarity index 100% rename from docs-site/src/components/diagrams/ScopeHierarchy.astro rename to docs/src/components/diagrams/ScopeHierarchy.astro diff --git a/docs-site/src/components/diagrams/ScopeTree.astro b/docs/src/components/diagrams/ScopeTree.astro similarity index 100% rename from docs-site/src/components/diagrams/ScopeTree.astro rename to docs/src/components/diagrams/ScopeTree.astro diff --git a/docs-site/src/components/diagrams/StateProduction.astro b/docs/src/components/diagrams/StateProduction.astro similarity index 100% rename from docs-site/src/components/diagrams/StateProduction.astro rename to docs/src/components/diagrams/StateProduction.astro diff --git a/docs-site/src/components/diagrams/StoreLayout.astro b/docs/src/components/diagrams/StoreLayout.astro similarity index 100% rename from docs-site/src/components/diagrams/StoreLayout.astro rename to docs/src/components/diagrams/StoreLayout.astro diff --git a/docs-site/src/components/diagrams/TreeMerge.astro b/docs/src/components/diagrams/TreeMerge.astro similarity index 100% rename from docs-site/src/components/diagrams/TreeMerge.astro rename to docs/src/components/diagrams/TreeMerge.astro diff --git a/docs-site/src/components/diagrams/Verdict.astro b/docs/src/components/diagrams/Verdict.astro similarity index 100% rename from docs-site/src/components/diagrams/Verdict.astro rename to docs/src/components/diagrams/Verdict.astro diff --git a/docs-site/src/content.config.ts b/docs/src/content.config.ts similarity index 100% rename from docs-site/src/content.config.ts rename to docs/src/content.config.ts diff --git a/docs-site/src/content/docs/build/advanced-sdk.mdx b/docs/src/content/docs/build/advanced-sdk.mdx similarity index 100% rename from docs-site/src/content/docs/build/advanced-sdk.mdx rename to docs/src/content/docs/build/advanced-sdk.mdx diff --git a/docs-site/src/content/docs/build/app-abi.mdx b/docs/src/content/docs/build/app-abi.mdx similarity index 100% rename from docs-site/src/content/docs/build/app-abi.mdx rename to docs/src/content/docs/build/app-abi.mdx diff --git a/docs-site/src/content/docs/build/collections.mdx b/docs/src/content/docs/build/collections.mdx similarity index 100% rename from docs-site/src/content/docs/build/collections.mdx rename to docs/src/content/docs/build/collections.mdx diff --git a/docs-site/src/content/docs/build/error-handling.mdx b/docs/src/content/docs/build/error-handling.mdx similarity index 100% rename from docs-site/src/content/docs/build/error-handling.mdx rename to docs/src/content/docs/build/error-handling.mdx diff --git a/docs-site/src/content/docs/build/examples.mdx b/docs/src/content/docs/build/examples.mdx similarity index 100% rename from docs-site/src/content/docs/build/examples.mdx rename to docs/src/content/docs/build/examples.mdx diff --git a/docs-site/src/content/docs/build/gotchas.mdx b/docs/src/content/docs/build/gotchas.mdx similarity index 100% rename from docs-site/src/content/docs/build/gotchas.mdx rename to docs/src/content/docs/build/gotchas.mdx diff --git a/docs-site/src/content/docs/build/guides/access-control.mdx b/docs/src/content/docs/build/guides/access-control.mdx similarity index 100% rename from docs-site/src/content/docs/build/guides/access-control.mdx rename to docs/src/content/docs/build/guides/access-control.mdx diff --git a/docs-site/src/content/docs/build/guides/blobs.mdx b/docs/src/content/docs/build/guides/blobs.mdx similarity index 100% rename from docs-site/src/content/docs/build/guides/blobs.mdx rename to docs/src/content/docs/build/guides/blobs.mdx diff --git a/docs-site/src/content/docs/build/guides/collections.mdx b/docs/src/content/docs/build/guides/collections.mdx similarity index 100% rename from docs-site/src/content/docs/build/guides/collections.mdx rename to docs/src/content/docs/build/guides/collections.mdx diff --git a/docs-site/src/content/docs/build/guides/cross-context.mdx b/docs/src/content/docs/build/guides/cross-context.mdx similarity index 100% rename from docs-site/src/content/docs/build/guides/cross-context.mdx rename to docs/src/content/docs/build/guides/cross-context.mdx diff --git a/docs-site/src/content/docs/build/guides/events.mdx b/docs/src/content/docs/build/guides/events.mdx similarity index 100% rename from docs-site/src/content/docs/build/guides/events.mdx rename to docs/src/content/docs/build/guides/events.mdx diff --git a/docs-site/src/content/docs/build/guides/index.mdx b/docs/src/content/docs/build/guides/index.mdx similarity index 100% rename from docs-site/src/content/docs/build/guides/index.mdx rename to docs/src/content/docs/build/guides/index.mdx diff --git a/docs-site/src/content/docs/build/host-functions.mdx b/docs/src/content/docs/build/host-functions.mdx similarity index 100% rename from docs-site/src/content/docs/build/host-functions.mdx rename to docs/src/content/docs/build/host-functions.mdx diff --git a/docs-site/src/content/docs/build/index.mdx b/docs/src/content/docs/build/index.mdx similarity index 100% rename from docs-site/src/content/docs/build/index.mdx rename to docs/src/content/docs/build/index.mdx diff --git a/docs-site/src/content/docs/build/migrations.mdx b/docs/src/content/docs/build/migrations.mdx similarity index 100% rename from docs-site/src/content/docs/build/migrations.mdx rename to docs/src/content/docs/build/migrations.mdx diff --git a/docs-site/src/content/docs/build/packaging-signing.mdx b/docs/src/content/docs/build/packaging-signing.mdx similarity index 100% rename from docs-site/src/content/docs/build/packaging-signing.mdx rename to docs/src/content/docs/build/packaging-signing.mdx diff --git a/docs-site/src/content/docs/build/permissioned-storage.mdx b/docs/src/content/docs/build/permissioned-storage.mdx similarity index 99% rename from docs-site/src/content/docs/build/permissioned-storage.mdx rename to docs/src/content/docs/build/permissioned-storage.mdx index 3f99ff8d6b..c4258fb3cc 100644 --- a/docs-site/src/content/docs/build/permissioned-storage.mdx +++ b/docs/src/content/docs/build/permissioned-storage.mdx @@ -381,8 +381,7 @@ not retroactively trusted: /> When two current writers rotate **concurrently** (DAG siblings, neither in the -other's causal history), the merge resolves deterministically -(`docs/adr/0001-shared-storage-concurrent-rotation.md`): +other's causal history), the merge resolves deterministically: 1. **Causal-first** — if one rotation happens-before the other, the later one wins unconditionally (its author rotated from full knowledge of the earlier). diff --git a/docs-site/src/content/docs/build/quickstart.mdx b/docs/src/content/docs/build/quickstart.mdx similarity index 100% rename from docs-site/src/content/docs/build/quickstart.mdx rename to docs/src/content/docs/build/quickstart.mdx diff --git a/docs-site/src/content/docs/build/sdk-macros.mdx b/docs/src/content/docs/build/sdk-macros.mdx similarity index 100% rename from docs-site/src/content/docs/build/sdk-macros.mdx rename to docs/src/content/docs/build/sdk-macros.mdx diff --git a/docs-site/src/content/docs/build/state-modeling.mdx b/docs/src/content/docs/build/state-modeling.mdx similarity index 100% rename from docs-site/src/content/docs/build/state-modeling.mdx rename to docs/src/content/docs/build/state-modeling.mdx diff --git a/docs-site/src/content/docs/build/storage-complexity.mdx b/docs/src/content/docs/build/storage-complexity.mdx similarity index 100% rename from docs-site/src/content/docs/build/storage-complexity.mdx rename to docs/src/content/docs/build/storage-complexity.mdx diff --git a/docs-site/src/content/docs/build/testing.mdx b/docs/src/content/docs/build/testing.mdx similarity index 100% rename from docs-site/src/content/docs/build/testing.mdx rename to docs/src/content/docs/build/testing.mdx diff --git a/docs-site/src/content/docs/build/tutorial.mdx b/docs/src/content/docs/build/tutorial.mdx similarity index 100% rename from docs-site/src/content/docs/build/tutorial.mdx rename to docs/src/content/docs/build/tutorial.mdx diff --git a/docs-site/src/content/docs/contribute/architecture.mdx b/docs/src/content/docs/contribute/architecture.mdx similarity index 96% rename from docs-site/src/content/docs/contribute/architecture.mdx rename to docs/src/content/docs/contribute/architecture.mdx index 896012c30a..188109dfd5 100644 --- a/docs-site/src/content/docs/contribute/architecture.mdx +++ b/docs/src/content/docs/contribute/architecture.mdx @@ -135,8 +135,6 @@ through the core libraries, all converging on `primitives`. | `tools/merodb` | Inspect and debug the node's RocksDB database. | diff --git a/docs-site/src/content/docs/contribute/crate-guide.mdx b/docs/src/content/docs/contribute/crate-guide.mdx similarity index 100% rename from docs-site/src/content/docs/contribute/crate-guide.mdx rename to docs/src/content/docs/contribute/crate-guide.mdx diff --git a/docs-site/src/content/docs/contribute/development.mdx b/docs/src/content/docs/contribute/development.mdx similarity index 91% rename from docs-site/src/content/docs/contribute/development.mdx rename to docs/src/content/docs/contribute/development.mdx index fe79745d7d..6b4b270942 100644 --- a/docs-site/src/content/docs/contribute/development.mdx +++ b/docs/src/content/docs/contribute/development.mdx @@ -187,14 +187,10 @@ The full guide is in `STYLE.md`; the rules that trip people up most: | Location | What's there | | --- | --- | -| `docs/adr/` | Architecture Decision Records (numbered, e.g. `0001-shared-storage-concurrent-rotation.md`). | -| `docs/design/` | Design documents for larger efforts (e.g. the unified-causal-log cutover/flip plans). | -| `docs/RELEASE.md` | Versioning and release process (versions are managed by `cargo-workspaces`). | +| `docs/` | The Astro Starlight documentation site (this site) — all authored docs. | | `AGENTS.md` (root + per-package) | Quick orientation and conventions for each area of the tree; a good first read for any crate. | -| `architecture/` | Generated per-crate-group reference pages describing the actor model and crate internals. | diff --git a/docs-site/src/content/docs/contribute/docs.mdx b/docs/src/content/docs/contribute/docs.mdx similarity index 91% rename from docs-site/src/content/docs/contribute/docs.mdx rename to docs/src/content/docs/contribute/docs.mdx index 4b4f236f12..f7b1c0e191 100644 --- a/docs-site/src/content/docs/contribute/docs.mdx +++ b/docs/src/content/docs/contribute/docs.mdx @@ -8,9 +8,7 @@ sidebar: import { Steps, Aside, LinkCard } from '@astrojs/starlight/components'; The documentation you're reading is an [Astro Starlight](https://starlight.astro.build/) -site under `docs-site/`. It is intentionally **separate from `architecture/`** — -the legacy hand-built site and its per-PR auto-update bot only touch -`architecture/`, so authored content here is never clobbered. +site under `docs/`. ## Run it locally @@ -19,7 +17,7 @@ the legacy hand-built site and its per-PR auto-update bot only touch 1. Install and start the dev server: ```sh - cd docs-site + cd docs npm install npm run dev # http://localhost:4321/core/ ``` @@ -33,7 +31,7 @@ the legacy hand-built site and its per-PR auto-update bot only touch A GitHub Action (`docs-ci.yml`) runs `npm run check` on every PR that touches -`docs-site/**`, so a broken build or a dead internal link fails the PR. +`docs/**`, so a broken build or a dead internal link fails the PR. ## How content is organized diff --git a/docs-site/src/content/docs/contribute/index.mdx b/docs/src/content/docs/contribute/index.mdx similarity index 89% rename from docs-site/src/content/docs/contribute/index.mdx rename to docs/src/content/docs/contribute/index.mdx index e0677dd1ae..f86324eaa4 100644 --- a/docs-site/src/content/docs/contribute/index.mdx +++ b/docs/src/content/docs/contribute/index.mdx @@ -34,9 +34,3 @@ This track orients you in the codebase and the development workflow. - -## Reference - - - - diff --git a/docs-site/src/content/docs/contribute/testing.mdx b/docs/src/content/docs/contribute/testing.mdx similarity index 100% rename from docs-site/src/content/docs/contribute/testing.mdx rename to docs/src/content/docs/contribute/testing.mdx diff --git a/docs-site/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx similarity index 100% rename from docs-site/src/content/docs/index.mdx rename to docs/src/content/docs/index.mdx diff --git a/docs-site/src/content/docs/journeys.mdx b/docs/src/content/docs/journeys.mdx similarity index 100% rename from docs-site/src/content/docs/journeys.mdx rename to docs/src/content/docs/journeys.mdx diff --git a/docs-site/src/content/docs/operate/admin-api.mdx b/docs/src/content/docs/operate/admin-api.mdx similarity index 100% rename from docs-site/src/content/docs/operate/admin-api.mdx rename to docs/src/content/docs/operate/admin-api.mdx diff --git a/docs-site/src/content/docs/operate/auth.mdx b/docs/src/content/docs/operate/auth.mdx similarity index 100% rename from docs-site/src/content/docs/operate/auth.mdx rename to docs/src/content/docs/operate/auth.mdx diff --git a/docs-site/src/content/docs/operate/config.mdx b/docs/src/content/docs/operate/config.mdx similarity index 100% rename from docs-site/src/content/docs/operate/config.mdx rename to docs/src/content/docs/operate/config.mdx diff --git a/docs-site/src/content/docs/operate/deployment.mdx b/docs/src/content/docs/operate/deployment.mdx similarity index 100% rename from docs-site/src/content/docs/operate/deployment.mdx rename to docs/src/content/docs/operate/deployment.mdx diff --git a/docs-site/src/content/docs/operate/index.mdx b/docs/src/content/docs/operate/index.mdx similarity index 100% rename from docs-site/src/content/docs/operate/index.mdx rename to docs/src/content/docs/operate/index.mdx diff --git a/docs-site/src/content/docs/operate/install.mdx b/docs/src/content/docs/operate/install.mdx similarity index 100% rename from docs-site/src/content/docs/operate/install.mdx rename to docs/src/content/docs/operate/install.mdx diff --git a/docs-site/src/content/docs/operate/meroctl.mdx b/docs/src/content/docs/operate/meroctl.mdx similarity index 100% rename from docs-site/src/content/docs/operate/meroctl.mdx rename to docs/src/content/docs/operate/meroctl.mdx diff --git a/docs-site/src/content/docs/operate/merod.mdx b/docs/src/content/docs/operate/merod.mdx similarity index 100% rename from docs-site/src/content/docs/operate/merod.mdx rename to docs/src/content/docs/operate/merod.mdx diff --git a/docs-site/src/content/docs/operate/networking.mdx b/docs/src/content/docs/operate/networking.mdx similarity index 100% rename from docs-site/src/content/docs/operate/networking.mdx rename to docs/src/content/docs/operate/networking.mdx diff --git a/docs-site/src/content/docs/operate/observability.mdx b/docs/src/content/docs/operate/observability.mdx similarity index 100% rename from docs-site/src/content/docs/operate/observability.mdx rename to docs/src/content/docs/operate/observability.mdx diff --git a/docs-site/src/content/docs/operate/runbooks.mdx b/docs/src/content/docs/operate/runbooks.mdx similarity index 100% rename from docs-site/src/content/docs/operate/runbooks.mdx rename to docs/src/content/docs/operate/runbooks.mdx diff --git a/docs-site/src/content/docs/operate/security.mdx b/docs/src/content/docs/operate/security.mdx similarity index 100% rename from docs-site/src/content/docs/operate/security.mdx rename to docs/src/content/docs/operate/security.mdx diff --git a/docs-site/src/content/docs/operate/troubleshooting.mdx b/docs/src/content/docs/operate/troubleshooting.mdx similarity index 100% rename from docs-site/src/content/docs/operate/troubleshooting.mdx rename to docs/src/content/docs/operate/troubleshooting.mdx diff --git a/docs-site/src/content/docs/protocol/applications.mdx b/docs/src/content/docs/protocol/applications.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/applications.mdx rename to docs/src/content/docs/protocol/applications.mdx diff --git a/docs-site/src/content/docs/protocol/blobs.mdx b/docs/src/content/docs/protocol/blobs.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/blobs.mdx rename to docs/src/content/docs/protocol/blobs.mdx diff --git a/docs-site/src/content/docs/protocol/capability-inheritance.mdx b/docs/src/content/docs/protocol/capability-inheritance.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/capability-inheritance.mdx rename to docs/src/content/docs/protocol/capability-inheritance.mdx diff --git a/docs-site/src/content/docs/protocol/concepts.mdx b/docs/src/content/docs/protocol/concepts.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/concepts.mdx rename to docs/src/content/docs/protocol/concepts.mdx diff --git a/docs-site/src/content/docs/protocol/context-lifecycle.mdx b/docs/src/content/docs/protocol/context-lifecycle.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/context-lifecycle.mdx rename to docs/src/content/docs/protocol/context-lifecycle.mdx diff --git a/docs-site/src/content/docs/protocol/crdt-internals.mdx b/docs/src/content/docs/protocol/crdt-internals.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/crdt-internals.mdx rename to docs/src/content/docs/protocol/crdt-internals.mdx diff --git a/docs-site/src/content/docs/protocol/data-anatomy.mdx b/docs/src/content/docs/protocol/data-anatomy.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/data-anatomy.mdx rename to docs/src/content/docs/protocol/data-anatomy.mdx diff --git a/docs-site/src/content/docs/protocol/divergence-recovery.mdx b/docs/src/content/docs/protocol/divergence-recovery.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/divergence-recovery.mdx rename to docs/src/content/docs/protocol/divergence-recovery.mdx diff --git a/docs-site/src/content/docs/protocol/encryption.mdx b/docs/src/content/docs/protocol/encryption.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/encryption.mdx rename to docs/src/content/docs/protocol/encryption.mdx diff --git a/docs-site/src/content/docs/protocol/execution.mdx b/docs/src/content/docs/protocol/execution.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/execution.mdx rename to docs/src/content/docs/protocol/execution.mdx diff --git a/docs-site/src/content/docs/protocol/glossary.mdx b/docs/src/content/docs/protocol/glossary.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/glossary.mdx rename to docs/src/content/docs/protocol/glossary.mdx diff --git a/docs-site/src/content/docs/protocol/governance-edge-cases.mdx b/docs/src/content/docs/protocol/governance-edge-cases.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/governance-edge-cases.mdx rename to docs/src/content/docs/protocol/governance-edge-cases.mdx diff --git a/docs-site/src/content/docs/protocol/governance.mdx b/docs/src/content/docs/protocol/governance.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/governance.mdx rename to docs/src/content/docs/protocol/governance.mdx diff --git a/docs-site/src/content/docs/protocol/hlc.mdx b/docs/src/content/docs/protocol/hlc.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/hlc.mdx rename to docs/src/content/docs/protocol/hlc.mdx diff --git a/docs-site/src/content/docs/protocol/identities.mdx b/docs/src/content/docs/protocol/identities.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/identities.mdx rename to docs/src/content/docs/protocol/identities.mdx diff --git a/docs-site/src/content/docs/protocol/key-rotation.mdx b/docs/src/content/docs/protocol/key-rotation.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/key-rotation.mdx rename to docs/src/content/docs/protocol/key-rotation.mdx diff --git a/docs-site/src/content/docs/protocol/limits.mdx b/docs/src/content/docs/protocol/limits.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/limits.mdx rename to docs/src/content/docs/protocol/limits.mdx diff --git a/docs-site/src/content/docs/protocol/networking.mdx b/docs/src/content/docs/protocol/networking.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/networking.mdx rename to docs/src/content/docs/protocol/networking.mdx diff --git a/docs-site/src/content/docs/protocol/operations.mdx b/docs/src/content/docs/protocol/operations.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/operations.mdx rename to docs/src/content/docs/protocol/operations.mdx diff --git a/docs-site/src/content/docs/protocol/overview.mdx b/docs/src/content/docs/protocol/overview.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/overview.mdx rename to docs/src/content/docs/protocol/overview.mdx diff --git a/docs-site/src/content/docs/protocol/projection.mdx b/docs/src/content/docs/protocol/projection.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/projection.mdx rename to docs/src/content/docs/protocol/projection.mdx diff --git a/docs-site/src/content/docs/protocol/receive-path.mdx b/docs/src/content/docs/protocol/receive-path.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/receive-path.mdx rename to docs/src/content/docs/protocol/receive-path.mdx diff --git a/docs-site/src/content/docs/protocol/security-model.mdx b/docs/src/content/docs/protocol/security-model.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/security-model.mdx rename to docs/src/content/docs/protocol/security-model.mdx diff --git a/docs-site/src/content/docs/protocol/storage.mdx b/docs/src/content/docs/protocol/storage.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/storage.mdx rename to docs/src/content/docs/protocol/storage.mdx diff --git a/docs-site/src/content/docs/protocol/sync-internals.mdx b/docs/src/content/docs/protocol/sync-internals.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/sync-internals.mdx rename to docs/src/content/docs/protocol/sync-internals.mdx diff --git a/docs-site/src/content/docs/protocol/sync.mdx b/docs/src/content/docs/protocol/sync.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/sync.mdx rename to docs/src/content/docs/protocol/sync.mdx diff --git a/docs-site/src/content/docs/protocol/tee-attestation.mdx b/docs/src/content/docs/protocol/tee-attestation.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/tee-attestation.mdx rename to docs/src/content/docs/protocol/tee-attestation.mdx diff --git a/docs-site/src/content/docs/protocol/upgrades.mdx b/docs/src/content/docs/protocol/upgrades.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/upgrades.mdx rename to docs/src/content/docs/protocol/upgrades.mdx diff --git a/docs-site/src/content/docs/protocol/write-path.mdx b/docs/src/content/docs/protocol/write-path.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/write-path.mdx rename to docs/src/content/docs/protocol/write-path.mdx diff --git a/docs-site/src/content/docs/protocol/xcall.mdx b/docs/src/content/docs/protocol/xcall.mdx similarity index 100% rename from docs-site/src/content/docs/protocol/xcall.mdx rename to docs/src/content/docs/protocol/xcall.mdx diff --git a/docs-site/src/content/docs/topics.mdx b/docs/src/content/docs/topics.mdx similarity index 100% rename from docs-site/src/content/docs/topics.mdx rename to docs/src/content/docs/topics.mdx diff --git a/docs-site/src/content/docs/using-the-docs.mdx b/docs/src/content/docs/using-the-docs.mdx similarity index 100% rename from docs-site/src/content/docs/using-the-docs.mdx rename to docs/src/content/docs/using-the-docs.mdx diff --git a/docs-site/src/middleware.ts b/docs/src/middleware.ts similarity index 100% rename from docs-site/src/middleware.ts rename to docs/src/middleware.ts diff --git a/docs-site/src/pages/llms.txt.ts b/docs/src/pages/llms.txt.ts similarity index 100% rename from docs-site/src/pages/llms.txt.ts rename to docs/src/pages/llms.txt.ts diff --git a/docs-site/src/scripts/diagrams.client.ts b/docs/src/scripts/diagrams.client.ts similarity index 100% rename from docs-site/src/scripts/diagrams.client.ts rename to docs/src/scripts/diagrams.client.ts diff --git a/docs-site/src/styles/theme.css b/docs/src/styles/theme.css similarity index 100% rename from docs-site/src/styles/theme.css rename to docs/src/styles/theme.css diff --git a/docs-site/tsconfig.json b/docs/tsconfig.json similarity index 100% rename from docs-site/tsconfig.json rename to docs/tsconfig.json From 1dc194cbba484591c76ec92f310ab81bf0e9d0a9 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 29 Jun 2026 07:36:44 +0000 Subject: [PATCH 2/2] docs(rotation-log): restore concurrent-rotation merge rationale The ADR that documented the writer-set merge rule was removed in this PR's folder consolidation. Inline the per-level rationale (causal-first / HLC / signer-pubkey) in the module doc and point to the published Permissioned Storage docs page so the 'why' survives. --- crates/storage/src/rotation_log.rs | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/crates/storage/src/rotation_log.rs b/crates/storage/src/rotation_log.rs index 50c2958eee..c85717c0af 100644 --- a/crates/storage/src/rotation_log.rs +++ b/crates/storage/src/rotation_log.rs @@ -10,8 +10,24 @@ //! //! # Design //! -//! Every accepted rotation appends an entry; the node-side reader compares -//! entries by **causal-first → HLC → signer-pubkey** ordering. +//! Every accepted rotation appends an entry; the node-side reader resolves +//! concurrent rotations by a three-level **causal-first → HLC → +//! signer-pubkey** ordering, each level a deterministic fallback for the one +//! above: +//! +//! - **Causal-first** — if one rotation happens-before another, the later one +//! wins unconditionally; its author rotated with full knowledge of the +//! earlier rotation, so honoring it can never lose information. +//! - **HLC tiebreak** — for truly concurrent rotations (neither in the +//! other's causal history) the larger `HybridTimestamp` wins. The HLC +//! embeds wall-time plus a node id, giving every node the same total order +//! without coordination. +//! - **Signer-pubkey tiebreak** — on the (astronomically rare) HLC tie, the +//! smaller signing-key bytes win, so the resolution is total and every node +//! converges on the same writer set. +//! +//! The user-facing writeup of this rule lives on the Permissioned Storage +//! docs page: . //! //! The log is materialized as an `UnorderedMap<[u8; 32], RotationLogEntry>` //! child of the Shared anchor entity (keyed by `delta_id`), so each rotation