Summary
Between 0.11.0-rc.9 and 0.11.0-rc.10, Open-subgroup TEE context replication stopped working over Docker/bridge networking. A context registered in an Open subgroup no longer replicates to an admitted mock-TEE replica when the two nodes are in separate containers. Binary/native (loopback) is unaffected — so it's transport/networking-sensitive, not a pure logic break.
This is what's turning merobox master CI red (Docker tee-g2-open-inheritance-replication, tee-matrix-open-join-with-created, tee-matrix-open-late-join), which surfaced it via the moving merod:prerelease tag.
Airtight A/B repro (local, same everything but the merod image)
Scenario: merobox bootstrap run tee-g2-open-inheritance-replication.yml --image <merod> (Docker mode, mock-TEE replica). Assertion polls meroctl context ls on the replica for up to 90s.
| merod image |
Docker mode |
Binary mode (--no-docker) |
0.11.0-rc.9 (rev 013250b7) |
✅ context replicates (seconds) |
✅ |
0.11.0-rc.10 (rev 42096e9c) |
❌ never replicates in 90s |
✅ |
Only the image changed; merobox, scenario, host, and network setup were identical across runs. rc.9 Docker passes, rc.10 Docker fails, both binary pass → rc.10 regression, Docker/transport-specific.
Pattern across the tee suite on rc.10 Docker: the only failures are the three scenarios that assert a positive Open-inheritance replication. Restricted variants, tee-r2-open-no-direct-row (Open but expects no replication row), and g1/g3/g4/r1 all pass — i.e. it's specifically the auto-follow/inheritance replication path.
How it hid as a flake
merobox's two CI runs for the #288/#289 merges were created 32s apart, but their tee-Docker jobs pulled prerelease ~48min apart, straddling the rc.10 image publish (built 11:56 UTC today). The earlier job got pre-rc.10 (green), the later got rc.10 (red) — same commit-ish, opposite result, which read as nondeterministic.
Candidate commits (UNCONFIRMED — need a bisect)
97 commits in 013250b7..42096e9c. Given it's Docker/transport-only, the sync/handshake and namespace-boundary hardening are the prime suspects:
#3167 fix(sync): bind inbound handshake identity to the transport with a proof of possession — most suspicious; "bind to transport" is inherently address-sensitive, and Docker's bridge presents different transport addresses than loopback#3134 fix(server): enforce context membership on WS/SSE subscribe (subscription IDOR)#3147 verify invitation signature before seeding namespace admin#3114 governance hardening for key delivery/rotation & namespace boundaries#3173 cap namespace governance-op collection (DoS)
I did not bisect (each step needs a per-commit merod image); the core team can bisect faster with per-commit images. Happy to help narrow.
Repro command
merobox bootstrap run workflow-examples/tee-g2-open-inheritance-replication.yml \
--image ghcr.io/calimero-network/merod:0.11.0-rc.10 # fails
merobox bootstrap run workflow-examples/tee-g2-open-inheritance-replication.yml \
--image ghcr.io/calimero-network/merod:0.11.0-rc.9 # passes
(with meroctl on PATH for the assert step; requires merobox with the mock-TEE steps + calimero-network/merobox#291's polling assert for a clean signal)
🤖 Generated with Claude Code
Summary
Between 0.11.0-rc.9 and 0.11.0-rc.10, Open-subgroup TEE context replication stopped working over Docker/bridge networking. A context registered in an Open subgroup no longer replicates to an admitted mock-TEE replica when the two nodes are in separate containers. Binary/native (loopback) is unaffected — so it's transport/networking-sensitive, not a pure logic break.
This is what's turning merobox master CI red (Docker
tee-g2-open-inheritance-replication,tee-matrix-open-join-with-created,tee-matrix-open-late-join), which surfaced it via the movingmerod:prereleasetag.Airtight A/B repro (local, same everything but the merod image)
Scenario:
merobox bootstrap run tee-g2-open-inheritance-replication.yml --image <merod>(Docker mode, mock-TEE replica). Assertion pollsmeroctl context lson the replica for up to 90s.--no-docker)0.11.0-rc.9(rev013250b7)0.11.0-rc.10(rev42096e9c)Only the image changed; merobox, scenario, host, and network setup were identical across runs. rc.9 Docker passes, rc.10 Docker fails, both binary pass → rc.10 regression, Docker/transport-specific.
Pattern across the tee suite on rc.10 Docker: the only failures are the three scenarios that assert a positive Open-inheritance replication. Restricted variants,
tee-r2-open-no-direct-row(Open but expects no replication row), and g1/g3/g4/r1 all pass — i.e. it's specifically the auto-follow/inheritance replication path.How it hid as a flake
merobox's two CI runs for the #288/#289 merges were created 32s apart, but their tee-Docker jobs pulled
prerelease~48min apart, straddling the rc.10 image publish (built 11:56 UTC today). The earlier job got pre-rc.10 (green), the later got rc.10 (red) — same commit-ish, opposite result, which read as nondeterministic.Candidate commits (UNCONFIRMED — need a bisect)
97 commits in
013250b7..42096e9c. Given it's Docker/transport-only, the sync/handshake and namespace-boundary hardening are the prime suspects:#3167fix(sync): bind inbound handshake identity to the transport with a proof of possession — most suspicious; "bind to transport" is inherently address-sensitive, and Docker's bridge presents different transport addresses than loopback#3134fix(server): enforce context membership on WS/SSE subscribe (subscription IDOR)#3147verify invitation signature before seeding namespace admin#3114governance hardening for key delivery/rotation & namespace boundaries#3173cap namespace governance-op collection (DoS)I did not bisect (each step needs a per-commit merod image); the core team can bisect faster with per-commit images. Happy to help narrow.
Repro command
(with
meroctlon PATH for the assert step; requires merobox with the mock-TEE steps + calimero-network/merobox#291's polling assert for a clean signal)🤖 Generated with Claude Code