-
Notifications
You must be signed in to change notification settings - Fork 36
fix(node,server): rate-limit and harden TEE attestation-verify entrypoints #2996
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
e068108
3ad1a71
9d681af
e25172e
5281f0a
3078316
4f2f797
27c17bc
deba1f4
007d5a9
852da58
1fc907b
60057c6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,9 +1,12 @@ | ||
| use actix::{AsyncContext, WrapFuture}; | ||
| use actix::{ActorFutureExt, AsyncContext, WrapFuture}; | ||
| use calimero_context_config::types::ContextGroupId; | ||
| use calimero_network_primitives::messages::NetworkEvent; | ||
| use calimero_network_primitives::specialized_node_invite::SpecializedNodeType; | ||
| use calimero_node_primitives::sync::BroadcastMessage; | ||
| use sha2::{Digest, Sha256}; | ||
| use tracing::{debug, error, info, warn}; | ||
|
|
||
| use crate::handlers::tee_attestation_throttle::Decision; | ||
| use crate::handlers::{specialized_node_invite, tee_attestation_admission}; | ||
| use crate::run::NodeMode; | ||
| use crate::NodeManager; | ||
|
|
@@ -143,13 +146,114 @@ pub(super) fn handle_specialized_broadcast( | |
| "Received TEE attestation announce on namespace topic" | ||
| ); | ||
|
|
||
| // Admission-control gates (TEE-01 / audit #48). The heavy | ||
| // `verify_attestation` path (outbound Intel-PCS fetch + DCAP | ||
| // verify) runs BEFORE any policy lookup, so an unguarded announce | ||
| // lets a malicious mesh peer amplify a 64 KiB gossip frame into a | ||
| // CPU verify + outbound PCS request by replaying a real quote under | ||
| // fresh nonces. Guard the spawn synchronously here, on the actor | ||
| // thread, before any verify work is scheduled. | ||
|
xilosada marked this conversation as resolved.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 SHA-256 of the full quote bytes computed on the actor thread before throttle check
Suggested fix:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Documented at the hash site in 4f2f797. The hash can't be deferred behind the gates: |
||
| // `quote_hash` is the key for both the durable `is_quote_hash_used` | ||
| // check and the throttle's dedup gate, so it must be computed before | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 SHA-256 of quote bytes computed on the actor thread before throttle check — bounded but worth noting The comment correctly notes that SHA-256 over ≤64 KiB is tens of microseconds. However, this computation happens synchronously on the Suggested fix:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Already documented at the hash site (commit 4f2f797). The hash can't be deferred behind the gates — it's the key for both Gate 1 (dedup) and the durable |
||
| // either can run — it cannot be deferred behind the rate/inflight | ||
| // gates. This is acceptable: `quote_bytes` is bounded to gossipsub's | ||
| // 64 KiB default at the transport, and SHA-256 over ≤64 KiB is tens | ||
| // of microseconds, negligible beside the PCS-fetch + DCAP verify the | ||
| // hash exists to gate. | ||
| let quote_hash: [u8; 32] = Sha256::digest(quote_bytes).into(); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 Durable dedup check uses The durable Suggested fix:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 27c17bc — the throttle is now keyed on |
||
| let group_id = ContextGroupId::from(namespace_id_bytes); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 SHA-256 of the full quote bytes computed on the actor thread before throttle check
Suggested fix:
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 SHA-256 of quote_bytes computed before throttle but after info! log leaks timing The Suggested fix:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Assessed — no change. The |
||
|
|
||
| // In-memory admission gates FIRST — per-group quote dedup + per-peer | ||
| // rate limit + global inflight-verify cap — so a flood is rejected on | ||
| // cheap in-memory state before we touch the store below. The returned | ||
| // permit must outlive the verify, so it is moved into the spawned | ||
| // task and held until completion. Key the throttle on | ||
| // `group_id.to_bytes()` so it and the durable check share one | ||
| // `ContextGroupId`-derived key (no divergence if the repr changes). | ||
| let now = std::time::Instant::now(); | ||
| let verify_permit = match this.tee_admission_throttle.check( | ||
| now, | ||
| source, | ||
| group_id.to_bytes(), | ||
| quote_hash, | ||
| ) { | ||
| Decision::Proceed(permit) => permit, | ||
| Decision::Duplicate => { | ||
| debug!( | ||
| %source, | ||
| quote_hash = %hex::encode(quote_hash), | ||
| "Dropping TeeAttestationAnnounce: recently-seen quote (dedup)" | ||
| ); | ||
| return true; | ||
| } | ||
| Decision::RateLimited => { | ||
| warn!( | ||
| %source, | ||
| "Dropping TeeAttestationAnnounce: per-peer attestation rate limit exceeded" | ||
| ); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 Durable store check consumes a throttle token even on a store-hit early return When Suggested fix:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Assessed — documented the trade-off in 60057c6; keeping the throttle-before-store order. This is the flip side of the reorder you (correctly) asked for last round: an already-admitted quote that is re-announced burns one per-peer token before the store read reveals it's admitted. It's bounded (the dedup entry the throttle just recorded suppresses the rest of the burst — subsequent re-announces hit I deliberately did not take the |
||
| return true; | ||
| } | ||
| Decision::AtCapacity => { | ||
| warn!( | ||
| %source, | ||
| "Dropping TeeAttestationAnnounce: attestation verify capacity saturated" | ||
| ); | ||
| return true; | ||
| } | ||
| }; | ||
|
|
||
| // Durable dedup (a store read), pulled earlier from `admit_tee_node`: | ||
| // a quote already admitted to this group never needs re-verifying. | ||
| // Run it only AFTER the cheap in-memory gates above, so an | ||
| // unauthenticated announce flood can't drive a store read per frame. | ||
| // A read error is non-fatal — the authoritative check still runs in | ||
| // `admit_tee_node` — so we proceed rather than drop a possibly- | ||
| // legitimate announce. On a hit we drop here: `verify_permit` is | ||
| // released as it goes out of scope, and the dedup entry the throttle | ||
| // just recorded keeps suppressing further re-announces of this quote. | ||
| // | ||
| // Trade-off of throttle-before-store: an already-admitted quote that | ||
| // is re-announced burns one per-peer token before we learn it's | ||
| // admitted. That is bounded (the dedup entry suppresses the rest of | ||
| // the burst) and rare (a node stops announcing once admitted), and it | ||
| // is the price of not doing a store read for every flood frame. We | ||
| // deliberately do NOT `forget_quote` here — keeping the entry is what | ||
| // makes the follow-up re-announces cheap — nor restore the token, | ||
| // which would just re-open the store read on the next frame. | ||
| match calimero_governance_store::is_quote_hash_used( | ||
| &this.datastore, | ||
| &group_id, | ||
| "e_hash, | ||
| ) { | ||
| Ok(true) => { | ||
| debug!( | ||
| %source, | ||
| quote_hash = %hex::encode(quote_hash), | ||
| "Dropping TeeAttestationAnnounce: quote already admitted to group" | ||
| ); | ||
| return true; | ||
| } | ||
| Ok(false) => {} | ||
| Err(err) => { | ||
| warn!( | ||
| %source, | ||
| error = %err, | ||
| "Failed to read quote-hash usage; proceeding to verify" | ||
| ); | ||
| } | ||
| } | ||
|
|
||
| let context_client = this.clients.context.clone(); | ||
| let quote_bytes = quote_bytes.clone(); | ||
| let public_key = *public_key; | ||
| let nonce = *nonce; | ||
| let group_key = group_id.to_bytes(); | ||
| let _ignored = ctx.spawn( | ||
| async move { | ||
| if let Err(err) = tee_attestation_admission::handle_tee_attestation_announce( | ||
| // Hold the inflight permit for the lifetime of the verify so | ||
| // the global concurrency cap stays accurate; dropped here. | ||
| let _verify_permit = verify_permit; | ||
| tee_attestation_admission::handle_tee_attestation_announce( | ||
| &context_client, | ||
| source, | ||
| quote_bytes, | ||
|
|
@@ -158,15 +262,38 @@ pub(super) fn handle_specialized_broadcast( | |
| namespace_id_bytes, | ||
| ) | ||
| .await | ||
| { | ||
| } | ||
| .into_actor(this) | ||
| .map(move |result, actor, _ctx| { | ||
| if let Err(err) = result { | ||
| warn!( | ||
| %source, | ||
| error = %err, | ||
| "Failed to handle TEE attestation announce" | ||
| ); | ||
| // The verify *errored* (a transient infra failure such | ||
| // as an Intel-PCS fetch error — an invalid quote instead | ||
| // returns Ok above). Forget the dedup entry so a | ||
| // re-announce of the same quote bytes (the fleet-join | ||
| // re-announce loop reuses the identical payload) can be | ||
| // re-verified rather than being suppressed for the full | ||
| // dedup TTL — which, at TTL ≫ the 30s admission window, | ||
| // would otherwise sink the whole join on one PCS hiccup. | ||
| // A genuinely invalid quote stays deduped, and the | ||
| // per-peer rate limit + inflight cap still bound any | ||
| // replay-driven re-verification. | ||
| // | ||
| // This `.map` runs only if the spawned future completes. | ||
| // The sole way it is dropped first is the actor stopping | ||
| // — but the throttle (and its in-memory dedup map) lives | ||
| // on this actor, so it is torn down together; a stale | ||
| // entry can't outlive its map to suppress a post-restart | ||
| // re-announce (a restarted node starts with an empty map). | ||
| actor | ||
| .tee_admission_throttle | ||
| .forget_quote(group_key, quote_hash); | ||
| } | ||
| } | ||
| .into_actor(this), | ||
| }), | ||
| ); | ||
| true | ||
| } | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.