Speculative Messaging v0.5: streams + POV lifts; factor out Off-Chain Block Verification and Super Chains#12659
Conversation
Design decisions settled during review: - Authoring always requires the newest root at the chosen tier; consumption lag is covered by extension proofs, never by anchoring older windowed roots. The window's sole purpose is authoring→inclusion pipeline slack (incl. elastic-scaling bursts). - Drop "anchor" terminology—indistinguishable from the requires entry it named; "required root" throughout, LateBlockProof.new_anchor → new_requires. - One canonical trust-tier table (Trust Domains); ack rules, pruning, flow control and appendices defer to it instead of restating. - consumed_streams(): id and resume state merged into one enum, grouped by source; Register variant dropped (no resume state exists—reads are head-only, pinned by the required root, ordered by the register's own monotonic fields). ChannelId struct replaces (ParaId, u8, u16) tuples. - One-required-root-per-source extended to bundled candidates; merging different roots would be unsound for inclusion-proof reads. - Receiver-side path for unsolicited opens (candidate opens via the messaging inherent). Corrections: commitment-tree example had the wrong trie shape (proof is 3 hashes, not 2); late-block pseudocode could silently skip stream verification on missing proofs; window-slack claim overstated. Super Chains factored out to super-chains-design.md (draft): future work, kept apart from the implementable core. Cycle Prevention renamed to Cycle Handling—candidate-level cycles are supported, not prevented.
Blocks no longer emit Requires nor ever see a StreamsRoot—a tree proof verified in the STF would bind to a provider-chosen hash, verifying nothing. Instead blocks output a consumption record (per touched stream, an Interval: where consumption started and ended) and the validate_block wrapper synthesizes the candidate's entries: intervals must chain across the bundle (equal, or advance-proven to move forward—closes a fabricated-read-context hole), one lift per stream binds the chain's endpoint to a committed root. One code path for steady state, partial consumption, resubmission and bundles; in-block catch-up proofs and separate late-block proofs are gone. Consequences carried through: - Fetch protocol reduced to two root-keyed request pairs (MessagesRequest/EventRequest); every request names the provides root the requester verified at its tier, every response is independently verifiable against it. No block hashes on the wire. - Event-stream liftability bounded: serve extension proofs from any block boundary within 25 h (mirrors availability retention); windowed heads' payloads mandatory, rest QoS. Range reads above the highwater via start_peaks. - Structs stripped to what verification consumes: lift roots, stream ids, old MMR state, block hashes all derived or positional, never declared. - POV reservation for worst-case lifts made a formula over STF-known quantities. Grounded in the existing validate_block mechanics (per-block loop, post-execution storage reads, signal re-assembly; lifts ride ParachainBlockData like LLv2's scheduling_proof).
…ssage-0.5 # Conflicts: # docs/offchain-block-verification-design.md # docs/speculative-messaging-design.md
|
All GitHub workflows were cancelled due to failure one of the required jobs. |
| ### Relay Chain Matching | ||
|
|
||
| When the relay chain processes candidates for inclusion, it performs commitment | ||
| matching. The relay chain only sees the minimal commitments (hashes), not | ||
| internal state. | ||
|
|
||
| The relay chain maintains, per (sender, receiver) pair, a small window of the | ||
| most recent provides roots: | ||
| The relay chain maintains **one small ring per sender**—the last W committed | ||
| `StreamsRoot`s—and nothing else: |
There was a problem hiding this comment.
On the relay-side matching (virtual window / enactment dependencies): I assume its value is concentrated in the speculative / same-block / super-chain case — a receiver consuming from a source candidate being included in the same relay block, which isn't yet in any header the receiver can read, the relay match is the piece that closes the loop, also acting as a supplement / final safety guard on top of Off-Chain Parachain Block Verification (which itself rests on collator slashable, economic, and probabilistic guarantees).
For inclusion-based batching, though, I'm not sure a dedicated relay field or UMP signal is needed to carry the StreamsRoot at all. The relay already commits every parachain head under mmr.rootHash: each MMR leaf carries, as its extra_data, the binary-Merkle root over all parachain heads (ParaHeadsRootProvider), and the head bytes are the leaf preimage — so that root anchors anything in a head, including the StreamsRoot committed into the header digest. Because the relay MMR commits history, a receiver anchors to a past head-leaf via an MMR inclusion proof — the MMR's history subsumes the ring's window, so no dedicated ring is needed to cover the authoring→inclusion gap.
If that's right, the whole inclusion-based verification could move into the receiver PVF — just a few composed proof layers (MMR inclusion → para_heads Merkle → the head digest → the consumed stream), with no relay-side changes at this stage.
Does that match your thinking, or am I missing a reason the relay-side carry/match is needed even for the plain inclusion consume tier — e.g. the PoV cost of carrying those proofs in every receiver block?
There was a problem hiding this comment.
I think you are right. For the inclusion based tier this should work as well, but indeed not for the speculative tier where this design is coming from. So we would need both eventually, while the proposed works for all tiers. A single root hash per candidate is not really an overhead worth mentioning, a vec maintaining a bit of backlog seems fine too. The relay commitment to para heads is flawed as it stands today too, because it can't actually provide all roots - they are collected each block and the number of parachains (including non-active) is not really bounded. The comment suggests that this is only needed for offchain XCMP, which we are replacing with speculative messaging, thus using this simple vec of recent provides root is actually a fix for that problem.
TL;DR: You are right that this would be an alternative, I would not see it as better though because it adds complexity and another code path - we need to provides inclusion matching for the speculative tier and that implementation is also flawed as it stands today.
2 participants
This PR takes the Speculative Messaging design to v0.5 (status: ready for review) and splits two subsystems into their own documents.
speculative-messaging-design.md(0.3 → 0.5, full rewrite of the detailed design):(ParaId, root)sets are replaced by a singleStreamsRootper sender block — the root of a keyed commitment tree over all of the sender's stream roots. Relay chain state shrinks to one fixed-size ring of recent roots per sender (W × 32 B), independent of stream, channel, or subscriber count. This is the reversal condition 0.3's own analysis (PR Speculative Messaging Design #10449) anticipated: streams multiply entry counts, and flat entries would have keyed persistent relay state and ossified relay-visible semantics.StreamId(channel / ack / broadcast / private kinds; consensus-critical 8-byte canonical encoding) keys per-stream MMRs. Channels become unidirectional with lossy out-of-band flow control (a latest-winsRegisteron the receiver's ack stream: acceptance, credit, watermark, close); broadcast event streams deliver native pub-sub (Pubsub mechanism for Parachains #606) with zero relay state per feed or subscriber.Requiresand never see aStreamsRoot(an STF-verified tree proof would bind to a provider-chosen hash). Blocks record per-stream consumption intervals; thevalidate_blockwrapper stitches them across the bundle (gaps must be advance-proven forward — closes a fabricated-read-context hole) and synthesizes the candidate's requires entries via one POV-carried lift per stream. One code path covers steady state, partial consumption, resubmission, and bundles; 0.3's in-block catch-up proofs and separate late-block proofs are gone. Lifts are regenerable by anyone from public data, so blocks never go stale.MessagesRequest/EventRequest); every response independently verifiable against a requester-named root. Lift-serving obligations bounded at 25 h, mirroring availability retention. Runtime API defining the node–runtime boundary added; worst-case lift POV reservation charged as STFproof_sizeweight.offchain-block-verification-design.md(new): generic mechanism for foreign nodes to verify unincluded parachain blocks (lineage, authorship, ack confidence) by executing the source chain's own relay-registered wasm against storage proofs anchored at included heads. Consumed by speculative messaging's pre-inclusion tiers and Low-Latency v2 ack verification. Covers trust anchors (included-only proof anchors, coupled to tip-only ack checking), theOffchainVerifyruntime API, runtime-upgrade pause/resume rules, network protocols, and security analysis.super-chains-design.md(new, draft/sketch): super-chain material factored out of the messaging design as explicit future work. The messaging design retains everything super chains need (mutual requires, atomic enactment groups, cycle handling) with no special cases.