Skip to content

[CHIA-3823] Add INTERNED_GENERATOR consensus path + serde_2026 prefix guard - #1438

Merged
richardkiss merged 5 commits into
mainfrom
interned-generator-consensus
Aug 3, 2026
Merged

[CHIA-3823] Add INTERNED_GENERATOR consensus path + serde_2026 prefix guard#1438
richardkiss merged 5 commits into
mainfrom
interned-generator-consensus

Conversation

@richardkiss

@richardkiss richardkiss commented May 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Implement the INTERNED_GENERATOR consensus path in run_block_generator2.

We also tweak check_generator_quote to skip checking serde_2026-serialized generators because it's more difficult to tell if they start with a q. We still check in check_generator_node.


Note

High Risk
This is consensus-critical generator serialization and validation; a mismatch with network rules would fork or reject valid blocks.

Overview
With INTERNED_GENERATOR, block generators are serde_2026-only: InternedBlockBuilder::finalize now writes serialize_2026 instead of backrefs, and run_block_generator2 / trusted-block coinspend helpers decode via node_from_bytes_2026 with a max_canonical_blob_size cap, then charge byte cost from the interned tree instead of raw wire length.

check_generator_quote no longer inspects the [0xff, 0x01] prefix when the interned flag is set (serde_2026 bytes are not quote-shaped on the wire); simple-generator “quoted” shape is still enforced in check_generator_node after decode. Pre-fork behavior is unchanged: serde_2026 blobs fail parse or quote checks without the flag, and classic generators fail once INTERNED_GENERATOR is on.

Adds node_from_bytes_2026 in serde_2026, fuzz round-trip coverage with INTERNED_GENERATOR, and unit tests for cross-encoding acceptance/rejection.

Reviewed by Cursor Bugbot for commit 616e486. Bugbot is set up for automated code reviews on this repo. Configure here.

Copilot AI review requested due to automatic review settings May 9, 2026 00:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates consensus-critical generator validation/execution to support the post-HF2 INTERNED_GENERATOR path and adds an explicit pre-activation guard rejecting serde_2026-prefixed generators unless INTERNED_GENERATOR is enabled. It also introduces a dedicated validation error code for invalid generator encoding and adds unit tests covering the relevant flag/prefix combinations.

Changes:

  • Add InvalidTransactionsGeneratorEncoding (mapped to error code 150) and use it to explicitly reject serde_2026 generators before INTERNED_GENERATOR activation.
  • Implement the INTERNED_GENERATOR execution path in run_block_generator2 by deserializing via node_from_bytes_auto, interning the tree, and computing base cost via total_cost_from_tree.
  • Switch trusted-block generator parsing to node_from_bytes_auto and add targeted tests for check_generator_quote / check_generator_node.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
crates/chia-consensus/src/validation_error.rs Adds a new consensus error code for invalid transactions generator encoding and maps it to numeric code 150.
crates/chia-consensus/src/run_block_generator.rs Adds serde_2026 prefix guarding, implements INTERNED_GENERATOR handling in run_block_generator2, updates trusted parsing to auto-detect serialization, and adds tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/chia-consensus/src/run_block_generator.rs Outdated
@coveralls-official

coveralls-official Bot commented May 9, 2026

Copy link
Copy Markdown

Coverage Report for CI Build 30489236762

Coverage increased (+0.08%) to 81.894%

Details

  • Coverage increased (+0.08%) from the base build.
  • Patch coverage: 7 uncovered changes across 2 files (123 of 130 lines covered, 94.62%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
crates/chia-consensus/src/run_block_generator.rs 119 113 94.96%
crates/chia-consensus/src/serde_2026.rs 10 9 90.0%
Total (3 files) 130 123 94.62%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 20010
Covered Lines: 16387
Line Coverage: 81.89%
Coverage Strength: 11650890.35 hits per line

💛 - Coveralls

@richardkiss
richardkiss force-pushed the interned-generator-consensus branch from b5e3d34 to a3743a9 Compare May 20, 2026 17:11
@richardkiss
richardkiss force-pushed the serde-2026-wiring branch 2 times, most recently from 72bb358 to 7ace3d5 Compare May 22, 2026 00:58
@richardkiss
richardkiss force-pushed the serde-2026-wiring branch 6 times, most recently from 9a438ab to 2cd665e Compare July 16, 2026 05:08
Base automatically changed from serde-2026-wiring to main July 16, 2026 18:04
run_block_generator2: when INTERNED_GENERATOR flag is set, deserialize
via node_from_bytes_auto (accepts classic, backrefs, or serde_2026),
intern the tree, and compute cost via interned_vbytes * cost_per_byte
instead of raw byte length. The deserialization blob cap is derived
from the network cost constants via max_canonical_blob_size().

check_generator_quote: add CRITICAL guard that explicitly rejects any
serde_2026-prefixed generator when INTERNED_GENERATOR is NOT active.
Without this, a serde_2026 generator would be silently accepted on the
pre-HF2 chain between code shipping and SIMPLE_GENERATOR activation.

The trusted-block coinspend extractors also dispatch through
node_from_bytes_auto so they can parse serde_2026 blocks post-HF2.

Also adds ErrorCode::InvalidTransactionsGeneratorEncoding (150) for
the new rejection path, and tests covering all four flag combinations:
- pre-SIMPLE_GENERATOR + serde_2026 -> rejected
- SIMPLE_GENERATOR only + serde_2026 -> rejected
- INTERNED_GENERATOR + serde_2026 -> accepted
- SIMPLE_GENERATOR | INTERNED_GENERATOR + serde_2026 -> accepted

Co-authored-by: Cursor <cursoragent@cursor.com>
@richardkiss
richardkiss force-pushed the interned-generator-consensus branch from 36f5311 to 25bb8cf Compare July 16, 2026 18:41
Comment thread crates/chia-consensus/src/run_block_generator.rs
richardkiss and others added 2 commits July 23, 2026 13:26
Per review: minimize new code paths when INTERNED_GENERATOR is not set.
The serde_2026 magic prefix starts with 0xfd, an invalid header byte in
classic CLVM serialization, so deployed nodes already reject such blobs
via node_from_bytes_backrefs() -> GeneratorRuntimeError. Remove the
explicit prefix guard and the InvalidTransactionsGeneratorEncoding
error code, and gate the node_from_bytes_auto() call sites in the
trusted-block helpers on the flag so the flag-off path is identical
to main.

Co-authored-by: Cursor <cursoragent@cursor.com>
The INTERNED_GENERATOR bypasses were too broad: they skipped the quote
checks for every program, re-allowing complex generators after
soft_fork9 whenever the interned flag is on. Narrow them:

- check_generator_quote() only skips the byte-level check for blobs
  that actually carry the serde_2026 magic prefix (whose header cannot
  match the [0xff, 0x01] quote shape); classic/backrefs encodings keep
  the byte check post-activation.
- check_generator_node() no longer skips on INTERNED_GENERATOR,
  restoring main's behavior. It is the quote enforcement point for
  serde_2026 blobs, which can only be checked post-deserialization.

Co-authored-by: Cursor <cursoragent@cursor.com>

@arvidn arvidn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the SIMPLE_GENERATOR flag was part of the 2.7.0 soft fork, and has activated. Since we used generator programs in the initial version of block compression we can't require simple generators throughout the whole chain, so we still need this flag to validate early blocks.

We don't need to support (and maybe even should fail) the combination INTERNED_GENERATOR set and SIMPLE_GENERATOR not set. If there's any opportunity to simplify anything.

One crazy idea is that the interned serialization could have an implicit quote around the whole expression.

Comment thread crates/chia-consensus/src/run_block_generator.rs Outdated
Comment thread crates/chia-consensus/src/run_block_generator.rs Outdated
Comment thread crates/chia-consensus/src/run_block_generator.rs Outdated
@richardkiss

Copy link
Copy Markdown
Contributor Author

the SIMPLE_GENERATOR flag was part of the 2.7.0 soft fork, and has activated. Since we used generator programs in the initial version of block compression we can't require simple generators throughout the whole chain, so we still need this flag to validate early blocks.

We don't need to support (and maybe even should fail) the combination INTERNED_GENERATOR set and SIMPLE_GENERATOR not set. If there's any opportunity to simplify anything.

One crazy idea is that the interned serialization could have an implicit quote around the whole expression.

We discussed this with Bram on the weekly architecture call a while back and he agreed to dropping the (q . X) wrapper — writing just X and so not running it at all, so the block is no longer a generator but carries a list of spends.

I'd propose that as a separate change on top of this one: it deletes the quote checks for this path entirely and changes what the generator tree hash commits to, so it touches the builders too. At the same time we could rename the flag to something like SERDE_2026_TX_LIST since it's no longer a generator, and the interning is just how the cost is computed.

Or I can do it all in this change if you prefer, since the interaction is a bit intricate. Let me know.

I can make INTERNED_GENERATOR without SIMPLE_GENERATOR a hard failure in this PR.

Comment thread crates/chia-consensus/src/build_interned_block.rs
@richardkiss
richardkiss force-pushed the interned-generator-consensus branch 2 times, most recently from 92ea0bd to bc87bba Compare July 28, 2026 01:17

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit bc87bba. Configure here.

Comment thread crates/chia-consensus/src/build_interned_block.rs
Per review: with INTERNED_GENERATOR active, serde_2026 is the only legal
generator encoding. Consensus paths now parse via node_from_bytes_2026
(magic prefix mandatory, no classic/backrefs fallback);
node_from_bytes_auto remains for non-consensus readers.
InternedBlockBuilder::finalize() emits serde_2026 accordingly.

The byte-level quote check has nothing left to do in this mode (the
encoding is enforced at parse time and can't be examined for the quote
shape), so it defers entirely to check_generator_node().

Co-authored-by: Cursor <cursoragent@cursor.com>
@richardkiss
richardkiss force-pushed the interned-generator-consensus branch from bc87bba to 4c9de44 Compare July 28, 2026 01:27
@danieljperry danieljperry changed the title Add INTERNED_GENERATOR consensus path + serde_2026 prefix guard [CHIA-3823] Add INTERNED_GENERATOR consensus path + serde_2026 prefix guard Jul 28, 2026
Per review: propagate the deserializer's SerializationError (the same
error the non-interned path produces for undecodable generators)
instead of mapping to GeneratorRuntimeError at the call site.

Co-authored-by: Cursor <cursoragent@cursor.com>
@richardkiss
richardkiss merged commit 8d1fa1d into main Aug 3, 2026
84 checks passed
@richardkiss
richardkiss deleted the interned-generator-consensus branch August 3, 2026 18:07
richardkiss added a commit that referenced this pull request Aug 14, 2026
serde_2026 acceptance activates at exactly the same height as
INTERNED_GENERATOR (single activation), and InternedBlockBuilder's cost
model (charging by interned vbytes) is only correct once INTERNED_GENERATOR
is active. There is no height at which "interned builder + classic
emission" is valid, so the default-off constructor flag was a footgun: a
default-off caller post-fork would emit blocks that fail #1438's strict
validation. finalize() now unconditionally serializes with serialize_2026.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants