serde_2026 block support by richardkiss · Pull Request #1439 · Chia-Network/chia_rs

richardkiss · 2026-05-09T00:25:46Z

Plumbing for serde_2026-serialized (post-HF2) block generators, plus serde_2026 emission
for the existing InternedBlockBuilder (#1436). Stacked on #1438.

What's here

Program::parse / from_json_dict recognize the serde_2026 magic prefix and use a
framing-only walk to find the byte boundary. Consensus caps are still enforced at
run_block_generator time. Classic-format inputs are unaffected: a leading 0xfd
byte was never parseable as classic CLVM serialization, so the only newly-accepted
inputs are previously-invalid serde_2026-prefixed blobs.
Accepting the prefix at parse time before the fork is deliberate: this release
ships ahead of activation and must be able to parse post-fork blocks without another
release. Consensus enforcement stays in the flag-conditioned run_block_generator2 —
pre-fork, a block carrying a serde_2026 generator is still invalid; the rejection
just moves from message parsing to generator execution.
tree_hash_auto(): tree hash of a blob in any supported serialization format.
Hashing is DAG-aware (tree_hash_cached): this gets called on unvalidated blobs
(the post-HF2 generator_root check on incoming blocks), and compact encodings can
have exponential expansion — the naive walk would be a pre-charge DoS. Work is
linear in unique nodes, which parsing bounds by the blob length.
(An earlier revision also added an unvalidated Program.from_program_bytes()
constructor for the emission side; it was dropped — since parse handles
serde_2026 framing, from_bytes serves that role with validation. Generator
blobs stay bytes until they cross into a Program.)
Non-consensus readers (additions_and_removals, wheel run_chia_program,
get_puzzle_and_solution_for_coin/2) switch to node_from_bytes_auto; identical
behavior for classic-format input. The dispatcher imposes no size policy in either
branch — trusted readers of historical blocks must accept every blob the chain ever
accepted, so the classic branch is byte-for-byte node_from_bytes_backrefs. The
cost-derived size cap is enforced only at the consensus entry point
(node_from_bytes_2026).
InternedBlockBuilder's finalize() always serializes the generator with
serialize_2026 — there is no classic-emission mode. serde_2026 acceptance activates
at exactly the same height as INTERNED_GENERATOR (single activation), and the
builder's cost accounting (interned vbytes) is only correct once INTERNED_GENERATOR
is active, so there is no height at which "interned builder + classic emission" would
be valid. A caller-controlled flag would have been a footgun: a default-off caller
post-fork would emit blocks that fail strict post-fork validation.

What moved out

The aggressive anytime Block2026Builder is no longer part of this PR. It becomes a
separate opt-in optimization PR later (seed parked at
park/block-2026-builder-optimization).

Tests

serde_2026 round-trip through run_block_generator2 + INTERNED_GENERATOR, with
spends/conditions matching an independently-built classic-format generator of the
same bundles run under classic rules, and finalize() cost matching the consensus
cost exactly.
tree_hash_auto agreement between serde_2026 and classic serializations of the same
tree.

Note

High Risk
Touches consensus-adjacent generator parsing, tree hashing of unvalidated blobs, and InternedBlockBuilder emission. Size-cap policy is split between consensus (node_from_bytes_2026) and policy-free historical readers.

Overview
Enables serde_2026 (post-HF2 interned) block generators end-to-end: parse them on the wire, hash them DAG-aware, run them on non-consensus paths, and always emit them from InternedBlockBuilder.

Readers vs consensus. node_from_bytes_auto is now a policy-free sniff-and-dispatch (no size cap; classic branch is still node_from_bytes_backrefs). The cost-derived blob cap lives only on node_from_bytes_2026. Trusted/historical readers (additions_and_removals, wheel run_chia_program, puzzle/solution extractors) switch to auto.

Protocol / Python. Program parse and JSON accept the magic prefix with a framing-only length walk (caps still at run_block_generator). Adds is_serde_2026_encoded, DAG-aware get_tree_hash for 2026 blobs, and Python tree_hash_auto for unvalidated generator_root checks.

Builder. InternedBlockBuilder::finalize() always serializes with serde_2026 (no classic-emission mode), matching interned-vbyte cost accounting and the single INTERNED_GENERATOR activation. Tests check round-trip spends vs a classic reference generator.

^{Reviewed by Cursor Bugbot for commit 1f16531. Bugbot is set up for automated code reviews on this repo. Configure here.}

Copilot

Pull request overview

Adds Python-facing support for building and hashing post-HF2 (INTERNED_GENERATOR / serde_2026) blocks, including a new anytime block builder, serde_2026-aware tree hashing, and serde_2026 framing support for Program round-trips.

Changes:

Introduces Block2026Builder (Rust + Python bindings/stubs) for building serde_2026 generators under the INTERNED_GENERATOR cost model.
Adds tree_hash_auto() (Python) and updates several readers/execution entrypoints to accept classic/backrefs/serde_2026 via node_from_bytes_auto.
Extends Program parsing/JSON round-trip to recognize the serde_2026 magic prefix, and adds Program.from_program_bytes() for unvalidated wrapping.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
wheel/stubtest.allowlist.3-9	Allowlists `Block2026Builder` for Python 3.9 stubtest edge-case.
wheel/src/run_program.rs	Switches `run_chia_program` deserialization to `node_from_bytes_auto`.
wheel/src/api.rs	Exposes `Block2026Builder`, `tree_hash_auto`, and re-exports `SERDE_2026_MAGIC_PREFIX`; updates deserialization paths to auto-detect formats.
wheel/python/chia_rs/chia_rs.pyi	Updates Python stubs: adds `Block2026Builder`, `tree_hash_auto`, `Program.from_program_bytes`, and moves `SERDE_2026_MAGIC_PREFIX`.
wheel/generate_type_stubs.py	Updates stub generation rules for new APIs/constants.
crates/clvm-utils/src/tree_hash.rs	Adds a unit test ensuring tree hash equality across standard/backrefs/serde_2026 encodings.
crates/chia-protocol/src/program.rs	Adds serde_2026-aware framing in `Streamable::parse`/`FromJsonDict`; adds Python `Program.from_program_bytes`.
crates/chia-consensus/src/lib.rs	Exposes new `build_block_2026` module.
crates/chia-consensus/src/build_block_2026.rs	Implements `Block2026Builder` anytime builder + tests + Python bindings.
crates/chia-consensus/src/additions_and_removals.rs	Switches trusted generator parsing to `node_from_bytes_auto`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

coveralls-official · 2026-05-09T01:12:29Z

Coverage Report for CI Build 32538043764

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage increased (+0.3%) to 82.23%

Details

Coverage increased (+0.3%) from the base build.
Patch coverage: 15 uncovered changes across 3 files (258 of 273 lines covered, 94.51%).
95 coverage regressions across 6 files.

Uncovered Changes

File	Changed	Covered	%
wheel/src/api.rs	15	6	40.0%
crates/chia-protocol/src/program.rs	92	88	95.65%
crates/chia-consensus/src/build_interned_block/additional_tests.rs	94	92	97.87%
Total (8 files)	273	258	94.51%

Coverage Regressions

95 previously-covered lines in 6 files lost coverage.

File	Lines Losing Coverage	Coverage
crates/chia-protocol/src/fullblock.rs	41	75.9%
crates/chia-consensus/src/conditions.rs	23	99.08%
crates/chia-protocol/src/unfinished_block.rs	20	82.17%
crates/chia-consensus/src/flags.rs	6	95.45%
crates/chia-protocol/src/bytes.rs	3	90.99%
crates/chia-protocol/src/program.rs	2	71.08%

Coverage Stats


Relevant Lines:	20287
Covered Lines:	16682
Line Coverage:	82.23%
Coverage Strength:	11438780.77 hits per line

💛 - Coveralls

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4972b1b. Configure here.}

General plumbing for serde_2026-serialized (post-HF2) block generators, plus an emission mode for the existing InternedBlockBuilder: - Program::parse / from_json_dict recognize the serde_2026 magic prefix and use a framing-only walk to find the byte boundary. Consensus caps are still enforced at run_block_generator time. - Program.from_program_bytes() wraps raw bytes without CLVM structure validation, for non-standard serializations. - tree_hash_auto() computes the tree hash of a blob in any supported serialization format. - Non-consensus readers (additions_and_removals, run_chia_program, get_puzzle_and_solution_for_coin/2) switch to node_from_bytes_auto. - InternedBlockBuilder gains a new_serde_2026() named constructor. Builders created with it serialize the generator with serialize_2026 in finalize() instead of the classic back-ref format. new() and new_with() keep their original signatures, and the classic path is unchanged and remains byte-identical. The Python binding keeps the serde_2026 kwarg (default False) and dispatches to the matching constructor. The builder does not see consensus flags; the caller decides based on INTERNED_GENERATOR activation. The aggressive anytime Block2026Builder moves out of this PR to a separate opt-in optimization PR (parked as park/block-2026-builder-optimization). Co-authored-by: Cursor <cursoragent@cursor.com>

serde_2026 acceptance activates at exactly the same height as INTERNED_GENERATOR (single activation), and InternedBlockBuilder's cost model (charging by interned vbytes) is only correct once INTERNED_GENERATOR is active. There is no height at which "interned builder + classic emission" is valid, so the default-off constructor flag was a footgun: a default-off caller post-fork would emit blocks that fail #1438's strict validation. finalize() now unconditionally serializes with serialize_2026. Co-authored-by: Cursor <cursoragent@cursor.com>

The cap is derived from serde_2026 vbyte math and current cost constants; applying it to classic/backrefs blobs retroactively bounds historical blocks by today's rules. Trusted readers (e.g. additions_and_removals) must accept every blob the chain ever accepted, so the classic branch is now byte-for-byte node_from_bytes_backrefs, uncapped — same as before the dispatcher existed. Consensus paths are unaffected: node_from_bytes_2026 keeps its own gate. Co-authored-by: Cursor <cursoragent@cursor.com>

The unvalidated constructor let Python wrap arbitrary bytes as a Program. Now that Program::parse handles serde_2026 framing, callers building post-HF2 generators can use from_bytes and get the framing walk; blobs stay plain bytes until they cross into a Program. Revisit if/when the node adopts the v1 raw-buffer block framing, where the generator is bytes end-to-end. Co-authored-by: Cursor <cursoragent@cursor.com>

A cap that silently applied to only one branch was confusing. The dispatcher is the trusted-reader entry point, so it now takes no size policy at all: both branches parse uncapped, like node_from_bytes_backrefs always did. clvmr's per-atom bound is satisfied with bytes.len(), which is policy-free (an atom of length L appears literally in the blob). The cost-derived cap lives only in the consensus entry point, node_from_bytes_2026; the size-bound fuzz target now exercises that gate directly. Co-authored-by: Cursor <cursoragent@cursor.com>

tree_hash_auto is called on unvalidated blobs (the post-HF2 generator_root check hashes generators from blocks straight off the wire), and backrefs/serde_2026 can encode trees whose expansion is exponential in the blob size. The uncached tree_hash walks the full expansion — a pre-charge DoS. Switch to tree_hash_cached, which memoizes shared pairs and stays linear in unique nodes (bounded by blob length). Regression test: a 64-deep doubling chain (2^64-leaf expansion from a <1 KiB blob) hashes correctly via both compact formats. node_from_bytes_auto's doc now states the full contract: policy-free (never a consensus entry point), parsing is untrusted-safe (linear), but traversal of the result must be DAG-aware. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

get_tree_hash(), run_rust(), and uncurry_rust() only knew classic/backrefs encoding, so serde_2026-encoded Programs hit the wrong path: get_tree_hash's tree_hash_from_bytes(...).unwrap() panicked (a Python-reachable panic), and run_rust/uncurry_rust failed with a misleading "unexpected end of buffer" from node_from_bytes_backrefs. All three now dispatch on the magic prefix, matching chia_consensus::serde_2026::node_from_bytes_auto's semantics (duplicated here since chia-protocol can't depend on chia-consensus). get_tree_hash's serde_2026 branch hashes with the DAG-safe tree_hash_cached, agreeing with the wheel's tree_hash_auto; its classic branch is unchanged byte-for-byte. Since the serde_2026 branch can error, get_tree_hash's signature changes to PyResult<Bytes32> (raises ValueError instead of panicking); the .pyi stub type is unaffected since a raise doesn't change it. Co-authored-by: Cursor <cursoragent@cursor.com>

richardkiss · 2026-08-27T21:17:20Z

Replaced by #1511

Copilot AI review requested due to automatic review settings May 9, 2026 00:25

Copilot started reviewing on behalf of richardkiss May 9, 2026 00:26 View session

Copilot AI reviewed May 9, 2026

View reviewed changes

cursor Bot reviewed May 9, 2026

View reviewed changes

Comment thread crates/chia-consensus/src/build_block_2026.rs Outdated

Comment thread crates/chia-consensus/src/build_block_2026.rs Outdated

richardkiss mentioned this pull request May 9, 2026

[CHIA-3854] Expose generator_interned_weight() to Python #1418

Closed

2 tasks

richardkiss force-pushed the block-2026-builder branch from b661d55 to be7877a Compare May 20, 2026 17:11

richardkiss force-pushed the serde-2026-wiring branch 2 times, most recently from 72bb358 to 7ace3d5 Compare May 22, 2026 00:58

richardkiss force-pushed the serde-2026-wiring branch from 732e071 to fbdd7c3 Compare June 24, 2026 22:05

richardkiss force-pushed the serde-2026-wiring branch from fbdd7c3 to a3019b8 Compare July 2, 2026 20:04

richardkiss force-pushed the serde-2026-wiring branch 6 times, most recently from 9a438ab to 2cd665e Compare July 16, 2026 05:08

Base automatically changed from serde-2026-wiring to main July 16, 2026 18:04

richardkiss force-pushed the block-2026-builder branch from be7877a to fc51809 Compare July 16, 2026 19:05

richardkiss changed the base branch from main to interned-generator-consensus July 16, 2026 19:05

richardkiss force-pushed the block-2026-builder branch 2 times, most recently from 48dcbe4 to efb05fd Compare July 16, 2026 19:08

cursor Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread crates/chia-consensus/src/build_block_2026.rs Outdated

richardkiss force-pushed the block-2026-builder branch from efb05fd to eae8264 Compare July 23, 2026 20:53

cursor Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread crates/chia-protocol/src/program.rs

richardkiss force-pushed the block-2026-builder branch 2 times, most recently from ab0dd75 to c22ae9b Compare July 23, 2026 22:58

richardkiss force-pushed the interned-generator-consensus branch 2 times, most recently from bc87bba to 4c9de44 Compare July 28, 2026 01:27

richardkiss force-pushed the block-2026-builder branch from c22ae9b to 4972b1b Compare July 29, 2026 20:45

cursor Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread crates/chia-consensus/src/build_interned_block.rs Outdated

richardkiss force-pushed the block-2026-builder branch from 4972b1b to e3a373d Compare July 30, 2026 02:35

Base automatically changed from interned-generator-consensus to main August 3, 2026 18:07

richardkiss force-pushed the block-2026-builder branch from e3a373d to 3bb84c1 Compare August 3, 2026 18:12

richardkiss changed the title ~~Add Block2026Builder, tree_hash_auto, and Program.from_program_bytes~~ serde_2026 block support Aug 13, 2026

richardkiss and others added 7 commits August 13, 2026 17:11

Add Program.is_serde_2026_encoded transitional probe

5dfd6e8

Co-authored-by: Cursor <cursoragent@cursor.com>

richardkiss mentioned this pull request Aug 24, 2026

serde_2026 emission + trusted readers (minimal) #1511

Draft

richardkiss closed this Aug 27, 2026

Navigation Menu

Uh oh!

serde_2026 block support - #1439

serde_2026 block support#1439
richardkiss wants to merge 8 commits into
mainfrom
block-2026-builder

richardkiss commented May 9, 2026 •

edited by cursor Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls-official Bot commented May 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

richardkiss commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

richardkiss commented May 9, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's here

What moved out

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls-official Bot commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for CI Build 32538043764

Coverage increased (+0.3%) to 82.23%

Details

Uncovered Changes

Coverage Regressions

Coverage Stats

💛 - Coveralls

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

richardkiss commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

richardkiss commented May 9, 2026 •

edited by cursor Bot

Loading

coveralls-official Bot commented May 9, 2026 •

edited

Loading