Skip to content

[CHIA-3854] Expose generator_interned_weight() to Python - #1418

Closed
richardkiss wants to merge 22 commits into
mainfrom
expose-interned-weight
Closed

[CHIA-3854] Expose generator_interned_weight() to Python#1418
richardkiss wants to merge 22 commits into
mainfrom
expose-interned-weight

Conversation

@richardkiss

@richardkiss richardkiss commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Stacked on top of #1377 (now merged).

  • Splits the interned tree cost calculation into interned_weight() (raw weight) and total_cost_from_tree() (weight × COST_PER_BYTE)
  • Exposes generator_interned_weight(program) -> int to Python: deserializes a generator, interns the tree, returns atom_bytes + 2*atoms + 3*pairs
  • Python callers multiply by constants.COST_PER_BYTE themselves, reusing the existing consensus constant instead of hardcoding 12000

This lets BlockTools.compute_block_cost in chia-blockchain compute the correct size cost when INTERNED_GENERATOR is active:

if prev_tx_height >= constants.HARD_FORK2_HEIGHT:
    size_cost = generator_interned_weight(bytes(generator)) * constants.COST_PER_BYTE
else:
    size_cost = len(bytes(generator)) * constants.COST_PER_BYTE

Test plan

  • Existing generator_cost Rust tests pass (refactor is behavior-preserving)
  • CI

Made with Cursor

@coveralls-official

coveralls-official Bot commented Apr 16, 2026 •
edited
Loading

Copy link
Copy Markdown

Coverage Report for CI Build 25578195093

Coverage decreased (-0.4%) to 80.198%

Details

  • Coverage decreased (-0.4%) from the base build.
  • Patch coverage: 210 uncovered changes across 8 files (467 of 677 lines covered, 68.98%).
  • 1 coverage regression across 1 file.

Uncovered Changes

File Changed Covered %
crates/chia-consensus/src/build_interned_block.rs 121 0 0.0%
crates/chia-consensus/src/build_block_2026.rs 416 368 88.46%
wheel/src/api.rs 24 9 37.5%
wheel/src/run_generator.rs 9 0 0.0%
crates/chia-consensus/src/solution_generator.rs 8 0 0.0%
crates/chia-protocol/src/program.rs 11 6 54.55%
crates/chia-consensus/src/run_block_generator.rs 42 40 95.24%
crates/chia-consensus/src/serde_2026.rs 8 6 75.0%

Coverage Regressions

1 previously-covered line in 1 file lost coverage.

File Lines Losing Coverage Coverage
crates/chia-protocol/src/program.rs 1 56.46%

Coverage Stats

Coverage Status
Relevant Lines: 18993
Covered Lines: 15232
Line Coverage: 80.2%
Coverage Strength: 11865135.21 hits per line
💛 - Coveralls

@richardkiss richardkiss mentioned this pull request Apr 22, 2026
Closed
3 tasks
@danieljperry danieljperry changed the title Expose generator_interned_weight() to Python [CHIA-3854] Expose generator_interned_weight() to Python Apr 24, 2026
@richardkiss
richardkiss force-pushed the expose-interned-weight branch from 1b430ce to 2d99df9 Compare May 2, 2026 00:20
richardkiss and others added 19 commits May 8, 2026 12:27
@richardkiss
Expose generator_interned_weight() to Python
8ec797a 
Split the interned tree cost calculation into two parts:
- interned_weight(): returns atom_bytes + 2*atoms + 3*pairs
- total_cost_from_tree(): interned_weight() * COST_PER_BYTE

The new generator_interned_weight() Python function deserializes a
generator program, interns the tree, and returns the raw weight.
Callers multiply by the COST_PER_BYTE consensus constant themselves,
avoiding a hardcoded multiplier on the Python side.

Made-with: Cursor
@richardkiss
Add generator_interned_weight to type stub generator
ddebde8 
The .pyi had it manually but generate_type_stubs.py didn't,
causing the Check chia_rs.pyi CI job to fail.

Made-with: Cursor
@richardkiss
add InternedBlockBuilder for INTERNED_GENERATOR cost model
169bfdc 
A clean, separate builder that avoids the Serializer/sentinel/restore
complexity of BlockBuilder. Cost is computed from total_cost_from_tree
on the interned quoted generator tree after each add; serialization
happens once in finalize() via node_to_bytes_backrefs. Includes Python
bindings matching the BlockBuilder interface.

Made-with: Cursor
@richardkiss
Wire serde_2026 serialization into chia_rs
1513655 
- Patch workspace clvmr to Chia-Network/clvm_rs@serde_2026 (v0.17.1,
  feature ser-2026) and lower version requirement from 0.17.4 to 0.17
- Add solution_generator_2026() using node_to_bytes_serde_2026
- Switch INTERNED_GENERATOR path in run_block_generator2 from
  node_from_bytes_backrefs to node_from_bytes_auto (auto-detect format)
- Replace intern_tree_limited / intern_tree (not in serde_2026 branch)
  with intern() in run_block_generator, spendbundle_conditions,
  generator_cost tests
- Drop ClvmFlags::LIMITS / ClvmFlags::MALACHITE mappings in flags.rs;
  those flags were added in clvmr 0.17.4 which post-dates serde_2026
- Add Python binding solution_generator_2026 + pyi / stub stubs

Made-with: Cursor
@richardkiss
Add Block2026Builder + switch non-consensus readers to node_from_byte…
01dbc9d 
…s_auto

Block2026Builder: anytime block builder for post-HF2 (serde_2026).
Accepts candidates in priority order, packs greedily with upper-bound
cost estimates, validates with exact interned_weight costing, then
refines in a background thread. Caller retrieves best block instantly
via best() which also returns included_indices for metadata tracking.
Usable as Python context manager.

Non-consensus CLVM readers (additions_and_removals, get_puzzle_and_solution,
get_coinspends_for_trusted_block, run_chia_program, generator_interned_weight)
switched from node_from_bytes_backrefs to node_from_bytes_auto so they
accept classic, backrefs, and serde_2026 formats transparently.

Consensus-critical paths unchanged (gated on INTERNED_GENERATOR flag).

Made-with: Cursor
@richardkiss
add Program.from_program_bytes() + Block2026Builder stubs
d1b38a7 
Program.from_program_bytes(blob) wraps raw bytes as a Program without
CLVM structure validation. Needed for serde_2026 format generators
which are validated at execution time by run_block_generator /
node_from_bytes_auto, not at construction time.

Also adds Block2026Builder type stubs to the .pyi file.

Made-with: Cursor
@richardkiss
add tree_hash_auto() for serde_2026 / backrefs programs
9a18f2b 
tree_hash_auto() deserializes using node_from_bytes_auto (handles
standard CLVM, backrefs, and serde_2026) then computes the tree hash.
Needed for generator_root computation on serde_2026 blocks.

Made-with: Cursor
@richardkiss
skip SIMPLE_GENERATOR quote check for INTERNED_GENERATOR blocks
11b6939 
serde_2026 blocks don't start with [0xff, 0x01] (standard CLVM quote
prefix) since the serialization format is different. When
INTERNED_GENERATOR is active, bypass the SIMPLE_GENERATOR quote check
in both the raw bytes and deserialized node validators.

Made-with: Cursor
@richardkiss
accept serde_2026 format in Program.from_json_dict
586cc08 
When deserializing a Program from JSON, try standard CLVM
serialized_length first. If that fails, fall back to
node_from_bytes_auto which handles backrefs and serde_2026. This
allows FullBlock JSON round-trips for blocks with serde_2026
generators (e.g. via RPC).

Made-with: Cursor
@richardkiss
add tests: SIMPLE_GENERATOR guard, tree_hash_auto equivalence
6795360 
- 5 tests for check_generator_quote/check_generator_node: verify
  SIMPLE_GENERATOR still rejects non-quote generators when
  INTERNED_GENERATOR is NOT set (pre-HF2 safety)
- 1 test for tree_hash_auto: verify it produces the same hash as
  tree_hash_from_bytes for standard CLVM and backrefs, and also
  works for serde_2026 (which tree_hash_from_bytes rejects)

Made-with: Cursor
@richardkiss
fix Streamable round-trip for serde_2026 generators
71d90ee 
Program::parse() now auto-detects serde_2026 magic prefix and uses
serialized_length_serde_2026() to delimit the blob. Without this,
FullBlock serialization/deserialization fails for post-HF2 blocks
that use serde_2026 generators.

Also updates clvmr dep to pick up the new function, and adapts to
API renames (intern -> intern_tree, node_from_bytes_auto gains
DeserializeLimits parameter).

Made-with: Cursor
@richardkiss
fix CI: fmt, clippy, stubtest, and trailing-bytes JSON parse
f160de8 
- cargo fmt across files modified by serde_2026 wiring
- clippy::manual_midpoint -> u64::midpoint in build_block_2026.rs
- supply Compression::default() to node_to_bytes_serde_2026 in tree_hash.rs test
- generate_type_stubs.py: replace embedded triple-quoted docstring with
  comments (was a Python SyntaxError when parsed by 3.14)
- generate_type_stubs.py: add tree_hash_auto stub (matches public API)
- Block2026Builder: add @disjoint_base + emit @Final for stubtest
- Program.from_json_dict: detect serde_2026 prefix and require all bytes
  consumed; raise InvalidClvm on trailing garbage (restoring chia/main
  behavior the test_streamable test relies on)
- regenerate chia_rs.pyi
@richardkiss
fix CI: map MALACHITE flag and allowlist 3.9 stubtest conflict
714c761 
- ConsensusFlags::from_clvm_flags / to_clvm_flags: add MALACHITE branch
  so the round-trip test (and ClvmFlags::FLAGS coverage check) passes
  against clvmr 0.17.7 which now defines that flag.
- Block2026Builder uses both @Final + @disjoint_base, which 3.10+ accepts
  but 3.9 stubtest rejects ("remove @disjoint_base"). Add it to the
  disjoint-base allowlist (already wired to 3.9 only) to silence that.
@richardkiss
fix CI: move Block2026Builder stubtest allowlist to 3-9 only
e81ca09 
The 3.10/3.11 disjoint-base allowlist would flag chia_rs.Block2026Builder
as an unused entry, since those Python versions accept @Final +
@disjoint_base together. Only 3.9 needs to silence the
"remove @disjoint_base" complaint.
@richardkiss
serde_2026: track upstream level: u32 API, drop Compression imports
3d2ef94 
clvm_rs PR #708 replaced the public `Compression` enum with a saturating
`level: u32` API. Update the three call sites here and bump the patched
clvmr pin to a67ee12a269000bb6e20266bfa40769bc50ab993:

- crates/chia-consensus/src/build_block_2026.rs
- crates/chia-consensus/src/solution_generator.rs
- crates/clvm-utils/src/tree_hash.rs

Each previously called the function as `..._serde_2026(a, n, Compression::default())`;
now just `..._serde_2026(a, n)`. The default level (0) is unchanged.

Made-with: Cursor
@richardkiss
serde_2026: track upstream API drop of DeserializeOptions
f3258c5 
clvm_rs no longer ships DeserializeOptions / DEFAULT_MAX_ATOM_LEN /
DEFAULT_MAX_INPUT_BYTES / node_from_bytes_auto — those defaults were
consensus-flavored without being consensus, and Arvid pushed back on
having them in clvm_rs. They live here now, where they belong.

Adds chia_consensus::serde_2026 with:
  - CONSENSUS_MAX_ATOM_LEN (= 1 MiB, matches the old default exactly)
  - node_from_bytes_auto(a, &[u8]) — sniffs SERDE_2026_MAGIC_PREFIX and
    dispatches to deserialize_2026 (with consensus caps) or
    node_from_bytes_backrefs

All 8 in-tree call sites switch to the new helper:
  - chia-consensus: additions_and_removals, run_block_generator (×3)
  - wheel: api (tree_hash_auto, get_puzzle_and_solution_*),
    run_generator, run_program

clvm-utils stays decoupled: its tree_hash test inlines the same
sniff-and-dispatch with its own 1 MiB constant.

Bumps clvmr to 37e09884 (the matching clvm_rs PR head).

Made-with: Cursor
@richardkiss
serde_2026: track upstream rename + new node_from_bytes_serde_2026
86bb49a 
clvm_rs renamed `deserialize_2026{,_from_stream}` to make the body-only
contract visible (`_body` suffix), added `node_from_bytes_serde_2026` as
the prefix-aware counterpart to `node_to_bytes_serde_2026`, and
parametrized `serialized_length_serde_2026` so it mirrors every
header-time validation the body deserializer does.

Updates:
  - chia_consensus::serde_2026::node_from_bytes_auto: use the new
    `node_from_bytes_serde_2026` (no manual prefix stripping).
  - chia-protocol::Program::{parse, from_json_dict}: pass
    `(usize::MAX, false)` to the length helper. Framing-only callers
    don't claim a consensus opinion; max_atom_len is enforced later, at
    run_block_generator time.
  - clvm-utils::tree_hash test: use `node_from_bytes_serde_2026`.

Bumps clvmr to af8cb91b.

Made-with: Cursor
@richardkiss @cursoragent
expose SERDE_2026_MAGIC_PREFIX as Python constant
873a1e1 
chia-blockchain needs the magic prefix to gate serde_2026 generators
behind HF2 (reject pre-HF2, accept after). The constant lives canonically
in clvmr::serde, but PyPI clvm_rs is capped at 0.2.x by chia-puzzles-py
and chia-base, so importing from clvm_rs Python isn't viable yet.
Re-export through chia_rs (which already pulls clvmr from git) until
serde_2026 ships and downstream caps relax.

Co-authored-by: Cursor <cursoragent@cursor.com>
@richardkiss @cursoragent
fixup: adapt to clvm_rs serde_2026 tip API after rebase onto main
146353a 
- serialize_2026/deserialize_2026 instead of node_to/from_bytes_serde_2026
  (the latter aliases were added in a now-superseded upstream commit)
- program_bytes::node_from_bytes_auto re-exports serde_2026::node_from_bytes_auto
- ValidationErr(.1) test refs changed to .0 (NodePtr field dropped on main)
- intern_tree_limited retained (intern_tree not yet in serde_2026 tip)

Co-authored-by: Cursor <cursoragent@cursor.com>
@richardkiss
richardkiss force-pushed the expose-interned-weight branch from 241aa88 to 146353a Compare May 8, 2026 20:14
richardkiss and others added 3 commits May 8, 2026 13:16
@richardkiss @cursoragent
fmt: cargo fmt --all
cfb3ce7 
Co-authored-by: Cursor <cursoragent@cursor.com>
@richardkiss @cursoragent
fix: remove duplicate node_from_bytes_auto imports and add consensus …
630a602 
…blob size cap

Remove duplicate `use chia_consensus::serde_2026::node_from_bytes_auto` imports
in wheel/src/api.rs and wheel/src/run_program.rs that caused E0252 build failures.

Add explicit 10 MiB blob size cap (`CONSENSUS_MAX_BLOB_SIZE`) in
`serde_2026::node_from_bytes_auto`, matching the consensus policy now that
clvm_rs serde_2026 no longer supplies defaults. Atom len cap (1<<20) was already
in place via `CONSENSUS_MAX_ATOM_LEN`.

Co-authored-by: Cursor <cursoragent@cursor.com>
@richardkiss @cursoragent
fix: update clvm-utils tests to use serialize_2026/deserialize_2026 A…
36eec19 
…PI names

Co-authored-by: Cursor <cursoragent@cursor.com>
@richardkiss

richardkiss commented May 9, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of sharded PRs for reviewability:

This approach enables:

  • Parallel review of independent pieces
  • Clear dependency chain
  • Easier rollback if issues found
  • Lower cognitive load per reviewer

@richardkiss richardkiss closed this May 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@richardkiss