Skip to content

methodical codegen switch - #17308

Open
kasey wants to merge 7 commits into
methodical-codegen-driverfrom
methodical-codegen-switch
Open

methodical codegen switch#17308
kasey wants to merge 7 commits into
methodical-codegen-driverfrom
methodical-codegen-switch

Conversation

@kasey

@kasey kasey commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Generate SSZ with methodical-ssz instead of fastssz

Part of the gloas-devnet-7 stacked series; based on methodical-codegen-driver, diff shown against it.

This replaces the fastssz sszgen codegen path with methodical-ssz. Generation targets are now declared in a per-package yaml config that lists the types to generate, rather than as objs/exclude_objs lists spread across BUILD.bazelproto/prysm/v1alpha1/BUILD.bazel alone loses roughly 300 lines of hand-maintained, order-dependent type lists (each fork's target had to repeat every earlier fork's list in exclude_objs).

Each target now emits a mainnet/minimal pair of build-tagged files unconditionally, instead of collapsing to a single untagged file when the two happen to be produce the same output (previous behavior/assumptions of build/gen). Keeping both files means the generated file set does not shift as builds are performed for different presets — under the old scheme a .minimal.ssz.go twin would appear or vanish depending on whether the generated code for the two different presets happened to coincide.

The change also lays the groundwork for progressive merkleization: a new //tools:disable_progressive_merkleization build flag is threaded through every codegen action. No type carries progressive annotations yet, so the flag is a no-op until they are added.

Scale and reading guidance

188 files, ~67k insertions — but 32 of those files are generated (~66k of the insertions). Skip every *.ssz.go, *.minimal.ssz.go and *.pb.go file. The reviewable surface is about 1,400 lines across:

  • tools/methodical.bzl (new, 199 lines) — the Bazel rules
  • build/gen/ — the make gen path
  • 13 new proto/**/*.yaml target configs
  • 49 BUILD.bazel files (mostly dep swaps)
  • ~91 hand-written Go files, the large majority of which are a mechanical fastssz→methodical type swap

Key changes

Bazel codegen rules

  • tools/ssz.bzl (the ssz_gen_marshal rule and SSZ_DEPS) is deleted; tools/methodical.bzl replaces it. No SSZ_DEPS references remain in the tree.
  • ssz_methodical cannot use the go toolchain inside the Bazel sandbox, so it feeds methodical a pre-built package inventory through the genception GOPACKAGESDRIVER. The rule walks go_pkg_info_aspect outputs over deps plus the injected ssz_lib, writes the collected package JSON paths (plus the stdlib entry) to methodical-pkg-list.json, and runs the tool with PACKAGE_JSON_INVENTORY / GOPACKAGESDRIVER / PACKAGES_BASE set. tools/genception/BUILD.bazel aliases methodicalgen to @com_github_offchainlabs_methodical_ssz//cmd/ssz.
  • ssz_gen_spectest is also added: it interpolates names from a yaml config and copies fixtures out of a consensus-spec-tests release tarball, needing none of the package-driver apparatus. It is defined but not yet wired to any target; .gitignore gains testing/spectest/methodical for its (large) generated output.
  • //tools:disable_progressive_merkleization is a new bool_flag in tools/BUILD.bazel (default False). ssz_methodical reads it via BuildSettingInfo and appends --disable-progressive, so bazel build --//tools:disable_progressive_merkleization //... flips every target with no per-rule edits.

Declarative target configs

  • 13 new yaml configs, each naming a package import path and the types to generate: proto/prysm/v1alpha1/{phase0,altair,bellatrix,capella,deneb,electra,fulu,gloas,non-core}.yaml, proto/engine/v1/engine.yaml, proto/eth/v1/gateway.yaml, proto/ssz_query/response.yaml, proto/ssz_query/testing/test_containers.yaml.
  • ssz_methodical rules take config_file, out, deps and an optional override_package_name (e.g. eth for proto/prysm/v1alpha1, v1 for proto/eth/v1).
  • The separate gloas_builder_api target and its gloas_builder_api.ssz.go output are gone; RequestAuthV1, SignedRequestAuthV1, BuilderPreferencesV1 and BuilderPreferencesRequestV1 are folded into gloas.yaml.
  • proto/eth/v1/gateway.yaml picks up BeaconBlockBody (not previously generated) and uses the generated Deposit_Data name.

make gen path (build/gen/)

  • loadSSZTargetsloadMethodicalTargets, now reading ssz_methodical rules and returning methodicalTarget{pkg, configFile, out, overridePkg}. The Starlark string-list evaluator (evalStringList, attrStringList, splitIncludes) is deleted along with the .pb.go staging shims stagePbgo/unstage and build/gen/ssz_test.go.
  • genMethodical runs go tool ssz gen twice per target — once with default tags, once with --build-tags=minimal so methodical's package loader picks up the //go:build minimal .pb.go sources — and always writes the //go:build !minimal file plus its .minimal.ssz.go twin. This requires make gen proto to have run first so both .pb.go variants are on disk.
  • SSZ_PROGRESSIVE=1 generates the progressive form. The default is progressive off so hash tree roots match the current non-progressive spectest fixtures. This is inverted in PRs higher up the stack once we move to the alpha.12 spectests.
  • genProto now decides tagged-vs-untagged by byte-comparing the mainnet and minimal output instead of consulting a hardcoded typeDiffering map, and emitMinimalPbgo is deleted (the SSZ step no longer needs a temp-dir minimal build). Consequences visible in the diff: proto/engine/v1/execution_engine.pb.go and proto/prysm/v1alpha1/beacon_state.pb.go gain //go:build !minimal plus new .minimal.pb.go twins, and proto/eth/v1/events.pb.go is checked in for the first time.
  • cache.go: the SSZ manifest now includes each target's yaml config and no longer walks protoInc include directories.

Consumer-side migration (mechanical)

The fastssz Marshaler, Unmarshaler, HashRoot and Hasher types are replaced by their github.com/OffchainLabs/methodical-ssz/ssz equivalents across ~110 files — beacon-chain/p2p/encoder, beacon-chain/db/kv, beacon-chain/core/signing, consensus-types/{interfaces,primitives,blocks}, beacon-chain/state/state-native/custom-types, encoding/ssz/detect, tools/pcli, the spectest ssz_static shims, and so on.

A choice was made in methodical to avoid replicating fastssz's scalar helpers which are a thin wrapper around encoding/binary, so the following users of those funcs have been changed:

  • consensus-types/primitives.MarshalUint64 / UnmarshalUint64 are new (they live in primitives rather than encoding/ssz because encoding/ssz transitively depends on primitives), and encoding/ssz.UnmarshalUint64 is added for callers outside that cycle.
  • beacon-chain/slasher/params.go, beacon-chain/db/slasherkv/slasher.go and beacon-chain/rpc/eth/light-client/handlers.go now call binary.LittleEndian directly. The slasher kind byte is written as []byte{byte(kind)} in place of ssz.MarshalUint8.
  • beacon-chain/p2p/types defines a local ErrIncorrectByteSize, replacing fastssz's sentinel.

fastssz is not fully removed: encoding/ssz/query/*, beacon-chain/rpc/prysm/beacon/ssz_query.go and testing/middleware/builder still use it, and it remains in go.mod and in proto/prysm/v1alpha1's go_library deps.

Behavior and test-visible changes

These go beyond the mechanical swap and deserve attention:

  • Gloas block body field roots. consensus-types/blocks/proofs.go gains a version.Gloas case in ComputeBlockBodyFieldRoots and a new computeGloasBlockBodyFieldRoots (BLS changes at index 9, signed execution payload bid at 10, payload attestations at 11, parent execution requests at 12). The Bellatrix/Capella/Deneb/Electra branches are now upper-bounded with && blockBody.version < version.Gloas. A new generic blockBodyListRoot helper routes through ssz.MerkleizeListSSZProgressive when the body is Gloas-or-later and features.Get().EnableProgressiveSSZ is set, otherwise MerkleizeListSSZ; kzg.go's topLevelRoots uses the same helper. Covered by the new TestComputeBlockBodyFieldRoots_Gloas_ProgressiveSSZGate.
  • Gloas spectests decode Gloas types. testing/spectest/shared/gloas/ssz_static's AttesterSlashing and IndexedAttestation cases now unmarshal AttesterSlashingGloas / IndexedAttestationGloas (previously the Electra variants). A new PROGRESSIVE_SSZ=1 env var flips features.EnableProgressiveSSZ for the run so the native-state custom HTR lines up with progressive fixtures; leave it unset for the current non-progressive fixture set.
  • SSZ error text changed. methodical's marshal/unmarshal errors do not name the offending field. consensus-types/blocks/partialdatacolumn_test.go expectations become generic ("list length is higher than max value", "bytes array does not have the correct length") with TODOs to restore the field names once methodical includes them. This is a user-visible error-message regression for SSZ failures, not only a test change.
  • Empty-vs-nil slices. methodical round-trips an empty list as a non-nil empty slice where fastssz produced nil, so reflect.DeepEqual on block wrappers no longer holds after a storage round-trip. New consensus-types/interfaces/testing.RequireBlocksEqual compares via require.DeepSSZEqual on the proto; beacon-chain/db/kv/kv_test.go drops its local requireBlocksEqual in favour of it and adds a hash-tree-root assertion on the retrieved block.
  • testing/spectest/shared/common/ssz_static: root comparison switches to [32]byte equality via bytesutil.ToBytes32, and the first subtest is renamed from fastssz to generated.
  • encoding/ssz/htrutils_fuzz_test.go drops FuzzForkRoot, which depended on fastssz's ErrSize sentinel to classify expected decode failures.
  • nogo_config.json excludes .*\.ssz\.go from the appendclipped and rangeint analyzers.
  • This branch also fixed some test flakes, unrelated to SSZ: beacon-chain/blockchain/head_test.go replaces an immediate require.Equal(t, 1, len(events)) with require.Eventually (flaky event-delivery race), and config/features/flags.go has a trailing-whitespace-only cleanup.

Notes for reviewers

  • The tool directive swaps fastssz/sszgen for methodical-ssz/cmd/ssz.
  • The two build paths disagree on the progressive default. //tools:disable_progressive_merkleization defaults to False, so Bazel does not pass --disable-progressive, while make gen passes it unless SSZ_PROGRESSIVE=1. The comment in build/gen/ssz.go says the Go default "mirrors the //tools:disable_progressive_merkleization default set in .bazelrc", but there is no .bazelrc change in this diff. This is inert while no type carries progressive annotations, but the two defaults should be reconciled before they matter. This gets resolved higher up the PR stack where the flags are flipped to default to progressive merkleization.
  • Buildifier formatting. A few BUILD.bazel edits are tab-indented rather than space-indented and will fail a formatting check: the fastssz dep and gloas_builder_api.proto entries in proto/prysm/v1alpha1/BUILD.bazel, and the methodical dep plus the proto/prysm/wrappers removal in beacon-chain/rpc/prysm/v1alpha1/validator/BUILD.bazel.
  • ssz_gen_spectest in tools/methodical.bzl has no callers yet — it is groundwork, and the .gitignore entry for testing/spectest/methodical is the only other trace of it.
  • Worth confirming the newly checked-in proto/eth/v1/events.pb.go and the two new .minimal.pb.go twins are intended outputs of the byte-comparison change rather than a manifest gap being papered over.

Acknowledgements

  • I have read CONTRIBUTING.md.
  • I have included a uniquely named changelog fragment file.
  • I have added a description with sufficient context for reviewers to understand this PR.
  • I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).

Stack created with GitHub Stacks CLIGive Feedback 💬

@kasey
kasey requested a review from prestonvanloon as a code owner August 5, 2026 20:39
@kasey
kasey force-pushed the methodical-codegen-switch branch 2 times, most recently from ef2aa1f to f77fb7b Compare August 5, 2026 22:00
@kasey
kasey force-pushed the methodical-codegen-driver branch from 730d29e to 346a8de Compare August 5, 2026 22:00
@kasey
kasey force-pushed the methodical-codegen-switch branch 3 times, most recently from 599d21c to 0150575 Compare August 6, 2026 05:49
@kasey
kasey force-pushed the methodical-codegen-driver branch from beba51a to 56e6c00 Compare August 6, 2026 13:27
@kasey
kasey force-pushed the methodical-codegen-switch branch from 0150575 to 1a01a0b Compare August 6, 2026 13:27
@kasey
kasey force-pushed the methodical-codegen-driver branch from 56e6c00 to a67a389 Compare August 6, 2026 17:25
@kasey
kasey force-pushed the methodical-codegen-switch branch 2 times, most recently from 369252e to 81c5678 Compare August 7, 2026 20:00
@kasey
kasey force-pushed the methodical-codegen-driver branch from a67a389 to e3b9bf1 Compare August 7, 2026 20:00
@kasey
kasey force-pushed the methodical-codegen-switch branch from 81c5678 to f464594 Compare August 11, 2026 14:29
@kasey
kasey force-pushed the methodical-codegen-driver branch from e3b9bf1 to 5c101d4 Compare August 11, 2026 14:29
@kasey
kasey force-pushed the methodical-codegen-switch branch from f464594 to e29dde3 Compare August 11, 2026 19:34
@kasey
kasey force-pushed the methodical-codegen-driver branch from 5c101d4 to 0440be1 Compare August 11, 2026 19:34
kasey added 7 commits August 12, 2026 15:31
Replaces the fastssz sszgen codegen path with methodical-ssz. Targets are
now declared in a per-package yaml config listing the types to generate,
rather than obj lists spread across BUILD.bazel, and each target emits a
mainnet/minimal pair of build-tagged files instead of collapsing to one
file when the two happen to match. Keeping both files unconditionally
means the generated file set does not shift as bounded sizes change.

The Bazel rule cannot use the go toolchain inside the sandbox, so
ssz_methodical feeds methodical a package inventory through the
genception GOPACKAGESDRIVER. The `make gen` path runs outside any sandbox
and resolves against the real module instead.

The consumer-side change is mechanical: the fastssz Marshaler,
Unmarshaler, HashRoot and Hasher types are replaced by their
methodical-ssz equivalents across ~110 files.

Adds //tools:disable_progressive_merkleization, which ssz_methodical
reads to pass --disable-progressive. No type carries progressive
annotations yet, so the flag is inert until they are added.
When the script copies files out of the bazel build sandbox it also
adds the same go build tags that `build/gen` would, so that protobuf
files generated via bazel don't come out looking different.
@kasey
kasey force-pushed the methodical-codegen-switch branch from e29dde3 to 87e3a93 Compare August 12, 2026 20:32
@kasey
kasey force-pushed the methodical-codegen-driver branch from 0440be1 to 7aba3e0 Compare August 12, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant