methodical codegen switch - #17308
Open
kasey wants to merge 7 commits into
Open
Conversation
kasey
force-pushed
the
methodical-codegen-switch
branch
2 times, most recently
from
August 5, 2026 22:00
ef2aa1f to
f77fb7b
Compare
kasey
force-pushed
the
methodical-codegen-driver
branch
from
August 5, 2026 22:00
730d29e to
346a8de
Compare
kasey
force-pushed
the
methodical-codegen-switch
branch
3 times, most recently
from
August 6, 2026 05:49
599d21c to
0150575
Compare
kasey
force-pushed
the
methodical-codegen-driver
branch
from
August 6, 2026 13:27
beba51a to
56e6c00
Compare
kasey
force-pushed
the
methodical-codegen-switch
branch
from
August 6, 2026 13:27
0150575 to
1a01a0b
Compare
kasey
force-pushed
the
methodical-codegen-driver
branch
from
August 6, 2026 17:25
56e6c00 to
a67a389
Compare
kasey
force-pushed
the
methodical-codegen-switch
branch
2 times, most recently
from
August 7, 2026 20:00
369252e to
81c5678
Compare
kasey
force-pushed
the
methodical-codegen-driver
branch
from
August 7, 2026 20:00
a67a389 to
e3b9bf1
Compare
kasey
force-pushed
the
methodical-codegen-switch
branch
from
August 11, 2026 14:29
81c5678 to
f464594
Compare
kasey
force-pushed
the
methodical-codegen-driver
branch
from
August 11, 2026 14:29
e3b9bf1 to
5c101d4
Compare
kasey
force-pushed
the
methodical-codegen-switch
branch
from
August 11, 2026 19:34
f464594 to
e29dde3
Compare
kasey
force-pushed
the
methodical-codegen-driver
branch
from
August 11, 2026 19:34
5c101d4 to
0440be1
Compare
Replaces the fastssz sszgen codegen path with methodical-ssz. Targets are now declared in a per-package yaml config listing the types to generate, rather than obj lists spread across BUILD.bazel, and each target emits a mainnet/minimal pair of build-tagged files instead of collapsing to one file when the two happen to match. Keeping both files unconditionally means the generated file set does not shift as bounded sizes change. The Bazel rule cannot use the go toolchain inside the sandbox, so ssz_methodical feeds methodical a package inventory through the genception GOPACKAGESDRIVER. The `make gen` path runs outside any sandbox and resolves against the real module instead. The consumer-side change is mechanical: the fastssz Marshaler, Unmarshaler, HashRoot and Hasher types are replaced by their methodical-ssz equivalents across ~110 files. Adds //tools:disable_progressive_merkleization, which ssz_methodical reads to pass --disable-progressive. No type carries progressive annotations yet, so the flag is inert until they are added.
When the script copies files out of the bazel build sandbox it also adds the same go build tags that `build/gen` would, so that protobuf files generated via bazel don't come out looking different.
kasey
force-pushed
the
methodical-codegen-switch
branch
from
August 12, 2026 20:32
e29dde3 to
87e3a93
Compare
kasey
force-pushed
the
methodical-codegen-driver
branch
from
August 12, 2026 20:32
0440be1 to
7aba3e0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generate SSZ with methodical-ssz instead of fastssz
This replaces the fastssz
sszgencodegen path with methodical-ssz. Generation targets are now declared in a per-package yaml config that lists the types to generate, rather than asobjs/exclude_objslists spread acrossBUILD.bazel—proto/prysm/v1alpha1/BUILD.bazelalone loses roughly 300 lines of hand-maintained, order-dependent type lists (each fork's target had to repeat every earlier fork's list inexclude_objs).Each target now emits a mainnet/minimal pair of build-tagged files unconditionally, instead of collapsing to a single untagged file when the two happen to be produce the same output (previous behavior/assumptions of
build/gen). Keeping both files means the generated file set does not shift as builds are performed for different presets — under the old scheme a.minimal.ssz.gotwin would appear or vanish depending on whether the generated code for the two different presets happened to coincide.The change also lays the groundwork for progressive merkleization: a new
//tools:disable_progressive_merkleizationbuild flag is threaded through every codegen action. No type carries progressive annotations yet, so the flag is a no-op until they are added.Scale and reading guidance
188 files, ~67k insertions — but 32 of those files are generated (~66k of the insertions). Skip every
*.ssz.go,*.minimal.ssz.goand*.pb.gofile. The reviewable surface is about 1,400 lines across:tools/methodical.bzl(new, 199 lines) — the Bazel rulesbuild/gen/— themake genpathproto/**/*.yamltarget configsBUILD.bazelfiles (mostly dep swaps)Key changes
Bazel codegen rules
tools/ssz.bzl(thessz_gen_marshalrule andSSZ_DEPS) is deleted;tools/methodical.bzlreplaces it. NoSSZ_DEPSreferences remain in the tree.ssz_methodicalcannot use the go toolchain inside the Bazel sandbox, so it feeds methodical a pre-built package inventory through the genceptionGOPACKAGESDRIVER. The rule walksgo_pkg_info_aspectoutputs overdepsplus the injectedssz_lib, writes the collected package JSON paths (plus the stdlib entry) tomethodical-pkg-list.json, and runs the tool withPACKAGE_JSON_INVENTORY/GOPACKAGESDRIVER/PACKAGES_BASEset.tools/genception/BUILD.bazelaliasesmethodicalgento@com_github_offchainlabs_methodical_ssz//cmd/ssz.ssz_gen_spectestis also added: it interpolates names from a yaml config and copies fixtures out of a consensus-spec-tests release tarball, needing none of the package-driver apparatus. It is defined but not yet wired to any target;.gitignoregainstesting/spectest/methodicalfor its (large) generated output.//tools:disable_progressive_merkleizationis a newbool_flagintools/BUILD.bazel(defaultFalse).ssz_methodicalreads it viaBuildSettingInfoand appends--disable-progressive, sobazel build --//tools:disable_progressive_merkleization //...flips every target with no per-rule edits.Declarative target configs
proto/prysm/v1alpha1/{phase0,altair,bellatrix,capella,deneb,electra,fulu,gloas,non-core}.yaml,proto/engine/v1/engine.yaml,proto/eth/v1/gateway.yaml,proto/ssz_query/response.yaml,proto/ssz_query/testing/test_containers.yaml.ssz_methodicalrules takeconfig_file,out,depsand an optionaloverride_package_name(e.g.ethforproto/prysm/v1alpha1,v1forproto/eth/v1).gloas_builder_apitarget and itsgloas_builder_api.ssz.gooutput are gone;RequestAuthV1,SignedRequestAuthV1,BuilderPreferencesV1andBuilderPreferencesRequestV1are folded intogloas.yaml.proto/eth/v1/gateway.yamlpicks upBeaconBlockBody(not previously generated) and uses the generatedDeposit_Dataname.make genpath (build/gen/)loadSSZTargets→loadMethodicalTargets, now readingssz_methodicalrules and returningmethodicalTarget{pkg, configFile, out, overridePkg}. The Starlark string-list evaluator (evalStringList,attrStringList,splitIncludes) is deleted along with the.pb.gostaging shimsstagePbgo/unstageandbuild/gen/ssz_test.go.genMethodicalrunsgo tool ssz gentwice per target — once with default tags, once with--build-tags=minimalso methodical's package loader picks up the//go:build minimal.pb.gosources — and always writes the//go:build !minimalfile plus its.minimal.ssz.gotwin. This requiresmake gen prototo have run first so both.pb.govariants are on disk.SSZ_PROGRESSIVE=1generates the progressive form. The default is progressive off so hash tree roots match the current non-progressive spectest fixtures. This is inverted in PRs higher up the stack once we move to the alpha.12 spectests.genProtonow decides tagged-vs-untagged by byte-comparing the mainnet and minimal output instead of consulting a hardcodedtypeDifferingmap, andemitMinimalPbgois deleted (the SSZ step no longer needs a temp-dir minimal build). Consequences visible in the diff:proto/engine/v1/execution_engine.pb.goandproto/prysm/v1alpha1/beacon_state.pb.gogain//go:build !minimalplus new.minimal.pb.gotwins, andproto/eth/v1/events.pb.gois checked in for the first time.cache.go: the SSZ manifest now includes each target's yaml config and no longer walksprotoIncinclude directories.Consumer-side migration (mechanical)
The fastssz
Marshaler,Unmarshaler,HashRootandHashertypes are replaced by theirgithub.com/OffchainLabs/methodical-ssz/sszequivalents across ~110 files —beacon-chain/p2p/encoder,beacon-chain/db/kv,beacon-chain/core/signing,consensus-types/{interfaces,primitives,blocks},beacon-chain/state/state-native/custom-types,encoding/ssz/detect,tools/pcli, the spectestssz_staticshims, and so on.A choice was made in methodical to avoid replicating fastssz's scalar helpers which are a thin wrapper around
encoding/binary, so the following users of those funcs have been changed:consensus-types/primitives.MarshalUint64/UnmarshalUint64are new (they live inprimitivesrather thanencoding/sszbecauseencoding/ssztransitively depends onprimitives), andencoding/ssz.UnmarshalUint64is added for callers outside that cycle.beacon-chain/slasher/params.go,beacon-chain/db/slasherkv/slasher.goandbeacon-chain/rpc/eth/light-client/handlers.gonow callbinary.LittleEndiandirectly. The slasherkindbyte is written as[]byte{byte(kind)}in place ofssz.MarshalUint8.beacon-chain/p2p/typesdefines a localErrIncorrectByteSize, replacing fastssz's sentinel.fastssz is not fully removed:
encoding/ssz/query/*,beacon-chain/rpc/prysm/beacon/ssz_query.goandtesting/middleware/builderstill use it, and it remains ingo.modand inproto/prysm/v1alpha1'sgo_librarydeps.Behavior and test-visible changes
These go beyond the mechanical swap and deserve attention:
consensus-types/blocks/proofs.gogains aversion.Gloascase inComputeBlockBodyFieldRootsand a newcomputeGloasBlockBodyFieldRoots(BLS changes at index 9, signed execution payload bid at 10, payload attestations at 11, parent execution requests at 12). The Bellatrix/Capella/Deneb/Electra branches are now upper-bounded with&& blockBody.version < version.Gloas. A new genericblockBodyListRoothelper routes throughssz.MerkleizeListSSZProgressivewhen the body is Gloas-or-later andfeatures.Get().EnableProgressiveSSZis set, otherwiseMerkleizeListSSZ;kzg.go'stopLevelRootsuses the same helper. Covered by the newTestComputeBlockBodyFieldRoots_Gloas_ProgressiveSSZGate.testing/spectest/shared/gloas/ssz_static'sAttesterSlashingandIndexedAttestationcases now unmarshalAttesterSlashingGloas/IndexedAttestationGloas(previously the Electra variants). A newPROGRESSIVE_SSZ=1env var flipsfeatures.EnableProgressiveSSZfor the run so the native-statecustomHTR lines up with progressive fixtures; leave it unset for the current non-progressive fixture set.consensus-types/blocks/partialdatacolumn_test.goexpectations become generic ("list length is higher than max value","bytes array does not have the correct length") withTODOs to restore the field names once methodical includes them. This is a user-visible error-message regression for SSZ failures, not only a test change.nil, soreflect.DeepEqualon block wrappers no longer holds after a storage round-trip. Newconsensus-types/interfaces/testing.RequireBlocksEqualcompares viarequire.DeepSSZEqualon the proto;beacon-chain/db/kv/kv_test.godrops its localrequireBlocksEqualin favour of it and adds a hash-tree-root assertion on the retrieved block.testing/spectest/shared/common/ssz_static: root comparison switches to[32]byteequality viabytesutil.ToBytes32, and the first subtest is renamed fromfastssztogenerated.encoding/ssz/htrutils_fuzz_test.godropsFuzzForkRoot, which depended on fastssz'sErrSizesentinel to classify expected decode failures.nogo_config.jsonexcludes.*\.ssz\.gofrom theappendclippedandrangeintanalyzers.beacon-chain/blockchain/head_test.goreplaces an immediaterequire.Equal(t, 1, len(events))withrequire.Eventually(flaky event-delivery race), andconfig/features/flags.gohas a trailing-whitespace-only cleanup.Notes for reviewers
tooldirective swapsfastssz/sszgenformethodical-ssz/cmd/ssz.//tools:disable_progressive_merkleizationdefaults toFalse, so Bazel does not pass--disable-progressive, whilemake genpasses it unlessSSZ_PROGRESSIVE=1. The comment inbuild/gen/ssz.gosays the Go default "mirrors the//tools:disable_progressive_merkleizationdefault set in.bazelrc", but there is no.bazelrcchange in this diff. This is inert while no type carries progressive annotations, but the two defaults should be reconciled before they matter. This gets resolved higher up the PR stack where the flags are flipped to default to progressive merkleization.BUILD.bazeledits are tab-indented rather than space-indented and will fail a formatting check: thefastsszdep andgloas_builder_api.protoentries inproto/prysm/v1alpha1/BUILD.bazel, and the methodical dep plus theproto/prysm/wrappersremoval inbeacon-chain/rpc/prysm/v1alpha1/validator/BUILD.bazel.ssz_gen_spectestintools/methodical.bzlhas no callers yet — it is groundwork, and the.gitignoreentry fortesting/spectest/methodicalis the only other trace of it.proto/eth/v1/events.pb.goand the two new.minimal.pb.gotwins are intended outputs of the byte-comparison change rather than a manifest gap being papered over.Acknowledgements
Stack created with GitHub Stacks CLI • Give Feedback 💬