Move Gloas spec tests into their own packages

[kasey]

Move Gloas spec tests into their own packages

Part of the gloas-devnet-7 stacked series; based on methodical-codegen-switch, diff shown against it.

The mainnet and minimal spec test packages had accumulated a large flat set of gloas__* files sharing a single go_test target with every other fork. Because Bazel test targets are the unit of granularity, running any gloas spec test meant compiling and pulling in the fixtures for phase0 through fulu as well. This PR moves the gloas files into testing/spectest/mainnet/gloas and testing/spectest/minimal/gloas with their own targets, so gloas can be iterated on independently.

The second half of the PR is tooling for the situation this series is in: during devnet development the config values Prysm carries are ahead of the pinned consensus-spec-tests release, so fixtures often have to come from a locally built spec rather than a published tarball. Now spectest fixtures are merged upstream, but having the option to run from a local tarball instead of always trying to download from the http archive seemed like a useful capability to preserve in case we need to use fixtures like this again, or have a reason to generate our own fixtures locally.
tools/download_spectests.bzl gains two opt-in ways to source spec test data from disk. tools/ssztrace is added to make root mismatches against a fixture debuggable — it dumps the merkleization of an SSZ object so a disagreement can be localized to the field that diverged. A corresponding python tool can be found at kasey/ssz-tracer to generate merkleization traces from remerkleable, giving a developer or LLM assistant the baseline canonical trace to diff.

Key changes

Spec test package split

• New testing/spectest/mainnet/gloas/BUILD.bazel and testing/spectest/minimal/gloas/BUILD.bazel, each with a go_default_test target carrying that preset's 42 gloas test files, data = ["@consensus_spec_tests//:test_data"], and only the //testing/spectest/shared/gloas/... dependencies. The minimal target keeps eth_network = "minimal" and the minimal tag; both keep the spectest tag, so existing --test_tag_filters=spectest invocations still pick them up.
• The corresponding srcs and deps entries are removed from testing/spectest/mainnet/BUILD.bazel and testing/spectest/minimal/BUILD.bazel.
• The moved *_test.go files change only their package clause (package mainnet / package minimal → package gloas). Test function names, helper calls, and preset arguments are untouched.

Local spec test data (tools/download_spectests.bzl)

Two new mechanisms, both taking precedence over CONSENSUS_SPEC_TESTS_VERSION and requiring no network access. All paths must be absolute; a source may be a tarball or an already-unpacked directory tree.

• CONSENSUS_SPEC_TESTS_DIR=/abs/dir — for each flavor (general, mainnet, minimal), _local_sources looks for <dir>/<flavor>.tar.gz, then <dir>/tests/<flavor>/, then <dir>/<flavor>/, in that order. Flavors that are absent are skipped, so supplying just mainnet is valid.
• CONSENSUS_SPEC_TESTS_<FLAVOR>=/abs/path (GENERAL, MAINNET, MINIMAL) — points one flavor at an arbitrarily named tarball or directly at an unpacked flavor tree, and wins over the directory entry for that flavor.
• _install_local normalizes each source to the tests/<flavor> layout the generated BUILD.bazel globs expect. Directories are symlinked (no large copy); tarballs are symlinked in as <flavor>.tar.gz, extracted to a staging dir, and then the detected internal prefix (tests/<flavor> or <flavor>) is symlinked into place.
• Requesting local sources but finding nothing is a hard fail rather than a silent fall back to downloading.
• All four variables are added to the repository rule's environ list.

Raw tarballs kept alongside extracted fixtures

• The release path now calls repository_ctx.download followed by repository_ctx.extract instead of download_and_extract, keeping <flavor>.tar.gz in the repo. Still one download, still integrity-checked and repository-cache-eligible.
• The generated BUILD.bazel gains exports_files(glob(["*.tar.gz"])) so rules that want the un-extracted archive can depend on @consensus_spec_tests//:<flavor>.tar.gz.

methodical-ssz spec test generation

• testing/spectest/methodical-spectest.yaml — the type relations config for methodical-ssz's spec test generator: each fork lists only the types it introduces or overrides (the generator inherits the rest from the most recent prior fork), with type_name for Prysm's fork-suffixed Go types and package: for the proto/engine/v1 execution-layer containers. Covers phase0 through gloas, including the gloas additions (ExecutionPayloadBid, PayloadAttestation*, Builder*, ExecutionPayloadEnvelope, ProposerPreferences, and the gloas variants of BeaconBlock/BeaconState/DataColumnSidecar).
• testing/spectest/BUILD.bazel — an ssz_gen_spectest target (the rule already exists in tools/methodical.bzl from earlier in the series) that generates into testing/spectest/methodical, using @consensus_spec_tests//:mainnet.tar.gz as its fixture source. This is what the exports_files change above is for.
• hack/update-go-spectest.sh — the generate-and-commit script, mirroring hack/update-go-ssz.sh. It builds every ssz_gen_spectest target, finds each generated package by its methodical_test.go marker, and mirrors the tree-artifact directory (test file plus testdata/) back into the source tree. Destinations are cleared first, since the tree artifact has no stable file list and stale fixtures would otherwise linger; copies are chmod u+w because Bazel outputs are read-only.

No generated output is committed in this PR — only the config, the target, and the script.

tools/ssztrace

A new go_binary that emits a merkleization trace for a gloas SSZ fixture, either as a flat generalized-index-to-root map or nested YAML, using Prysm's methodical-ssz-generated HashTreeRootWith and the tracer in github.com/OffchainLabs/methodical-ssz/ssz.

go run ./tools/ssztrace [-yaml] [-o out.yaml] <TypeName> <case_dir>

• newObject in tools/ssztrace/main.go maps spec test folder names to Prysm types, mirroring testing/spectest/shared/gloas/ssz_static/ssz_static.go; only types with a generated methodical HTR are registered, and an unknown name is an error naming the function to extend.
• <case_dir> is read for serialized.ssz_snappy and, if present, roots.yaml. The trace self-verifies against the value's generated HashTreeRoot internally; the fixture root is then cross-checked as a spec-version-skew signal only. A mismatch there prints a warning and still emits, because the trace is faithful to whatever the Go type computes.
• Intended for cross-library bisection: diff the flat map against the Python/remerkleable tracer's output and take the highest generalized index whose root differs.

Behavior and configuration notes

• Test target paths change. Gloas spec tests no longer run as part of //testing/spectest/mainnet:go_default_test or //testing/spectest/minimal:go_default_test; they are //testing/spectest/mainnet/gloas:go_default_test and //testing/spectest/minimal/gloas:go_default_test. Anything that names those preset targets explicitly needs updating; tag-filtered and //... runs are unaffected.
• Downloading is still the default. The local-source env vars are opt-in, and CI behavior is unchanged when none of them are set.
• @consensus_spec_tests//:<flavor>.tar.gz only exists when the source is an actual tarball, not an unpacked directory, so the ssz_gen_spectest target cannot be built against a directory-sourced repo.
• When only some flavors are supplied, scope the test target (e.g. //testing/spectest/mainnet/...) instead of //..., since the other flavors' filegroups resolve to empty globs.
• Bazel keys the repository rule on the env var value, not file contents. Changing data at the same path requires bazel sync --configure or a different path to force a re-fetch.
• testing/spectest/README.md documents all of the above; tools/download_spectests.bzl also picked up buildifier formatting churn in the pre-existing download paths.

Acknowledgements

• I have read CONTRIBUTING.md.
• I have included a uniquely named changelog fragment file.
• I have added a description with sufficient context for reviewers to understand this PR.
• I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[terencechain]

I think we missed a few fields in testing/spectest/methodical-spectest.yaml

[terencechain]

are we missing ExecutionRequests → ExecutionRequestsGloas?
and how about the following?

BuilderDepositRequest
BuilderExitRequest
AttestationGloas
AttesterSlashingGloas
IndexedAttestationGloas
AggregateAttestationAndProofGloas,
SignedAggregateAttestationAndProofGloas

[nalepae]

Nit: update-go-ssz.sh does not exist any more.

[nalepae]

Where can this script be found?

[nalepae]

Before this PR, make test alreay had no equivalent to

bazel test //... --test_tag_filters=spectest --repo_env=CONSENSUS_SPEC_TESTS_VERSION=nightly

You can run the spectests only by using

make test mainnet-spectest minimal-spectest

but it's not possible to specify the equivalent of the --repo_env=CONSENSUS_SPEC_TESTS_VERSION value.

After this PR, this thing remains, and additionnaly, it's not possible to specify the equivalent of the --repo_env=CONSENSUS_SPEC_TESTS_DIR value.

These 2 things can be added in this current PR, or I can create a GH issue to track it, to be sure we implement this before totally remove Bazel.