ci: add spec-tests sync workflows

[vaclav-ssvlabs]

Summary

Adds GitHub Actions workflows to automatically sync generated spec test JSON files to the dedicated spec-tests repository.

• generate-spec-tests composite action: sets up Go, runs generators for ssv/qbft/types spectests, copies output to ../spec-tests
• sync-spec-tests-pr: on PR open/synchronize, pushes generated files to a matching branch in spec-tests repo and creates/updates a mirrored PR
• sync-spec-tests-merge: on push to main, regenerates files and merges the corresponding spec-tests PR
• test.yaml: bumped to ubuntu-24.04, node24-compatible action versions

Motivation

1. The specs are a dependency for ssv-node implementation. The jsons blow up the size of the dependency to an unacceptable level. Compilation errors may occur.
2. Small changes in spec make huge changes in jsons. This is problematic for modern tools and humans.

[greptile-apps]

Greptile Summary

This PR introduces GitHub Actions workflows to automatically sync generated spec-test JSON files from ssv-spec to a dedicated spec-tests repository, plus a composite action to consolidate Go setup and generation steps.

• generate-spec-tests composite action: Consolidates Go setup and go generate steps previously scattered in test.yaml — clean refactor.
• sync-spec-tests-pr: On PR open/synchronize, generates spec tests and pushes them to a matching branch in the spec-tests repo, creating or updating a mirrored PR. Has two P2 issues: force-pushing to the mirrored PR branch (violates the project's no-force-push-to-PR-branches policy) and using a bare #PR_NUMBER reference in the PR body that will auto-link to the wrong repo.
• sync-spec-tests-merge: On push to main, regenerates files and merges the corresponding spec-tests PR. Force-push is also present here. The approach of resolving the original PR via commits/{sha}/pulls is sound; the timing caveat is correctly acknowledged.
• test.yaml: Pinned to ubuntu-24.04 and node24-compatible action versions — straightforward improvement.
• Both sync workflows validate SPEC_TESTS_REPO only after the expensive generation step; adding an early check would avoid wasted CI minutes when the variable is misconfigured.

Confidence Score: 4/5

• Safe to merge after addressing the cross-repo PR link bug and acknowledging the force-push trade-off for auto-generated content.
• The core automation logic is correct and well-structured. The two remaining items are a broken cross-repo reference in the PR body (easy one-line fix) and the force-push pattern which, while flagged by the custom rule, may be intentional for auto-generated content. Neither blocks the primary sync flow.
• sync-spec-tests-pr.yaml — contains both the cross-repo reference issue and the force-push pattern.

Important Files Changed

Filename	Overview
.github/actions/generate-spec-tests/action.yaml	New composite action that sets up Go 1.22, runs go generate for ssv/qbft/types spectests, and copies output to ../spec-tests. Logic is sound; the find+copy pattern safely handles multiple spectest directories.
.github/workflows/sync-spec-tests-pr.yaml	On PR open/synchronize, generates spec tests and syncs to a matching branch in spec-tests repo. Two issues: (1) force-push to PR branch overwrites review history per custom rule; (2) cross-repo #PR_NUMBER reference in the PR body will link to the wrong repo.
.github/workflows/sync-spec-tests-merge.yaml	On push to main, finds the merged PR by commit SHA, pushes final generated files to the spec-tests branch, then merges the mirrored PR. Force-push is present here too. The API timing caveat is acknowledged in the error message.
.github/workflows/test.yaml	Refactored to use the new composite action, pinned to ubuntu-24.04 and node24-compatible action versions. Clean and straightforward improvement.

Sequence Diagram

sequenceDiagram
    participant Dev as Developer
    participant SSV as ssv-spec (GitHub)
    participant GHA as GitHub Actions
    participant ST as spec-tests repo

    Dev->>SSV: Open / push to PR branch
    SSV->>GHA: Trigger sync-spec-tests-pr
    GHA->>GHA: checkout + go generate (ssv/qbft/types)
    GHA->>GHA: Get GitHub App token
    GHA->>ST: git clone spec-tests
    GHA->>ST: git checkout -B {branch} origin/{base}
    GHA->>ST: cp generated files + git commit
    GHA->>ST: git push --force origin {branch}
    GHA->>ST: gh pr create / gh pr edit

    Dev->>SSV: Merge PR to main
    SSV->>GHA: Trigger sync-spec-tests-merge
    GHA->>GHA: checkout + go generate (ssv/qbft/types)
    GHA->>SSV: gh api commits/{SHA}/pulls → PR number + head_ref
    GHA->>GHA: Get GitHub App token
    GHA->>ST: git clone spec-tests
    GHA->>ST: git checkout -B {branch} origin/{branch}
    GHA->>ST: cp final generated files + git commit
    GHA->>ST: git push --force origin {branch}
    GHA->>ST: gh pr merge {spec-tests PR} --merge --delete-branch

Loading

_{Reviews (1): Last reviewed commit: "ci: add spec-tests sync workflows" | Re-trigger Greptile}

[GalRogozinski]

Thanks @vaclav-ssvlabs
greptile says:

using a bare #PR_NUMBER reference in the PR body that will auto-link to the wrong repo.

Codex:

1. sync-spec-tests-pr.yaml can succeed without creating anything, but sync-spec-tests-merge.yaml later assumes the branch/PR exists and will fail.

At sync-spec-tests-pr.yaml:108, the PR sync workflow exits early on git diff --cached --quiet with "No changes to sync" and never pushes the branch or opens the PR. But the merge workflow later hard-requires both to exist at sync-spec-tests-merge.yaml:109 and sync-spec-tests-merge.yaml:131. That means an ssv-spec PR that produces no fixture diff against its base branch can still cause the post-merge main workflow to fail. Either the PR workflow needs to create the mirror branch/PR even for the no-op case, or the merge workflow needs to tolerate them being absent.

Probably non issue but still pasting:
2. Fork PRs will fail because this workflow uses a private-key secret on pull_request.

At sync-spec-tests-pr.yaml:34, the workflow unconditionally calls actions/create-github-app-token using SPEC_TESTS_APP_PRIVATE_KEY, but GitHub does not expose repository secrets to fork-originated pull_request runs. If external contributors are expected to open PRs, this job will fail immediately unless it explicitly skips forks or uses a different event model.

[GalRogozinski]

Hey @vaclav-ssvlabs after the known issues, there may be more.

I recommend the following:

1. On a ci-stage branch we should merge this and Move jsons to spec-tests repo #602. This way we have the complete feature and we don't need the 3 reviews.
2. For tests, We need to tweak the environment so that the automation will work w.o main being merged. After we are confident we can collect the 3 reviews and merge to main.

wdyt?

I also opened #619 to fix the no changes edge case. Codex claims the greptile bug is a non-issue, but we should verify.
After you review my PR DM me and let's see if we can properly test this.

[momosh-ssv]

Thanks for putting this together — moving the JSONs to a dedicated repo is a big quality-of-life win. Spent some time walking through the workflows and the path-resolution helpers to make sure I understood the trust and failure-mode story. A few questions and suggestions below — nothing blocking, mostly things we'd love to talk through.

[vaclav-ssvlabs]

I reacted to all comments regarding workflow, IMHO, the majority of them as very minor/low-risk things. There are a few of them that are worth pursuing, but since this is my last day, I don't have time to follow up and test all the things. So I will leave the solutions for you guys. Good luck!

[GalRogozinski]

Picked up the outstanding review threads from @momosh-ssv now that @vaclav-ssvlabs has moved on. Addressed in ada2cb7:

Thread	Action
App token shares a runner with go generate	Fixed — sync-spec-tests-pr is now two jobs: generate (checkout + go generate + upload tarball, no secrets) and sync (download + push with App token, no ssv-spec checkout).
HEAD_REF ref-shape validation	Fixed — strict regex guard before it reaches git checkout -B / git push / gh pr create --head.
gh pr view <branch> can return a closed PR	Fixed — gh pr list --head ... --state open in both pr.yaml and merge.yaml.
PR_TITLE must stay in env:	Fixed — invariant documented above the env: block.
Token left in .git/config, PR-number branch namespacing, empty GENERATED_PATHS, no-PR-on-push-to-main, base-ref drift	Dropped by the reviewer as low risk — no change.
go.mod-anchored generator paths	Already fixed in cc74c02 and confirmed by the reviewer.

Also swapped the sync commit message from #${PR_NUMBER} to the full PR URL, same auto-link issue Greptile flagged for the PR body.

Note the split adds an artifact round-trip (one tarball, since tens of thousands of small JSONs would be far slower uploaded individually) plus one extra runner start per PR. Worth it to keep the token off the same runner as PR-controlled code.

Verified end to end on ada2cb75d: both jobs green, the artifact round-tripped, and the sync job cloned spec-tests, force-pushed spec-test-ci, and the new gh pr list --state open lookup correctly found and updated the existing ssvlabs/spec-tests#5. (Correcting myself — an earlier version of this comment said the repo variables weren't configured yet. They are; the workflow has been running for real.)

---

Unrelated defect the verified run exposed, not raised by any reviewer, not fixed here since it changes what this PR ships:

The spec-tests output currently contains every generated file twice, under two different path shapes:

types/tests/....json                          <- written directly by the anchored generator
types/spectest/generate/tests/....json        <- copied from the repo by the composite action

Cause: the merge from main in 5354bf5 resurrected 269 JSON files under ssv/spectest/generate/{tests,state_comparison} and types/spectest/generate/{tests,state_comparison} that 860caf4e2 had deleted (main had regenerated them via #601). The composite action's "Copy generated files to spec-tests output directory" step then rakes those stale files in and mirrors them under the old path.

Since cc74c02, every generator writes straight to ../spec-tests/<module>/... via SpecTestsDirForModule, and every reader resolves through SpecTestsDirFrom — I grepped, nothing reads the in-repo */spectest/generate/{tests,state_comparison} paths any more. So the fix is to delete those 269 dead files and drop the now-redundant copy step from the composite action. Worth confirming main won't just resurrect them again on the next merge.

[MatheusFranco99]

Reviewed with a focus on correctness and the operational surface this adds. The direction is right and the security reasoning in sync-spec-tests-pr.yaml is genuinely above average for CI work — the generate/sync job split, untrusted values passed via env: rather than spliced into shell, the branch-name shape check, the --state open scoping. Those comments explaining why are valuable; please keep them.

Two themes in the findings:

1. sync-spec-tests-merge.yaml didn't get the same hardening as the PR workflow. The missing branch-name validation there is reachable and can force-push spec-tests main, bypassing the mirror PR. The root cause is that ~60 lines of sync logic are copy-pasted between the two files and have drifted.
2. Identity of the mirror branch. Keying it on head.ref makes it non-unique across forks and lets it take a protected name; keying it on the PR number removes both problems and most of the validation burden.

Requesting changes on the first of those. Everything else is either a small fix or a question.

Blocking:

• force-push-to-main path in sync-spec-tests-merge.yaml
• extract the shared sync script (root cause of the above)
• branch naming on head.ref

Non-blocking but worth doing now: concurrency groups on both workflows, SHA-pinning the actions in the token-holding jobs, the make lint / CI version divergence, and the local-development docs.

Open question rather than a finding: whether fork PRs are in scope at all — see my comment on the sync job. As written they fail on missing secrets, and that failure is what makes the merge-time path above dangerous.

One nit on the PR description itself: it says the composite action "copies output to ../spec-tests", but the action's own comment correctly notes the generators write there directly and there's nothing to copy. Worth updating so the description doesn't send the next reader looking for a copy step.

Not raised as findings, for the record: the os.RemoveAll on a path outside the repo, the absence of an override for the hardcoded ../spec-tests, and assorted style nits — discussed and deliberately deferred.

[iurii-ssv]

Couple minor things.

[iurii-ssv]

LGTM

[MatheusFranco99]

Approving — re-reviewed at b6380e0.

The blocking finding is closed: `25abb89c8` applies the same shape check and `main`/`master` refusal to `PR_HEAD_REF` before it reaches any `git`/`gh` call, and `750163aec` makes the fork check fail closed, so the force-push-to-`spec-tests`-`main` path is unreachable rather than just narrowed. The two design calls I raised alongside it — the shared sync script and PR-number branch naming — I accept as declined: the divergence that motivated the extraction was the missing validation, and the fork skip plus ssv-spec's one-branch-per-name constraint keeps the collision class out of reach without a policy change. Everything non-blocking landed too: PR-workflow `concurrency`, SHA-pinned actions, `make lint` pinned to v2.12.2 with a working installer, the local-development section, and the duplicate fixture tree dropped in `181b69ca4`.

Resolved all my threads. One item stays open by choice, recorded here rather than as a blocker: `sync-spec-tests-merge.yaml` has no serialization, and since every sync replaces the whole generated tree, the merge order of two closely-spaced ssv-spec merges decides what `spec-tests` `main` ends up holding — the runs do share state through `gh pr merge`, even though they clone into separate tmpdirs and push separate branches. The likely outcome is a loud `gh pr merge` failure on the second mirror PR (its merge base predates the first), which needs a human to unstick. Your per-commit group prevents duplicate runs of the same commit rather than interleaving of different ones; a static group with `cancel-in-progress: false` is what actually serializes, at the cost of the middle-run eviction you described. Happy for that to be a follow-up.