Skip to content

Pin ko to v0.18.1 in CI to fix kodata symlink failures - #2002

Merged
malancas merged 2 commits into
sigstore:mainfrom
nellshamrell:pin-ko-version-fix-e2e
Jul 30, 2026
Merged

Pin ko to v0.18.1 in CI to fix kodata symlink failures#2002
malancas merged 2 commits into
sigstore:mainfrom
nellshamrell:pin-ko-version-fix-e2e

Conversation

@nellshamrell

Copy link
Copy Markdown
Contributor

Summary

Every e2e and build workflow currently fails at the Install policy-controller step (make ko-policy-controllerko resolve) with:

tarring kodata: kodata symlink ".../cmd/webhook/kodata/HEAD" resolves to
".../.git/HEAD" which is outside the kodata root ".../cmd/webhook/kodata"

This affects all branches and PRs (dependabot, and open PRs like #1997 and #2000), not any individual change.

Root cause

The workflows install ko via ko-build/setup-ko without a version: input, so they use setup-ko's default latest-release. ko v0.19.0 (ko-build/ko#1619) added a strict check that rejects kodata symlinks whose targets resolve outside the kodata directory. cmd/webhook/kodata/ contains such symlinks (present since 2022):

  • HEAD../../../.git/HEAD
  • refs../../../.git/refs
  • LICENSE../../../LICENSE
  • VENDOR-LICENSE../../../third_party/VENDOR-LICENSE

So when CI silently moved to ko ≥ v0.19.0, ko resolve began failing repo-wide.

The upstream relaxation ko-build/ko#1699 ("allow kodata symlinks within the source tree") is merged but not yet released (latest release is v0.19.1; the fix landed after it).

Fix

Pin setup-ko to ko v0.18.1 (the last release before the strict check) across all workflows that install ko, with an explanatory comment. This is an explicit, temporary workaround: v0.19.0's check is a legitimate hardening (it blocks kodata symlinks from packing arbitrary host files into images), so the pin should be removed once a ko release includes #1699.

Validation

Reproduced locally with the real ko binaries against ./cmd/webhook:

  • ko v0.19.1 → fails at kodata tarring with the exact CI error.
  • ko v0.18.1 → packs kodata successfully (only fails later at publish, because no local kind cluster — unrelated to kodata).
  • Removing only the .git symlinks is insufficient: ko v0.19.x then rejects LICENSE next. Pinning is the minimal correct fix.

Notes

  • This unblocks e2e for all currently-open PRs. e2e may still surface unrelated failures further along, but it will get past ko resolve.
  • Stale # will use the latest release available for ko comments are removed, since ko is now pinned.
  • A follow-up to remove the pin once ko#1699 ships would be appropriate.

All e2e and build workflows install ko via ko-build/setup-ko without a
version input, so they use setup-ko's default `latest-release`. ko v0.19.0
(ko-build/ko#1619) added a strict check that rejects kodata symlinks whose
targets resolve outside the kodata directory. cmd/webhook/kodata contains
such symlinks (HEAD/refs -> .git, LICENSE -> ../../../LICENSE,
VENDOR-LICENSE -> ../../../third_party/VENDOR-LICENSE), so every
`make ko-policy-controller` (ko resolve) now fails at the "Install
policy-controller" step across all branches and PRs:

    tarring kodata: kodata symlink ".../cmd/webhook/kodata/HEAD" resolves to
    ".../.git/HEAD" which is outside the kodata root

The upstream relaxation (ko-build/ko#1699, which widens the allowed root to
the enclosing source tree) is merged but not yet released. As a temporary
fix, pin setup-ko to ko v0.18.1 (the last release before the strict check)
so CI is unblocked. Once a ko release includes sigstore#1699, remove the pin.

Verified locally: ko v0.19.1 fails at kodata tarring exactly as CI does,
while ko v0.18.1 packs kodata successfully. Removing only the .git symlinks
is insufficient because ko v0.19.x then rejects LICENSE next, so pinning is
the minimal correct fix. Stale "will use the latest release available for
ko" comments are removed since ko is now pinned.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0ce53efa-a7e9-4046-a24e-34c406601950
Signed-off-by: Nell Shamrell <nellshamrell@gmail.com>
The TUF test fixtures across three packages shared a baked-in expiry of
2026-07-18T08:24:13Z, which has now passed. Any test run after that date
failed constructing a TUF client with "final root.json is expired",
reddening CI on main and every open PR.

Affected packages/tests:
- pkg/reconciler/trustroot: TestReconcile/With_repository and the
  trusted_root.json variant (embedded testdata/*).
- pkg/apis/policy/v1alpha1: TestTrustRootValidation (inline base64 blobs).
- pkg/tuf: TestDownloadTargetFromSerializedMirror and
  TestClientFromSerializedMirror (inline base64 blobs, byte-identical to
  the v1alpha1 ones).

Regenerate all three together (they must share the same trust anchor):
- Reconciler testdata via `make generate-testdata`.
- The inline validRepository/rootJSON blobs in pkg/tuf and
  pkg/apis/policy/v1alpha1 from the freshly generated tufRepo.tar and
  root.json.

Also refresh the stale "next expiration" comments and replace the manual
kind-cluster generation notes with a pointer to `make generate-testdata`.

Note: the scaffolding repo builder hardcodes a 6-month expiry, so these
fixtures will need periodic regeneration (next: 2027-01-28).

Fixes sigstore#1999

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0ce53efa-a7e9-4046-a24e-34c406601950
Signed-off-by: Nell Shamrell <nellshamrell@gmail.com>
@nellshamrell

Copy link
Copy Markdown
Contributor Author

Update: with the ko pin in place, the approved run showed the e2e suite passing again — the only remaining failures were Run unit tests and the TrustRoot "Bring Your Own Keys" e2e, both caused by the expired TUF test fixtures fixed in #2000. I've cherry-picked #2000's Regenerate expired TUF test fixtures commit onto this branch so #2002 is green on its own. Verified locally: go test ./pkg/tuf/... ./pkg/reconciler/trustroot/... ./pkg/apis/policy/v1alpha1/... all pass. (This does overlap with #2000; whichever merges second will simply no-op those fixture changes.)

@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 30.73%. Comparing base (675921f) to head (ead6e34).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2002   +/-   ##
=======================================
  Coverage   30.73%   30.73%           
=======================================
  Files         122      122           
  Lines        7409     7409           
=======================================
  Hits         2277     2277           
  Misses       4897     4897           
  Partials      235      235           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Hayden-IO
Hayden-IO requested a review from a team July 30, 2026 19:04
@malancas
malancas merged commit 5c56a09 into sigstore:main Jul 30, 2026
95 checks passed
@github-actions github-actions Bot added this to the v1 milestone Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants