Pin ko to v0.18.1 in CI to fix kodata symlink failures - #2002
Conversation
All e2e and build workflows install ko via ko-build/setup-ko without a version input, so they use setup-ko's default `latest-release`. ko v0.19.0 (ko-build/ko#1619) added a strict check that rejects kodata symlinks whose targets resolve outside the kodata directory. cmd/webhook/kodata contains such symlinks (HEAD/refs -> .git, LICENSE -> ../../../LICENSE, VENDOR-LICENSE -> ../../../third_party/VENDOR-LICENSE), so every `make ko-policy-controller` (ko resolve) now fails at the "Install policy-controller" step across all branches and PRs: tarring kodata: kodata symlink ".../cmd/webhook/kodata/HEAD" resolves to ".../.git/HEAD" which is outside the kodata root The upstream relaxation (ko-build/ko#1699, which widens the allowed root to the enclosing source tree) is merged but not yet released. As a temporary fix, pin setup-ko to ko v0.18.1 (the last release before the strict check) so CI is unblocked. Once a ko release includes sigstore#1699, remove the pin. Verified locally: ko v0.19.1 fails at kodata tarring exactly as CI does, while ko v0.18.1 packs kodata successfully. Removing only the .git symlinks is insufficient because ko v0.19.x then rejects LICENSE next, so pinning is the minimal correct fix. Stale "will use the latest release available for ko" comments are removed since ko is now pinned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ce53efa-a7e9-4046-a24e-34c406601950 Signed-off-by: Nell Shamrell <nellshamrell@gmail.com>
The TUF test fixtures across three packages shared a baked-in expiry of 2026-07-18T08:24:13Z, which has now passed. Any test run after that date failed constructing a TUF client with "final root.json is expired", reddening CI on main and every open PR. Affected packages/tests: - pkg/reconciler/trustroot: TestReconcile/With_repository and the trusted_root.json variant (embedded testdata/*). - pkg/apis/policy/v1alpha1: TestTrustRootValidation (inline base64 blobs). - pkg/tuf: TestDownloadTargetFromSerializedMirror and TestClientFromSerializedMirror (inline base64 blobs, byte-identical to the v1alpha1 ones). Regenerate all three together (they must share the same trust anchor): - Reconciler testdata via `make generate-testdata`. - The inline validRepository/rootJSON blobs in pkg/tuf and pkg/apis/policy/v1alpha1 from the freshly generated tufRepo.tar and root.json. Also refresh the stale "next expiration" comments and replace the manual kind-cluster generation notes with a pointer to `make generate-testdata`. Note: the scaffolding repo builder hardcodes a 6-month expiry, so these fixtures will need periodic regeneration (next: 2027-01-28). Fixes sigstore#1999 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0ce53efa-a7e9-4046-a24e-34c406601950 Signed-off-by: Nell Shamrell <nellshamrell@gmail.com>
|
Update: with the ko pin in place, the approved run showed the e2e suite passing again — the only remaining failures were |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2002 +/- ##
=======================================
Coverage 30.73% 30.73%
=======================================
Files 122 122
Lines 7409 7409
=======================================
Hits 2277 2277
Misses 4897 4897
Partials 235 235 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
Every e2e and build workflow currently fails at the Install policy-controller step (
make ko-policy-controller→ko resolve) with:This affects all branches and PRs (dependabot, and open PRs like #1997 and #2000), not any individual change.
Root cause
The workflows install ko via
ko-build/setup-kowithout aversion:input, so they use setup-ko's defaultlatest-release. ko v0.19.0 (ko-build/ko#1619) added a strict check that rejects kodata symlinks whose targets resolve outside the kodata directory.cmd/webhook/kodata/contains such symlinks (present since 2022):HEAD→../../../.git/HEADrefs→../../../.git/refsLICENSE→../../../LICENSEVENDOR-LICENSE→../../../third_party/VENDOR-LICENSESo when CI silently moved to ko ≥ v0.19.0,
ko resolvebegan failing repo-wide.The upstream relaxation ko-build/ko#1699 ("allow kodata symlinks within the source tree") is merged but not yet released (latest release is v0.19.1; the fix landed after it).
Fix
Pin
setup-koto ko v0.18.1 (the last release before the strict check) across all workflows that install ko, with an explanatory comment. This is an explicit, temporary workaround: v0.19.0's check is a legitimate hardening (it blocks kodata symlinks from packing arbitrary host files into images), so the pin should be removed once a ko release includes #1699.Validation
Reproduced locally with the real ko binaries against
./cmd/webhook:.gitsymlinks is insufficient: ko v0.19.x then rejectsLICENSEnext. Pinning is the minimal correct fix.Notes
ko resolve.# will use the latest release available for kocomments are removed, since ko is now pinned.