Found by the breaker while landing #43 (ByteBuddy checksum verification in agent/build.sh).
Where
agent/test_build.sh::test_a_correctly_hashed_cached_jar_still_builds needs a local copy of byte-buddy-1.18.11.jar whose SHA-256 matches the pinned constant to exercise the happy path. agent/lib/ is gitignored, and the test's curl is stubbed to block network access (deliberately, so the suite is hermetic). On a clean checkout with no pre-populated cache, fixture_jar() finds nothing and the case prints SKIP, not FAIL.
Why it matters
./run-tests.sh still prints ALL TESTS PASSED (exit 0) when this case skips — only a 1 skipped check(s) warning line marks it. Demonstrated: a mutant build.sh that always exits non-zero at the checksum step (i.e. never successfully builds anything, in any configuration) passes the entire test suite on a fresh clone, because the only test that would catch it is the one that skipped.
This is exactly the blind spot that matters for #43's own risk: a wrong pinned hash bricks every real install, and nothing in CI/the gate would catch it before it reaches a production box, unless a developer happens to have a valid cached jar locally when they run the tests.
Why not fixed alongside #43
Closing the gap needs one of two real design decisions, not a mechanical fix:
- allow the test suite a narrow, explicit exception to fetch a verified fixture once (reintroducing controlled network access into an otherwise hermetic suite), or
- change gate policy so an expected-but-unmet fixture dependency fails the gate instead of skipping it (which has consequences for any other test that legitimately skips for environment reasons).
Also worth noting: this repo currently has no .github/workflows/ CI and no scripts/gate.sh — run-tests.sh is run locally, so there's no CI leg to lean on for guaranteed fixture pre-population.
Suggested fix
Pick one:
- Add a narrow, clearly-commented exception:
run-tests.sh (or a one-time setup step) fetches and pins-verifies the real jar into a test-only cache location before invoking test_build.sh, if not already present.
- Make an unresolved fixture dependency a hard failure of the suite (
exit 1) rather than a skip, forcing whoever runs the gate to either populate the cache or explicitly acknowledge the gap.
Confidence: Confirmed (mutant demonstrated). Severity: Low probability (requires both a broken build.sh and an unpopulated cache) but meaningful blast radius, since it's the exact failure mode #43 exists to prevent.
Found by the breaker while landing #43 (ByteBuddy checksum verification in
agent/build.sh).Where
agent/test_build.sh::test_a_correctly_hashed_cached_jar_still_buildsneeds a local copy ofbyte-buddy-1.18.11.jarwhose SHA-256 matches the pinned constant to exercise the happy path.agent/lib/is gitignored, and the test'scurlis stubbed to block network access (deliberately, so the suite is hermetic). On a clean checkout with no pre-populated cache,fixture_jar()finds nothing and the case printsSKIP, notFAIL.Why it matters
./run-tests.shstill printsALL TESTS PASSED(exit 0) when this case skips — only a1 skipped check(s)warning line marks it. Demonstrated: a mutantbuild.shthat always exits non-zero at the checksum step (i.e. never successfully builds anything, in any configuration) passes the entire test suite on a fresh clone, because the only test that would catch it is the one that skipped.This is exactly the blind spot that matters for #43's own risk: a wrong pinned hash bricks every real install, and nothing in CI/the gate would catch it before it reaches a production box, unless a developer happens to have a valid cached jar locally when they run the tests.
Why not fixed alongside #43
Closing the gap needs one of two real design decisions, not a mechanical fix:
Also worth noting: this repo currently has no
.github/workflows/CI and noscripts/gate.sh—run-tests.shis run locally, so there's no CI leg to lean on for guaranteed fixture pre-population.Suggested fix
Pick one:
run-tests.sh(or a one-time setup step) fetches and pins-verifies the real jar into a test-only cache location before invokingtest_build.sh, if not already present.exit 1) rather than a skip, forcing whoever runs the gate to either populate the cache or explicitly acknowledge the gap.Confidence: Confirmed (mutant demonstrated). Severity: Low probability (requires both a broken build.sh and an unpopulated cache) but meaningful blast radius, since it's the exact failure mode #43 exists to prevent.