Skip to content

agent/test_build.sh: the happy-path case silently skips on every clean checkout, so a permanently-broken build.sh would still pass the gate #46

Description

@ghostpsalm

Found by the breaker while landing #43 (ByteBuddy checksum verification in agent/build.sh).

Where

agent/test_build.sh::test_a_correctly_hashed_cached_jar_still_builds needs a local copy of byte-buddy-1.18.11.jar whose SHA-256 matches the pinned constant to exercise the happy path. agent/lib/ is gitignored, and the test's curl is stubbed to block network access (deliberately, so the suite is hermetic). On a clean checkout with no pre-populated cache, fixture_jar() finds nothing and the case prints SKIP, not FAIL.

Why it matters

./run-tests.sh still prints ALL TESTS PASSED (exit 0) when this case skips — only a 1 skipped check(s) warning line marks it. Demonstrated: a mutant build.sh that always exits non-zero at the checksum step (i.e. never successfully builds anything, in any configuration) passes the entire test suite on a fresh clone, because the only test that would catch it is the one that skipped.

This is exactly the blind spot that matters for #43's own risk: a wrong pinned hash bricks every real install, and nothing in CI/the gate would catch it before it reaches a production box, unless a developer happens to have a valid cached jar locally when they run the tests.

Why not fixed alongside #43

Closing the gap needs one of two real design decisions, not a mechanical fix:

  • allow the test suite a narrow, explicit exception to fetch a verified fixture once (reintroducing controlled network access into an otherwise hermetic suite), or
  • change gate policy so an expected-but-unmet fixture dependency fails the gate instead of skipping it (which has consequences for any other test that legitimately skips for environment reasons).

Also worth noting: this repo currently has no .github/workflows/ CI and no scripts/gate.shrun-tests.sh is run locally, so there's no CI leg to lean on for guaranteed fixture pre-population.

Suggested fix

Pick one:

  1. Add a narrow, clearly-commented exception: run-tests.sh (or a one-time setup step) fetches and pins-verifies the real jar into a test-only cache location before invoking test_build.sh, if not already present.
  2. Make an unresolved fixture dependency a hard failure of the suite (exit 1) rather than a skip, forcing whoever runs the gate to either populate the cache or explicitly acknowledge the gap.

Confidence: Confirmed (mutant demonstrated). Severity: Low probability (requires both a broken build.sh and an unpopulated cache) but meaningful blast radius, since it's the exact failure mode #43 exists to prevent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    audit:securityFinding from a security audit

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions