test(integration): fail when a pinned toolchain leaves upstream support (Go, Node, Debian) - #138
Merged
Merged
Conversation
Adds section 22: LIVE checks that every toolchain sandy PINS is still upstream- supported — Go (proxy + base image), Node (base image), and Debian (base image). This catches ROT, not regression, and the distinction cost real security coverage. run-tests.sh asserts the base Go pin is not an *ancient* minor, but that is a static floor: it keeps passing a pin that was supported when written and has since aged out. Exactly what happened before 1.6.0 — the proxy pinned golang:1.24-bookworm, a floating minor tag that delivered 12 patch releases while 1.24 was supported, then went silent for ~6 months once Go dropped it. The monthly --pull refresh kept running and kept refreshing the Debian layer, but no longer the Go stdlib implementing the proxy's TLS/HTTP/CONNECT I/O. Nothing failed; the mechanism went quiet and no test could see it. Only a live check sees that, so it lives in the integration suite: this one has network, run-tests.sh deliberately does not. Sources are authoritative and self-updating, so there is no hardcoded version list to rot in its own right: go.dev/dl (lists only supported majors), nodejs/Release schedule.json, and endoflife.date for Debian. Go is binary (supported or not). Node and Debian DEGRADE instead — Active LTS -> Maintenance -> EOL, and regular -> LTS -> EOL — and the middle tier still ships security fixes, so entering it WARNS (plan the bump) while only EOL FAILS. Verified against the pre-1.6.0 pins: Node 22 warns 'entered Maintenance 2025-10-21' and bookworm warns 'left regular support 2026-07-11'. Scope is exactly the three rot-capable pins. Python is deliberately not checked separately: sandy does not pin it, so Debian's clock IS Python's clock. Rust (--default-toolchain stable), Bun, uv, the npm agent CLIs and the grok installer all resolve at build time and cannot rot. Coverage note: §49's proxy assertion is '^FROM golang:[0-9.]+-trixie', whose version part floats, so golang:1.24-trixie passes it today. The proxy — which parses untrusted wire bytes at the isolation boundary — was the one WITHOUT a supported-minor guard. This closes that. Unreachable upstream skips rather than fails: a flaky fetch must not red a suite that is otherwise about sandy's own behavior. Two bugs caught by testing the code rather than reading it: this suite has no 'warn' helper (pass/fail/skip/info only), and 'python3 - ARG <<HEREDOC' takes the PROGRAM from stdin, so a piped JSON payload was consumed by the heredoc and json.load(sys.stdin) saw nothing. Both fixed; the parsers now use a SINGLE- quoted python3 -c, which leaves stdin free for the pipe and (unlike the double quotes behind the §68 corruption) performs no bash expansion. Verified live: Go 1.26 supported [1.25 1.26]; Node 24 Active LTS (maint 2026-10-20, EOL 2028-04-30); Debian trixie in regular support (EOL 2028-08-09). Negative control: a simulated Go 1.24 pin FAILS. 25 headers/guards/closers balanced; run-tests.sh §80(b) unaffected; zero new shellcheck warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rappdw
force-pushed
the
test/go-support-freshness-check
branch
from
August 12, 2026 13:31
82d1f1d to
ec773d4
Compare
rappdw
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap that let the egress proxy's freshness mechanism go quiet for ~6 months with every test still green.
Rot vs regression — the distinction that cost real coverage
run-tests.shasserts the base Go pin isn't an ancient minor, but that's a static floor: it keeps passing a pin that was supported when written and has since aged out. That is exactly what happened before 1.6.0.golang:1.24-bookwormis a floating minor tag. It delivered 12 real patch releases while 1.24 was supported — then went silent once Go dropped it. The monthly--pullrefresh kept running and kept refreshing the Debian layer, but no longer the Go stdlib implementing the proxy's TLS/HTTP/CONNECT I/O. Nothing errored. The pin never changed — the world did — so no static check could see it.What's checked
Exactly the three rot-capable pins, against authoritative, self-updating sources (no hardcoded version list that could rot in its own right):
FROM golang:X.Y+ baseARG GO_VERSIONgo.dev/dl/?mode=json(lists only supported majors)setup_NN.xnodejs/Releaseschedule.jsonFROM debian:<suite>-slimendoflife.datePython is deliberately not checked separately — sandy doesn't pin it, so Debian's clock is Python's clock. Rust (
--default-toolchain stable), Bun, uv, the npm agent CLIs and the grok installer all resolve at build time and cannot rot.Tiers: warn vs fail
Go is binary. Node and Debian degrade — Active LTS → Maintenance → EOL, and regular → LTS → EOL — and the middle tier still ships security fixes. So entering it warns ("plan the bump"); only EOL fails.
Validated against the pre-1.6.0 pins:
Coverage note (the ironic part)
§49's proxy assertion is
^FROM golang:[0-9.]+-trixie— the version floats, sogolang:1.24-trixiepasses it today. The five guards from #131 protect the base image. So the proxy — the component parsing untrusted TLS ClientHello and HTTP Host bytes at the isolation boundary — was the one without a supported-minor guard. This closes that.Verified live
The negative control is the point: it proves the check detects the rot, rather than merely agreeing with today's state.
Two bugs caught by running the code rather than reading it
warnhelper (onlypass/fail/skip/info) — would have been a runtime "command not found".python3 - ARG <<HEREDOCtakes the program from stdin, so the piped JSON was consumed by the heredoc andjson.load(sys.stdin)saw nothing. Both parsers now use a single-quotedpython3 -c, which leaves stdin free for the pipe and — unlike the double quotes behind the §68 corruption — performs no bash expansion.Unreachable upstream skips rather than fails: a flaky fetch must not red a suite that's otherwise about sandy's behavior.
25 headers/guards/closers balanced;
run-tests.sh§80(b) unaffected; zero new shellcheck warnings;bash -nclean.🤖 Generated with Claude Code