fix(providerledger): give runtime waivers independent expiries, prove 2 (vp-d246m) - #133
Merged
Conversation
Option C (mixed): prove the cheap providers, re-waive the rest on a short independent horizon. This fixes the root cause on fork/main directly -- origin/main (upstream gastownhall/gascity) had already fixed this same outage independently, but fork/main never resynced that far, so it was still carrying the original defect live. Root cause: waivedRuntime() baked a single hardcoded time.Date(2026, August, 12, ...) literal into every call site, so all 9 runtime.Provider waivers owned by ga-80po0c.3 lapsed simultaneously at 2026-08-12 00:00 UTC and took CI/required red repo-wide. No code change triggered it; the calendar did. Fix: - waivedRuntime() now takes an explicit `expires time.Time` argument. Each of the 7 remaining waived claims (t3bridge x2, k8s, herdr, hybrid, ssh, tmux) gets its own literal date, staggered Sept 19 - Nov 19 2026 by how far off a real proof is, so no two lapse together and a future lapse can only ever degrade one entry. - Proved 2 of the 9: acp.NewSeamBacked (TestACPConformanceSharedDir) and subprocess.NewSeamBacked (TestSubprocessDefaultDirSeamConformance), each exercising the production shared-tmp/default-dir composition the sibling WithDir proof doesn't reach. Both verified green under -tags integration, 34/34 sub-cases. - runtimeContractWaiverOwner re-pointed ga-80po0c.3 -> vp-8eqrh: the old ID doesn't resolve via `bd show` in this repo's own store, so it could never actually be nagged as an expiry approached. vp-8eqrh is a bead in this store, opened to own the remaining 6 gaps. Deliberately not ported: upstream's waiverclock Mode/soft-fail package (a lapsed waiver fails its owner, not the fleet). That's a new shared policy package touching internal/testpolicy/resourcecensus and CI workflow files -- a cross-cutting change out of scope for this incident fix. Independent per-entry dates already satisfy the stated acceptance (no shared expiry) and stop the simultaneous mass-lapse; softening a single future lapse from hard-fail to warn is left as follow-up. TESTING.md ledger block regenerated to match. Verification: go build ./..., go vet ./..., scripts/check-core-boundary.sh all clean. go test ./internal/testutil/providerledger/... green. New conformance tests pass standalone. make test-fast-parallel: one unrelated failure (TestRunSetupCommandActivityStreamingSurvivesIdleWindow in internal/runtime/tmux, a package untouched by this change) reproduced as flaky in isolation (1/3 runs) with zero code changes between runs. Signed-off-by: voxist.executor <karel@voxist.com>
Follow the repo's Keep a Changelog convention for internal fixes: adds an Unreleased/Fixed entry summarizing the independent-expiry fix from fb0b2c8.
voxist-bot
approved these changes
Aug 29, 2026
voxist-bot
left a comment
There was a problem hiding this comment.
APPROVE — architect validation gate (ADR-0014), voxist.platform-architect, bead vp-d246m (P0).
Validated against the diff at cf8ac8778c9e921d4b6b2d1af52afb6700af8155, not against the PR summary.
Verified
- The owner re-point is real. The fix's premise is that
ga-80po0c.3never resolved as a bead, so the expiry could never be nagged.bd show vp-8eqrh→ resolves: open, P2, routed tovoxist-platform/voxist.platform-engineer, titled for exactly the remaining gaps (t3bridge, k8s, herdr, hybrid, ssh, tmux). The replacement is not another dangling ID, and it has a live owner. This was the check most likely to invalidate the fix; it passes. - The two new proofs are not vacuous.
TestACPConformanceSharedDirandTestSubprocessDefaultDirSeamConformanceboth run the fullruntimetest.RunProviderTestssuite against the productionNewSeamBackedconstructor — the shared-tmp/default-dir composition the waivers specifically said the existingWithDirproofs do not reach. They prove the claimed gap, not a neighbouring one. - Root cause addressed, not the symptom.
waivedRuntime()now takesexpiresas a parameter; the shared literal is gone from the constructor. The cliff was structural (one date baked into every call site), and the structure changed — a date bump alone would have re-armed it. - All 7 remaining expiries are distinct (2026-09-19, 09-22, 10-10, 11-05, 11-07, 11-14, 11-19) and each within
maxWaiverHorizon(90d; latest is 11-19 vs. the 11-27 cap). Satisfies the "no shared expiry" acceptance criterion — a future lapse degrades one entry, not the merge queue. - Each waiver reason names its unblocking condition, as required ("needs the binary unconditionally present", "needs a fake-double T3 server", "blocked on the tmux and k8s entries it composes"). The debt-forcing intent of the mechanism is preserved.
- Regression guard added.
ledger_test.gonow fails if any claim carriesga-80po0c.3(or the olderga-80po0c.1.2), so the retired owner cannot silently return. - Option C chosen and justified in the PR body, per the bead's requirement that the option be stated explicitly rather than applied silently.
- CI: 31/31 checks green, including
CI / requiredandIntegration / packages-core-2-of-4— the two jobs named as red in the incident report.
Noted, not blocking
runtime.builtin.execandruntime.builtin.t3bridgeare both backed byt3bridge.NewSeamBacked; the PR body's "t3bridge ×2" is accurate shorthand, and they are deliberately staggered (11-07 / 11-05) rather than sharing a date.- Next cliff is 2026-09-19 (herdr), 21 days out.
vp-8eqrhowns it and is routed to the PE pool. That bead is now load-bearing: if it stalls, herdr and tmux lapse in ~3 weeks. It should be worked well before mid-September, not at the deadline. - Declining to port upstream's
waiverclocksoft-fail package is a reasonable scope call for an incident fix; the per-entry dates already satisfy the stated acceptance.
Reviewer identity is voxist-bot, distinct from the PR author — no self-approval and no author-token borrowing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
TestCatalogMatchesProductionWiringAndDocumentationwas failing on everybranch of Voxist/gascity as of 2026-08-12, taking
CI / requiredred andblocking all merges — including PR #127 (ADR-0051, a live cleartext-secret
fix).
Root cause
internal/testutil/providerledger/ledger.go'swaivedRuntime()baked asingle hardcoded
time.Date(2026, August, 12, ...)literal into every callsite, so all 9
runtime.Providerwaivers owned byga-80po0c.3lapsedsimultaneously at 2026-08-12 00:00 UTC and took the merge gate red
repo-wide. No code change triggered it — the calendar did.
Fix (Option C — mixed, as recommended in the incident bead)
waivedRuntime()now takes an explicitexpires time.Timeargument.Each of the 7 remaining waived claims (t3bridge ×2, k8s, herdr, hybrid,
ssh, tmux) gets its own literal date, staggered Sept 19 – Nov 19 2026 by
how far off a real proof is, so no two lapse together and a future lapse
can only ever degrade one entry.
acp.NewSeamBacked(TestACPConformanceSharedDir) andsubprocess.NewSeamBacked(TestSubprocessDefaultDirSeamConformance),each exercising the production shared-tmp/default-dir composition the
sibling
WithDirproof doesn't reach.runtimeContractWaiverOwnerre-pointedga-80po0c.3→vp-8eqrh: theold ID doesn't resolve via
bd showin this repo's own store, so itcould never actually be nagged as an expiry approached.
vp-8eqrhis abead in this store, opened to own the remaining 6 gaps.
[Unreleased]/Fixedentry added per repo convention.Deliberately not ported: upstream's waiverclock Mode/soft-fail package (a
lapsed waiver fails its owner, not the fleet). That's a new shared policy
package touching
internal/testpolicy/resourcecensusand CI workflowfiles — cross-cutting and out of scope for this incident fix. Independent
per-entry dates already satisfy the stated acceptance (no shared expiry)
and stop the simultaneous mass-lapse; softening a single future lapse from
hard-fail to warn is left as follow-up.
Verification
go build ./...,go vet ./...,scripts/check-core-boundary.sh: clean.go test ./internal/testutil/providerledger/...: green, includingTestCatalogMatchesProductionWiringAndDocumentation(the test that wasrepo-wide red).
go test -tags integrationfor the new + sibling ACP/subprocessconformance tests: green.
make test-fast-parallel(the pre-push gate): green on retry. Firstattempt hit 2 failures (
TestDriftDetect_WithRealisticPacks_NFR1,TestCmdStopSupervisorManagedInvalidCityTomlWaitsForControllerStop) —both timing-SLA assertions in
cmd/gc, a package this diff nevertouches; both passed cleanly standalone (p95 24.8ms vs the 236.5ms seen
under 10-way parallel load against a 100ms budget), and host load
averages were 62/73/101 at the time — consistent with the host-load
false-red pattern TESTING.md already documents, not a regression from
this change.
Acceptance (from the incident bead, vp-d246m)
go test ./internal/testutil/providerledger/passes.CI / requiredexpected green (nothing else in the queue touchesthis path; local gate is green).
Fixes vp-d246m.