Skip to content

fix(providerledger): give runtime waivers independent expiries, prove 2 (vp-d246m) - #133

Merged
bourgois merged 2 commits into
mainfrom
gc/vp-d246m
Aug 29, 2026
Merged

fix(providerledger): give runtime waivers independent expiries, prove 2 (vp-d246m)#133
bourgois merged 2 commits into
mainfrom
gc/vp-d246m

Conversation

@bourgois

Copy link
Copy Markdown
Collaborator

Symptom

TestCatalogMatchesProductionWiringAndDocumentation was failing on every
branch of Voxist/gascity as of 2026-08-12, taking CI / required red and
blocking all merges — including PR #127 (ADR-0051, a live cleartext-secret
fix).

Root cause

internal/testutil/providerledger/ledger.go's waivedRuntime() baked a
single hardcoded time.Date(2026, August, 12, ...) literal into every call
site, so all 9 runtime.Provider waivers owned by ga-80po0c.3 lapsed
simultaneously at 2026-08-12 00:00 UTC and took the merge gate red
repo-wide. No code change triggered it — the calendar did.

Fix (Option C — mixed, as recommended in the incident bead)

  • waivedRuntime() now takes an explicit expires time.Time argument.
    Each of the 7 remaining waived claims (t3bridge ×2, k8s, herdr, hybrid,
    ssh, tmux) gets its own literal date, staggered Sept 19 – Nov 19 2026 by
    how far off a real proof is, so no two lapse together and a future lapse
    can only ever degrade one entry.
  • Proved 2 of the 9: acp.NewSeamBacked (TestACPConformanceSharedDir) and
    subprocess.NewSeamBacked (TestSubprocessDefaultDirSeamConformance),
    each exercising the production shared-tmp/default-dir composition the
    sibling WithDir proof doesn't reach.
  • runtimeContractWaiverOwner re-pointed ga-80po0c.3vp-8eqrh: the
    old ID doesn't resolve via bd show in this repo's own store, so it
    could never actually be nagged as an expiry approached. vp-8eqrh is a
    bead in this store, opened to own the remaining 6 gaps.
  • CHANGELOG.md [Unreleased]/Fixed entry added per repo convention.

Deliberately not ported: upstream's waiverclock Mode/soft-fail package (a
lapsed waiver fails its owner, not the fleet). That's a new shared policy
package touching internal/testpolicy/resourcecensus and CI workflow
files — cross-cutting and out of scope for this incident fix. Independent
per-entry dates already satisfy the stated acceptance (no shared expiry)
and stop the simultaneous mass-lapse; softening a single future lapse from
hard-fail to warn is left as follow-up.

Verification

  • go build ./..., go vet ./..., scripts/check-core-boundary.sh: clean.
  • go test ./internal/testutil/providerledger/...: green, including
    TestCatalogMatchesProductionWiringAndDocumentation (the test that was
    repo-wide red).
  • go test -tags integration for the new + sibling ACP/subprocess
    conformance tests: green.
  • make test-fast-parallel (the pre-push gate): green on retry. First
    attempt hit 2 failures (TestDriftDetect_WithRealisticPacks_NFR1,
    TestCmdStopSupervisorManagedInvalidCityTomlWaitsForControllerStop) —
    both timing-SLA assertions in cmd/gc, a package this diff never
    touches; both passed cleanly standalone (p95 24.8ms vs the 236.5ms seen
    under 10-way parallel load against a 100ms budget), and host load
    averages were 62/73/101 at the time — consistent with the host-load
    false-red pattern TESTING.md already documents, not a regression from
    this change.

Acceptance (from the incident bead, vp-d246m)

  • go test ./internal/testutil/providerledger/ passes.
  • CI / required expected green (nothing else in the queue touches
    this path; local gate is green).
  • Chosen option (C) and justification stated in the commit message.
  • Waived entries no longer share a single expiry date.

Fixes vp-d246m.

Option C (mixed): prove the cheap providers, re-waive the rest on a short
independent horizon. This fixes the root cause on fork/main directly --
origin/main (upstream gastownhall/gascity) had already fixed this same
outage independently, but fork/main never resynced that far, so it was
still carrying the original defect live.

Root cause: waivedRuntime() baked a single hardcoded
time.Date(2026, August, 12, ...) literal into every call site, so all 9
runtime.Provider waivers owned by ga-80po0c.3 lapsed simultaneously at
2026-08-12 00:00 UTC and took CI/required red repo-wide. No code change
triggered it; the calendar did.

Fix:
- waivedRuntime() now takes an explicit `expires time.Time` argument.
  Each of the 7 remaining waived claims (t3bridge x2, k8s, herdr, hybrid,
  ssh, tmux) gets its own literal date, staggered Sept 19 - Nov 19 2026 by
  how far off a real proof is, so no two lapse together and a future
  lapse can only ever degrade one entry.
- Proved 2 of the 9: acp.NewSeamBacked (TestACPConformanceSharedDir) and
  subprocess.NewSeamBacked (TestSubprocessDefaultDirSeamConformance),
  each exercising the production shared-tmp/default-dir composition the
  sibling WithDir proof doesn't reach. Both verified green under
  -tags integration, 34/34 sub-cases.
- runtimeContractWaiverOwner re-pointed ga-80po0c.3 -> vp-8eqrh: the old
  ID doesn't resolve via `bd show` in this repo's own store, so it could
  never actually be nagged as an expiry approached. vp-8eqrh is a bead in
  this store, opened to own the remaining 6 gaps.

Deliberately not ported: upstream's waiverclock Mode/soft-fail package
(a lapsed waiver fails its owner, not the fleet). That's a new shared
policy package touching internal/testpolicy/resourcecensus and CI
workflow files -- a cross-cutting change out of scope for this incident
fix. Independent per-entry dates already satisfy the stated acceptance
(no shared expiry) and stop the simultaneous mass-lapse; softening a
single future lapse from hard-fail to warn is left as follow-up.

TESTING.md ledger block regenerated to match.

Verification: go build ./..., go vet ./..., scripts/check-core-boundary.sh
all clean. go test ./internal/testutil/providerledger/... green. New
conformance tests pass standalone. make test-fast-parallel: one unrelated
failure (TestRunSetupCommandActivityStreamingSurvivesIdleWindow in
internal/runtime/tmux, a package untouched by this change) reproduced as
flaky in isolation (1/3 runs) with zero code changes between runs.

Signed-off-by: voxist.executor <karel@voxist.com>
Follow the repo's Keep a Changelog convention for internal fixes: adds
an Unreleased/Fixed entry summarizing the independent-expiry fix from
fb0b2c8.

@voxist-bot voxist-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — architect validation gate (ADR-0014), voxist.platform-architect, bead vp-d246m (P0).

Validated against the diff at cf8ac8778c9e921d4b6b2d1af52afb6700af8155, not against the PR summary.

Verified

  1. The owner re-point is real. The fix's premise is that ga-80po0c.3 never resolved as a bead, so the expiry could never be nagged. bd show vp-8eqrhresolves: open, P2, routed to voxist-platform/voxist.platform-engineer, titled for exactly the remaining gaps (t3bridge, k8s, herdr, hybrid, ssh, tmux). The replacement is not another dangling ID, and it has a live owner. This was the check most likely to invalidate the fix; it passes.
  2. The two new proofs are not vacuous. TestACPConformanceSharedDir and TestSubprocessDefaultDirSeamConformance both run the full runtimetest.RunProviderTests suite against the production NewSeamBacked constructor — the shared-tmp/default-dir composition the waivers specifically said the existing WithDir proofs do not reach. They prove the claimed gap, not a neighbouring one.
  3. Root cause addressed, not the symptom. waivedRuntime() now takes expires as a parameter; the shared literal is gone from the constructor. The cliff was structural (one date baked into every call site), and the structure changed — a date bump alone would have re-armed it.
  4. All 7 remaining expiries are distinct (2026-09-19, 09-22, 10-10, 11-05, 11-07, 11-14, 11-19) and each within maxWaiverHorizon (90d; latest is 11-19 vs. the 11-27 cap). Satisfies the "no shared expiry" acceptance criterion — a future lapse degrades one entry, not the merge queue.
  5. Each waiver reason names its unblocking condition, as required ("needs the binary unconditionally present", "needs a fake-double T3 server", "blocked on the tmux and k8s entries it composes"). The debt-forcing intent of the mechanism is preserved.
  6. Regression guard added. ledger_test.go now fails if any claim carries ga-80po0c.3 (or the older ga-80po0c.1.2), so the retired owner cannot silently return.
  7. Option C chosen and justified in the PR body, per the bead's requirement that the option be stated explicitly rather than applied silently.
  8. CI: 31/31 checks green, including CI / required and Integration / packages-core-2-of-4 — the two jobs named as red in the incident report.

Noted, not blocking

  • runtime.builtin.exec and runtime.builtin.t3bridge are both backed by t3bridge.NewSeamBacked; the PR body's "t3bridge ×2" is accurate shorthand, and they are deliberately staggered (11-07 / 11-05) rather than sharing a date.
  • Next cliff is 2026-09-19 (herdr), 21 days out. vp-8eqrh owns it and is routed to the PE pool. That bead is now load-bearing: if it stalls, herdr and tmux lapse in ~3 weeks. It should be worked well before mid-September, not at the deadline.
  • Declining to port upstream's waiverclock soft-fail package is a reasonable scope call for an incident fix; the per-entry dates already satisfy the stated acceptance.

Reviewer identity is voxist-bot, distinct from the PR author — no self-approval and no author-token borrowing.

@bourgois
bourgois merged commit 0c239f2 into main Aug 29, 2026
82 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants