Skip to content

P0: main CI red — CI failure on 03a9b1b #3066

Description

@github-actions

P0: main went red

Workflow: CI concluded failure on main.

Failing run: https://github.com/edobry/minsky/actions/runs/31995649730
HEAD SHA: 03a9b1bfe513b022e0775617b69628478ad94ee6
Head commit: feat(mt#4199): Check the closing message's "waiting on you" against the substrate

What this is

A turn ends saying an ask or task is awaiting the principal. The substrate says it is already terminal. They go to act on it and find nothing to act on.

This happened twice in two days — mem#669 R17 and R18 — and both times the cost landed on the principal rather than on the agent, which is the asymmetry that justifies spending anything on it at all. A wrong internal premise costs a rework cycle; a wrong "this needs you" spends their attention on an action that then fails.

Why a guard, when a cue already says exactly this

/check-premise cue (i) — "about to tell the principal that an open item requires their action … re-read the item's LIVE state THIS turn" — names this shape precisely and shipped as mt#3216 on 2026-07-25. It did not contain the class.

mt#4191 then measured why, and the number settles it: the skill has never been invoked — zero times across 558 conversations and 1488 transcript files — while eight tasks have been filed to improve its cue text. The cue tier here is not weak, it is inert. Both recurrences failed for the same structural reason: the trigger lands while composing the CLOSING MESSAGE, and Stop is the only interception point that runs there.

How it decides

Two halves, deliberately ordered.

The prose gate does no IO and always runs first. A pending-on-principal phrase and a resolvable entity ref must fall within PROXIMITY_CHARS of each other, over the elided tail. Proximity is what separates a claim from a coincidence — without it, any message mentioning a task anywhere and signing off anywhere would fire.

The substrate read happens only on a gate hit, bounded and fail-open. A cold hook bootstrap + connect measures 3.3–5.5s (mt#2430), so an unconditional read would be a genuine per-turn regression.

Ships calibration-first per ADR-024: record-only, no advisory, no attentionCost. The stream records suppressed gate-hits beside fired ones, so both the false-positive and the miss rate are measurable before any enforcement posture is chosen.

Three decisions worth your eye

  • routed and suspended are NOT terminal. A suspended ask genuinely IS awaiting the principal — that true negative is the one this must not spoil.
  • An unresolved ref is dropped, never treated as terminal. Most are uuid-form links or ids from another backend. Treating a lookup miss as a contradiction is the failure direction that would make the guard worse than nothing.
  • LOOKUP_TIMEOUT_MS is 6s, not a tighter-looking number. The bootstrap runs inside the race, so 4s would time out on the first hit in a fresh process — systematically, not occasionally. The residual is left measurable rather than assumed away: a cold miss records lookup-unavailable.

A scope deviation from the spec, recorded not buried

SC2 as written names three entity kinds — ask, PR, task. This ships two.

The spec's In-scope line reasoned from refs_status, which does resolve all three in one call. True of the COMMAND, irrelevant to the guard: a PR's state lives on GitHub, so resolving one inside a Stop hook puts a network round-trip on the closing-message path, where asks and tasks are one local Postgres query. Those are not the same cost and should not have been written as one line.

What is lost: "PR #N is waiting on your review", asserted after it merged, is a real instance of this family and is not caught. It is also the rarer shape — the reviewer bot, not the principal, is the review actor (decision-defaults.mdc §User does not review PRs), so a pending-on-principal claim about a PR is uncommon by construction. [sc2-deferred: mt#4207], which owns closing it without adding that network call. mt#4199's spec carries the same record.

Success criteria

# Criterion Where
1 Extracts entity refs from the closing message and reads live state collectEntityRefs + lookupLiveStates. Uses the UNION of bare refs and minsky:// link targets rather than scanMessage's findings alone — the refinement the planning audit recorded, because those findings are linkification-shaped and a correctly-linked ref appears only among linkTargets
2 Fires on asserted-pending vs terminal classifyResolved. Asks and tasks only — see the deviation above. [sc2-deferred: mt#4207] for the PR arm
3 State side exact, assertion side prose-matched with its phrase set recorded PENDING_ASSERTION_PATTERNS; families recorded per claim in the calibration payload
4 Does not fire on a correct report, or a bare mention tests: AT3, the suspended/routed case, the PLANNING case
5 Calibration-first, evaluation stream over fired AND suppressed run returns only calibration, carrying fired plus suppressionReasons
6 Replay reports the historical fire rate before an enforcement posture scripts/replay-stale-state-assertion-gate.ts — results below
7 /check-premise cue (i) updated DOWNGRADED to optional during planning, on mt#4191's measurement — a ninth cue-text edit cannot be a success criterion for a task whose premise is that cue text never reaches the agent. R17/R18's sub-shapes are recorded in mem#669 instead

Execution evidence

$ bun test --preload ./tests/setup.ts --timeout=15000 ./.minsky/hooks/turn-end-stale-state-assertion-scan.test.ts
 18 pass  0 fail  35 expect() calls

$ bun test --preload ./tests/setup.ts --timeout=20000 ./.minsky/hooks/
 5333 pass  0 fail  10766 expect() calls   (149 files)

$ validate_typecheck -> pass, 0 errors (8 projects, incl. tsconfig.hooks.json)
$ validate_lint      -> pass, 0 errors, 0 warnings (3720 files)

SC6 — the corpus replay. Only the GATE is replayable. The finding is not, and the reason is not a missing harness: an entity's state today is not its state when the message was written, so replaying the finding would resolve historical refs against present-day rows and report a confident wrong number. The gate rate therefore bounds the guard from above — the ceiling on how often the substrate read runs, and so on how often a finding is possible.

$ bun scripts/replay-stale-state-assertion-gate.ts --limit 400
corpus:            ~/.claude/projects/-Users-edobry-Projects-minsky
files scanned:     558
closing messages:  34472
gate hits:         141  (0.409%)
parity mismatches: 0

One closing message in 244 opens a connection; the other 243 cost a few regex scans. The replay also runs the gate from both the .minsky/hooks source and the generated .claude/hooks copy the harness executes, exiting non-zero on disagreement — zero mismatches across all 34472 messages, so a missed recompile surfaces here rather than in production.

All 15 sampled hits are genuine claims of the target shape ("needs your call" on a named task, "sitting in your inbox" on an ask). That is evidence about precision; a sample of 15 is not a rate.

Negative control — the proximity bound. Removed it; a ref 250 chars from the phrase was captured:

(fail) proximity is required: a ref far from the phrase is not captured
 17 pass  1 fail

Negative control — unresolved refs dropped. Made an unresolved ref terminal:

(fail) a ref the substrate cannot resolve is DROPPED, not treated as terminal
 17 pass  1 fail

Negative control — the open-state list. Added suspended to the terminal set:

(fail) AT2 — a suspended or routed ask is NOT a contradiction
 17 pass  1 fail

All three reverted; 18/18 on the committed tree. Each breaks exactly one test, so each is known to discriminate rather than merely to pass.

Notes for review

  • Registry integration surfaced three requirements the tests caught, not inspection: an interceptor-descriptions entry, an interceptor-coordinates entry, and the Stop dispatcher's derived timeout (63s → 71s in .claude/settings.json). Adding a Stop guard has three non-obvious touchpoints.
  • Coordinates classified lexicalRecorder — the guard is a hybrid, named by the half that can be WRONG. The structural decision cannot false-positive; the prose gate can. Same call the family's other hybrid (turn-end-unescalated-incident-scan) makes.
  • Testable-design split: classifyResolved is pure and takes two state maps, so the terminal-vs-open decision is tested by handing it maps rather than by patching the database the guard reaches itself.
  • Scope left open deliberately. mem#669 R18 is the same family and would NOT be caught here — it was an assertion about a mechanism's effect, not an entity's state. The durable shape is any closing-message assertion the substrate contradicts. Widening should be driven by this guard's own evidence, not assumed; recorded in mem#669 rather than pre-empted.

Deploy verification

This PR regenerates src/generated/interceptor-catalog.json, which the deployed cockpit's
/interceptors page renders — so it is a deploy-surface change, not a no-impact one, even though
the file is generated data and the diff is one added row.

After merge I will run deployment_wait-for-latest for the cockpit service with notBefore set to
the merge timestamp, confirm SUCCESS, and confirm the runtime actually started by reading
/api/health's service identity rather than a bare 200 — a deployment predating the merge must not
satisfy the wait.

Task: mt#4199

Co-Authored-By: minsky-ai[bot] <minsky-ai[bot]@users.noreply.github.com>

What this means

A push to main triggered CI and the workflow above did not conclude success. Per
CLAUDE.md user preference ("main must never be broken"), this is severity-1.

Diagnostic checklist

  1. Open the failing run URL above; identify which job/step failed.
  2. Check whether the offending PR was merged with a known-failing required check
    (operator-API bypass via gh api PUT /merge despite enforce_admins).
  3. Confirm enforce_admins is currently enabled:
    gh api repos/edobry/minsky/branches/main/protection --jq .enforce_admins.enabled
    
    Expected: true post-mt#1938. If false, that is itself a separate finding.

Recovery

  1. Open a hotfix branch off current main.
  2. Apply the smallest fix that turns CI green (often a formatter pass or a config
    flip).
  3. Land via the standard Minsky session flow:
    tasks_create → session_start → session_commit → session_pr_create → /review-pr → session_pr_merge.
  4. Verify the post-merge main build is green within ~5 minutes.
  5. Close this issue with a link to the hotfix PR.

Cross-references

Metadata

Metadata

Assignees

No one assigned

    Labels

    main-redAuto-filed by .github/workflows/main-watch.yml when main CI fails (mt#1938)p0Severity 0: production breakage requiring immediate attention

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions