Commit f6f8d5a
feat(mt#4988): Watch mt#4996's reopen triggers so the accept is not decide-and-forget
## Summary
mt#4988 was filed to build burst detection plus a pipeline capturing perishable Railway deploy logs.
**mt#4996 (merged today, PR #3650) removed the reason for both**, and this PR ships what survived.
The task was explicitly queued behind that decision — mt#5006 filed it as *"Reconsider scope first:
largely obviated if mt#4996 accepts the cadence."* The re-scope is recorded as
`## AMENDMENT 2026-09-05` on the task, with superseding criteria and the originals left visible.
**Retired.** The Railway-log capture lost its consumer: mt#4996's three reopen triggers are all
answerable from `review_timing` alone, so nothing reads a deploy log. And surfacing every recovered
burst became alert noise — a recovered burst is now documented expected behaviour at a ~13-day
cadence, and mt#2719's SC5 already excludes `provider_timeout` from paging on a self-healing premise
mt#4996 measured holding at 99.25%.
**Shipped.** mt#4996's accept installed three thresholds and **nothing evaluated them** — a gap
created today. Only one had partial coverage: `reviewer-pre-submit-failure/v1` (mt#4881) pages
per-occurrence on an unrecovered timeout, covering trigger 1's single case but not its aggregate,
and neither of the other two, which are distributional and would go unnoticed indefinitely.
## Key changes
- **`services/reviewer/src/timeout-regime-watch.ts`** (new) — a daily in-process check over a
rolling 30 days of `review_timing`, following `findings-aggregation.ts`'s scheduler shape
(`enabled` flag, `parsePositiveIntEnv`, `isRunning` re-entrancy guard, cycle never throws). The
evaluator is a pure function of its inputs; the query takes an injected `nowMs` with a real
default. Records all three readings against their thresholds every cycle, and notifies only on a
**new** crossing.
- **`services/reviewer/src/server.ts`** — registration beside the other schedulers, opt-in via
`TIMEOUT_REGIME_WATCH_ENABLED`.
- Defaults are mt#4996's recorded values, not round numbers: 2 unrecovered events, 95.00% recovery,
115s p99.9, 30-day window, 118s completing-round cap.
**Deliberately NOT an operator incident.** `ask-emitter.ts`'s operator-incident path renders
*"Reviewer is down — …"* and *"Only you can clear this — the reviewer cannot recover on its own."*
Both are false here: nothing is down, and the remedy is re-running mt#4996's analysis, which an
agent can do. So this notifies through the existing `AlertSink` at `warn`, and the body says so
outright — *"Nothing is broken and no one is paged"* — with a test asserting that sentence so the
framing cannot erode later. **The `OperatorIncidentContext` union is untouched**, so the consumer
set the spec enumerated for it is not disturbed and no contract propagates. Recorded on the task as
`## Implementation record 2026-09-05`.
## Two edge cases the thresholds turn on
- **A quiet window reports recovery as not-computable, never as a number.** Zero timeout events is
the common case in this corpus. Reporting 0% would fire the trigger on every quiet window;
reporting 100% would make it unfalsifiable exactly where there is nothing to measure. The value is
`null`, and a null never crosses.
- **The p99.9 is computed only over rounds BELOW the cap.** A round recorded at the cap is censored
by the timeout mechanism rather than measured (mem#1373) — the correction that changed mt#1897's
conclusion after three passes computed percentiles over cap artifacts. Encoding it here keeps the
trigger from inheriting the same error.
## Correction to the spec's own burst figures
The summary asserted *"37 of 47 timeouts fall on 6 of 42 days; bursts arrive roughly monthly."*
Re-run under the spec's own `tool_use_active is true` filter: **7 burst days in 93** (≈ one per 13
days, so *more* frequent than monthly) and **59 of 115 timeout rows on burst days** (51%, so *less*
concentrated than 79%); over the same last-42-day window, 2 burst days and 24 of 39 rows, not 6 and
37. Recorded as measured, **not diagnosed** — this pass did not establish why the inherited numbers
differ, and says so rather than guessing.
## Acceptance tests
AT1, AT2 and AT4 were superseded by the amendment (AT1′/AT2′/AT4′); **AT3 is retained unchanged**
and is the no-duplicate-with-mt#4881 constraint. Numbering below follows the task spec.
Execution evidence:
```
$ cd services/reviewer && bun test --preload ../../tests/setup.ts src/timeout-regime-watch.test.ts
(pass) evaluateTimeoutRegime > AT2': the measured baseline crosses nothing
(pass) evaluateTimeoutRegime > AT3: ONE unrecovered event does not cross — mt#4881 already pages per occurrence
(pass) evaluateTimeoutRegime > AT4': two unrecovered events cross the aggregate trigger
(pass) evaluateTimeoutRegime > AT1': recovery below 95% crosses
(pass) evaluateTimeoutRegime > recovery exactly at the threshold does not cross
(pass) evaluateTimeoutRegime > a quiet window reports recovery as NOT COMPUTABLE, never as zero
(pass) evaluateTimeoutRegime > p99.9 above 115s crosses; at the threshold it does not
(pass) evaluateTimeoutRegime > no completing rounds reports null, and null never crosses
(pass) sampleTimeoutRegime > Postgres string aggregates are coerced to numbers
(pass) sampleTimeoutRegime > a NULL percentile (no completing rounds) stays null rather than becoming 0
(pass) runTimeoutRegimeWatchCycle > AT1'/SC2': a crossing surfaces exactly once, and not again while it persists
(pass) runTimeoutRegimeWatchCycle > a trigger that clears and re-crosses notifies again
(pass) runTimeoutRegimeWatchCycle > AT2'/AT3: a baseline window with one unrecovered event surfaces nothing
(pass) runTimeoutRegimeWatchCycle > a query failure is swallowed, not thrown — the watch never crashes the service
(pass) runTimeoutRegimeWatchCycle > a missing alert sink does not prevent the crossing being detected
(pass) buildTimeoutRegimeAlertBody > names the crossed trigger, its value, its threshold, and where to reopen
(pass) buildTimeoutRegimeAlertBody > says plainly that this is not an incident
(pass) loadTimeoutRegimeWatchConfig > defaults are mt#4996's recorded values, not round numbers
(pass) loadTimeoutRegimeWatchConfig > the recovery rate is carried as basis points
19 pass / 0 fail / 53 expect() calls
```
**AT1′** — recovery below 95% surfaces exactly once: covered by the `AT1'` evaluator case (92.5%
crosses) and the `AT1'/SC2'` cycle case, which asserts one alert on the first cycle and **no second
alert** while the crossing persists. **AT2′** — the baseline window crosses nothing (evaluator +
cycle). **AT3 (retained)** — one unrecovered event does not cross, so no duplicate lands alongside
`reviewer-pre-submit-failure/v1`; asserted at both the evaluator and cycle level, and confirmed on
REAL data below. **AT4′** — two unrecovered events cross the aggregate trigger.
**SC1′** (every reading recorded with its threshold each cycle) — asserted by
`result.readings` having all three entries on a healthy cycle. **SC2′** (surfaces only on a
crossing, at most once per crossing) — the suppression pair above, plus the clears-and-re-crosses
case proving suppression is not permanent. **SC4′** (config, defaults are mt#4996's values) — the
two `loadTimeoutRegimeWatchConfig` cases. **SC3/SC5** — see Live verification.
**Full reviewer suite** (`server.ts` is touched, so the whole package ran):
`2584 pass / 0 fail across 98 files`.
**Local checks (session `be46aa3f`):** typecheck pass, 0 errors, 8 projects including
`services/reviewer` (whose tsconfig sets `noUncheckedIndexedAccess`); lint **0 errors / 0 warnings**
across 4,391 files; prettier clean. `validatedWorkspace` confirmed as the session dir on both.
## Live verification
The unit tests inject a fake DB, so they say nothing about whether the SQL is valid or returns what
the evaluator expects. The module's query was run verbatim against production `review_timing`, twice.
**1. The real 30-day window — the healthy case.**
```
reviews_with_timeout | timeout_events | unrecovered_events | p999_ms | completing_rounds
36 | 51 | 1 | 108004.608 | 27170
```
Through the shipped thresholds: unrecovered 1 < 2 (not crossed); recovery 50/51 = **98.04%** ≥ 95%
(not crossed); p99.9 **108.0s** < 115s (not crossed). **All three read healthy** — the correct
verdict today, matching mt#4996's accept. This also independently confirms the figures the re-scope
reasoned from: 51 events with 1 unrecovered over 30 days is exactly what mt#4996's per-day table
predicts.
**2. SC5 — replayed over the REAL 2026-09-04 burst, which produces a real crossing.**
A healthy window cannot show that the triggers FIRE, so the same query was replayed over the burst
day that started this whole investigation:
```
reviews_with_timeout | timeout_events | unrecovered_events | recovery_pct | p999_ms | completing_rounds
9 | 9 | 1 | 88.89 | 116141.456 | 1455
```
Through the shipped thresholds: recovery **88.89% < 95% → CROSSES**; p99.9 **116.1s > 115s →
CROSSES**; unrecovered **1 < 2 → does not cross**. So on real burst data the watch fires on two of
three triggers **and stays silent on the one `reviewer-pre-submit-failure/v1` already paged for** —
which is AT3's no-duplicate constraint holding on production data rather than on a fixture. This
discharges SC5 as written ("verified against a real or replayed burst, not only a synthetic one").
Worth stating plainly so the numbers are not over-read: the shipped **window is 30 days**, and one
burst day does not move a 30-day p99.9 (108.0s above). That is the intended behaviour — a single bad
day should not reopen a settled decision — and the replay above is a *narrowed* window used to prove
the trigger arithmetic fires on genuine degradation, not a claim that today's 30-day regime crosses.
## Deploy verification:
All three changed files return `true` from `isDeploySurfaceFile`, verified by running the predicate
over this PR's actual changed-file list rather than recalling a pattern set:
```
$ bun -e 'import { isDeploySurfaceFile } from "./packages/domain/src/deployment/deploy-surface.ts";
for (const f of process.argv.slice(1)) console.log(isDeploySurfaceFile(f), f);' \
services/reviewer/src/timeout-regime-watch.ts services/reviewer/src/timeout-regime-watch.test.ts \
services/reviewer/src/server.ts
true services/reviewer/src/timeout-regime-watch.ts
true services/reviewer/src/timeout-regime-watch.test.ts
true services/reviewer/src/server.ts
```
So this is **not** `[no-deploy-impact]`. After merge I will run `deployment_wait-for-latest` against
the `reviewer` service with `notBefore` set to the merge timestamp and `expectCommitSha` set to the
merge SHA, read `buildIdentity`, and assert the `/health` body's `service` field is
`minsky-reviewer` rather than accepting the status code.
**The watch itself ships disabled** (`TIMEOUT_REGIME_WATCH_ENABLED` defaults to `false`, matching
`findings-aggregation.ts`), so the deploy carries the code without starting the scheduler. That is
deliberate: it makes this merge a no-op at runtime, and enabling it is a one-variable change once
the deploy is confirmed healthy. **No new external-system integration** — the check reads a
first-party Postgres table through the DB handle the service already holds and notifies through an
alert sink it already constructs — so no credential or scope is required and no live-exercise beyond
deploy health is owed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01YT3CvxJhrcGeVCv8P3DGPD
Co-Authored-By: minsky-ai[bot] <minsky-ai[bot]@users.noreply.github.com>5 files changed
Lines changed: 924 additions & 0 deletions
File tree
- services/reviewer
- src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
200 | 200 | | |
201 | 201 | | |
202 | 202 | | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
203 | 259 | | |
204 | 260 | | |
205 | 261 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
620 | 620 | | |
621 | 621 | | |
622 | 622 | | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
623 | 651 | | |
624 | 652 | | |
625 | 653 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| |||
1952 | 1953 | | |
1953 | 1954 | | |
1954 | 1955 | | |
| 1956 | + | |
| 1957 | + | |
| 1958 | + | |
| 1959 | + | |
| 1960 | + | |
| 1961 | + | |
| 1962 | + | |
| 1963 | + | |
| 1964 | + | |
| 1965 | + | |
| 1966 | + | |
| 1967 | + | |
1955 | 1968 | | |
1956 | 1969 | | |
1957 | 1970 | | |
| |||
0 commit comments