Capacity recovery: typed errors, shared deadline retry, guard fix, turn checkpoint/resume - #242
Open
Calmingstorm wants to merge 17 commits into
Open
Capacity recovery: typed errors, shared deadline retry, guard fix, turn checkpoint/resume#242Calmingstorm wants to merge 17 commits into
Calmingstorm wants to merge 17 commits into
Conversation
New src/llm/errors.py: LLMError(RuntimeError) hierarchy — Capacity (model-scoped, retryable), RateLimit (account-scoped, fast-fail after internal rotation), Transport, Auth, Request. retry_after stays a plain attribute for the two duck-typed consumers. openai_codex: stream readers parse the SSE error object structurally (type/code/retry_after; error events and response.failed both handled); capacity markers (service_unavailable_error / server_is_overloaded / server_error) escape _send_with_retries immediately as LLMCapacityError — no inner retry burn, no client-breaker count, never account rotation. Terminal raises typed with identical message strings (401→Auth, 429-exhausted→RateLimit, 5xx/conn→Transport, other→Request); 429 rotation via _mark_limited unchanged. ollama/kimi exhaustion raises wrapped the same way (kimi 429 carries the parsed Retry-After).
src/llm/model_breaker.py: ModelCapacityBreaker keyed provider:effective- model, owned by a registry (BotServices-scoped — survives client rebuilds and live reloads). Counted once per failed LOGICAL GENERATION, true single-probe half-open with token-attributed resolution (a stray non-probe failure can neither escalate nor release someone else's probe), adaptive cooldown doubling to a cap and resetting on success. src/llm/recovery.py: generate_with_recovery — THE shared policy for chat/ agents/loops. Monotonic deadline (default 300s; resume passes remaining budget so restarts never refresh it), full-jitter backoff capped 45s, retry_after honoured as a clamped floor, waits bounded by remaining budget and raced against the caller's cancel event. Retryable: Capacity/ Transport/CircuitOpen (waits THROUGH the client breaker). Fast-fail: Auth/Request/RateLimit (rotation already exhausted — quota semantics unchanged). Unclassified exceptions escape immediately (the agents bare-except defect is fixed, not spread). CircuitOpenError gains an optional model stamp.
Chat (_call_llm): the CircuitOpenError-only sleep-and-retry pair is replaced by generate_with_recovery — capacity/transport/breaker-open retried up to the generation deadline, cancellable by /stop (graceful stop path, not an error turn), fast-fail classes unchanged. Loops (_call_loop_llm): same recovery in-iteration with retry_circuit_open= False — CircuitOpenError still escapes to the loop manager (pinned asymmetry preserved; gateway bypass untouched per RFC-001 §4.3). Agents: recovery moved INSIDE the iteration callbacks (_agent_generate, breaker keyed on the agent's effective model); the manager's bare-except single retry ladder is REMOVED — lifetime exhaustion still wins as TIMEOUT, everything else fails fast with readable errors. Gateway: capacity/CircuitOpen no longer feed the sticky guard counter (the double penalty that latched llm_* UNAVAILABLE until restart); capacity marks a self-expiring transient DEGRADED instead. New notify_generation_success(provider) clears a latched guard from bypass paths, driven by immutable response provenance only (missing provenance = no-op, never a guess). SubsystemGuard grows transient-DEGRADED with lazy expiry; real failures/successes still own the counter. Characterization pins amended DELIBERATELY (agent ladder tests, chat breaker-retry pins) — called out for review. New tests/test_guard_capacity.py.
src/turn_state/store.py: one SQLite DB (data/turn_state/turns.sqlite3 in production wiring) holding checkpoints, ledger rows, leases, and terminal tombstones so state transitions are atomic; content-addressed blob dir beside it (tmp+rename). KnowledgeStore conventions: WAL, busy_timeout, one connection, write lock, available-property degradation. Fencing is three-part per Odin's round-3 clarification: every state write is conditional on (turn_generation, revision, lease_token); an expired owner replaying an old revision or a cleared token gets StaleTurnError and must stop. Heartbeats extend the lease WITHOUT bumping the revision or advancing last_progress_at; checkpoint writes advance last_progress_at only when progressed=True (recovery waits never fake progress). Recovery deadlines persist as absolute UTC. Ledger: PREPARED→RUNNING→APPLIED/DEFINITELY_FAILED with OUTCOME_UNKNOWN for interrupted work; intent validation rejects empty/duplicate tool_call_ids BEFORE execution; effect fingerprint stored as secondary reconciliation evidence only. Boot sweep suspends stale-lease ACTIVE turns and marks their in-flight ops OUTCOME_UNKNOWN (never rerun). Three TTL clocks: 24h resumable from last REAL progress, 7d diagnostic payloads then tombstone, 90d ledger — OUTCOME_UNKNOWN/MANUAL rows never auto-expire. Terminal payloads compact immediately; identity + ledger tombstones remain. Fail-closed contract: init failure disables the feature loudly; mid-turn write failures raise TurnStateUnavailable. Also: two stale-ladder agent pins amended deliberately (test_recovery), loop-path fake gateway updated (test_trajectory_completeness).
src/turn_state/codec.py classifies EVERY _ChatTurn field as PERSISTED (restored verbatim — includes all six one-shot guard flags, continuation and validation budgets, post-compression messages, the possibly-rebound system_prompt, op details, pending validations) or RECONSTRUCTED with the reason (message re-fetched; _cancel fresh from channel_state; tools re-derived from CURRENT catalog+permissions — current policy wins; policy by stable name; trace closed one-way + fresh linked resume segment). tests/test_turn_checkpoint_codec.py carries the census pin: a new _ChatTurn field fails the test until classified — permanently enforcing that resume can never silently re-arm guard budgets (hard rule). Also: stuck-tracker export/import (fingerprints + warned), FULL trajectory persistence with an iteration_revision guard (extra rows dropped on restore — no double-append), base64 image blocks externalized to content-addressed blobs and re-inlined on restore.
New src/turn_state/durability.py: TurnDurability drives the five-point write invariant; disabled instances no-op so the loop keeps one code path. Admission in _prepare_chat_turn (Discord-source chat turns only, resolved the same way the trajectory source is; web shims excluded — v1 fence). Hook placement: WI-1 in _execute_tool_calls (response transcript + PREPARED intents durable BEFORE any execution; malformed intent batches bounce back as matched error results without executing), WI-2 in _run_one_tool (RUNNING gates the effect, fail-closed), WI-3 settles per tool (APPLIED / DEFINITELY_FAILED / OUTCOME_UNKNOWN for interrupted executions, via a no-downgrade guarded settle in the wait_for-timeout race), WI-4 one fenced checkpoint per settled batch, WI-5 at both guard- injection continue points (consumed one-shot flags durable before the LLM retry). on_generation_start persists the absolute UTC recovery deadline before every LLM call. Capacity budget exhaustion now SUSPENDS instead of discarding: work preserved, clean user message naming auto-resume plus the manual resume window; suspension-persistence failure degrades honestly to the plain error (no false preservation claims). Terminal settlement: COMPLETED / FAILED / CANCELLED (cancellation carried past _clear_active clearing the shared event — a cancelled turn never looks resumable), best-effort where no further external effect can follow; durability failures mid-turn halt the turn through the escape guard (fail-closed, proven by integration test: second batch never starts after the store dies). tests/test_write_invariant_integration.py drives the REAL runner against a real store for all of the above.
New src/discord/turn_resume.py (TurnResumeManager). Explicit path: a bare resume/continue from the ORIGINAL requester, running inside the normal intake pipeline (lock/delivery/session inherited); the trigger is consumed as a command and never enters the frozen transcript. Auto path: registered at suspension, in-process only (restart => explicit-only by design); the waiter samples a session baseline, then actively claims the model breaker probe slot when the cooldown elapses — the resumed generation IS the capacity probe, and a re-suspension re-arms the waiter with the breaker escalating cooldown as natural pacing. Auto-resume stands down fail-safe when the session advanced or cannot be read. Admission (both paths): re-fetch the original message; same author + unchanged content digest required; deleted/edited => TERMINAL_REJECTED (payload compacted, never executable folklore); tools re-derived from the CURRENT catalog + permission filter — current security policy wins; single resume winner via the fenced resume lease. Replay: unmatched tool_use blocks are repaired from the ledger (APPLIED replays its stored result; unknown/never-ran stated truthfully as result blocks) — matched blocks guaranteed, NOTHING re-executed. The resumed generation carries only its REMAINING recovery budget from the persisted UTC deadline (one attempt when exhausted — never a fresh five minutes); later generations budget normally. run_resumed() enters the same guard envelope as run(); the iteration loop starts from the restored index. Intake: explicit-resume branch ahead of tool_loop.run; the error-path session marker now says the work is PRESERVED and resumable when a suspension row exists (was: "may ask to retry").
Two schema-only top-level sections (deliberately NOT added to the tracked config.yml template — six sections already live schema-only, and this keeps the deploy free of the skip-worktree dance): llm_recovery (generation deadline, backoff cap, model-breaker knobs) and turn_state (enabled, db path, auto_resume, the three retention clocks). Wiring: ModelBreakerRegistry + recovery-policy source + TurnStateStore are BotServices-owned (client rebuilds and live reloads never reset breaker or store state); gateway receives both; ToolLoopDeps carries the store; TurnResumeManager constructed when the store is up, its suspension callback attached to the runner, and the manager handed to the message pipeline for explicit resume. Housekeeping cleanup_stale gains the three-clock TTL sweep (guarded, house per-step style); shutdown_services closes the store.
Branch coverage for the new paths: /stop before the first iteration and during a recovery wait (both TERMINAL_CANCELLED, graceful), task-cancel with a failing durable settle (cancellation still propagates promptly), escape-path settle failure (the ORIGINAL fail-closed error wins — the epilogue guard widened to BaseException so a cancellation delivered inside the settle can never replace it), failed suspension (plain error, no false preservation claims, TERMINAL_FAILED), duplicate tool_call ids (bounced matched, nothing executed), the housekeeping three-clock sweep (runs + swallows store failures), kimi Retry-After parsing on 429 exhaustion, and guard no-ops for unregistered names. test_agent_trajectory recovery pin amended deliberately (the manager ladder is gone; recovery_attempts stays 0 and the field remains for shape compatibility). Typed-object deps given real types for mypy (BotServices, MessagePipelineDeps, durability suspend closure). coverage-baseline.json regenerated (--update-baseline): all new files ratcheted, 26 files improved, 0 decreased, total 88.2%.
B1 (ledger fencing): every operation transition is now ONE atomic statement conditional on generation + EXPECTED REVISION + lease token + ACTIVE status + live lease (server-side EXISTS predicate) AND a legal prior op state — APPLIED can never re-enter RUNNING, a stale-revision owner mutates nothing, and intent recording uses plain fenced INSERTs (never OR REPLACE; an id collision raises LedgerIntentError and resets nothing). _verify_lease and its TOCTOU window are gone. B2 (redelivery fail-open): admission returns a disposition; an existing identity REFUSES fresh execution — terminal rows answer 'already processed', live-lease rows 'in progress', suspended (or expired-ACTIVE, swept here) rows point at resume. Existing identity never means 'run without durability'; only store-unavailable does. B3 (cancellability): the in-flight provider attempt is raced against the cancel event — /stop cancels and awaits the attempt task itself, then releases any held probe. The recovery budget still deliberately bounds WAITING, not a healthy in-flight stream (the v3.58.2 wall killed healthy >10-min xhigh generations; client transport limits own the attempt) — documented in the module contract. B4 (stranded ACTIVE): the boot sweep now suspends ALL ACTIVE rows (the store is single-process — nothing can legitimately be in flight at construction), and housekeeping runs a periodic expired-ACTIVE defense sweep. Fast-restart-inside-the-lease reproduced and pinned. B5 (resume stranding): reconstruction (payload restore, tool derivation, transcript repair, cancel check) runs BEFORE the execution lease is acquired; the residual post-acquire window releases the fenced lease back to SUSPENDED (release_acquired_sync). A corrupt payload self-heals to TERMINAL_REJECTED inside load_resumable instead of raising. B6 (auto-resume races): the advance check anchors on the session length captured synchronously AT SUSPENSION (only the suspending turn's own +2 intake bookkeeping is legitimate growth; no sampling window exists), and waiter cleanup is ownership-sensitive (compare-and-pop by task identity). B7 (typed pool exhaustion): CodexAuthPool raises LLMRateLimitError (all rate-limited) / LLMAuthError (all failed / no credentials) at the source — fast-fail at the recovery layer, never an unclassified defect; messages unchanged, RuntimeError-compatible. B8 (storage): turn-state dirs 0700, DB + WAL/SHM + blobs 0600 (secure tmp+rename writes); assistant tool_use arguments are secret-scrubbed at snapshot time (audit-storage parity; results were already scrubbed at source, credential-bearing user messages never reach a turn). CI: executor.py baseline aligned to CI's measurement (env-variant lines); new housekeeping sweep branches covered. Suite 7,886/5, lint/type gates new=0, coverage findings=0 at 88.3%.
R2-B1 (lease vs long work): the owner now BEATS the lease alive — a per-turn heartbeat task (lease_ttl/3 cadence) started at admission and resume, stopped on suspend/terminal settlement, exiting cleanly on a stolen lease or dead store. _fenced_update carries the COMPLETE live fence (ACTIVE status + unexpired lease joined generation/revision/token), so an expired owner can neither checkpoint (which used to silently renew the lease) nor settle. Pinned: a tool 2.5x the TTL completes; the expired-checkpoint renewal repro is rejected. R2-B2 (runtime admission fail-open): once the store was wired available, an admission I/O failure (raised OR disposition) refuses execution with a bounded notice — identity-unverifiable never means run-unledgered. Legacy runs remain only for durability off/failed at wiring. R2-B3 (auto-resume off-by-one): intake appends the user message BEFORE the turn, so the suspension capture already includes it — legal growth is exactly +1 (the preservation marker). Pinned through the REAL MessagePipeline path end-to-end (suspend via pipeline.run → marker → auto-resume admits → TERMINAL_COMPLETED → reply on the original message). R2-B4 (orphaned attempt): _attempt_cancellable's finally now cancels AND awaits both helper tasks — cancelling the recovery owner can no longer leave a live provider request behind an abandoned probe. Pinned. R2-B5 (scrub coverage): storage scrub = the audit-parity tool-aware redaction (_scrub_tool_input_for_storage, e.g. email bodies) composed with a RECURSIVE secret-pattern scrub, applied to EVERY persisted copy of tool arguments (transcript tool_use blocks AND trajectory iterations). Pinned: nested auth headers and email bodies never persist raw. R2-B6 (OUTCOME_UNKNOWN enforcement): unresolved operations HALT continuation — auto-resume stands down permanently; explicit resume returns the op list, transitions them to MANUAL_RESOLUTION_REQUIRED, and closes the turn TERMINAL_REJECTED. No generation happens (pinned: zero LLM calls). 'Never auto-rerun' is enforcement now, not prompt advice. Also: corrupt checkpoint payloads self-heal to TERMINAL_REJECTED inside load_resumable; two auto-resume tests fixed to wait for TERMINAL status (they raced the in-flight ACTIVE resume under coverage instrumentation). Suite 7,905/5, lint/type gates new=0, coverage findings=0 at 88.4%.
1 (revision race): the in-memory lease revision now advances UNDER the
same write lock as the database mutation — a heartbeat interleaved with a
checkpoint can no longer read a stale revision, falsely StaleTurnError,
and stop beating a lease the turn still owns. Pinned with 40 interleaved
concurrent checkpoint+heartbeat cycles.
2 (wired-store death): TurnDurability.admit distinguishes store-None
(durability off at wiring — legacy) from a present-but-dead store (was
wired available; identity unverifiable — refuses with admission_error).
3 (session TOCTOU): the authoritative session-advance check now runs
UNDER the channel lock immediately before reconstruction/acquisition;
the pre-lock check only avoids queueing pointlessly. Pinned: a session
advancing while auto-resume queues on the lock stands it down.
4 (op-details copy): _op_tool_details persists through the same
tool-aware + recursive scrub as the transcript and trajectory copies
(input rewritten only when present; result/error strings pattern-
scrubbed; non-dict legacy entries pass through). Pinned: nested auth
values never persist raw in ANY copy.
5 (structural validation): codec.validate_payload runs inside
restore_field_values — codec version, policy, fields envelope, presence
of every persisted field, and construction-critical types are checked
BEFORE any lease exists; CheckpointInvalidError terminally rejects. The
'{}' payload repro now lands TERMINAL_REJECTED, never bounced back to
SUSPENDED for an infinite retry loop.
6 (zero-LLM resume path): the explicit-resume check moved ahead of
prompt/history assembly in intake — get_task_history (whose compaction
can invoke the LLM) is never called for a resume trigger. Pinned at the
MessagePipeline level with an instrumented get_task_history proving zero
calls end-to-end.
Suite 7,914/5, lint/type gates new=0, coverage findings=0 at 88.4%.
1 (revision ABA): every revision value — the expected WHERE revision, the new SET revision, and the shared lease publish — now derives from ONE read taken under the write lock. The pre-lock capture let a delayed heartbeat pair a fresh WHERE with a stale SET and REGRESS both the row and the lease (Odin forced DB 1→0). Pinned: a stale lease copy is fenced out entirely and can never regress the row. 2 (validation + fall-through): validate_payload now requires an exact non-negative non-bool generation_seq, exact-int typed fields, and transcript shape (every entry a message object; content a string or a list of block objects — messages=[None] and content=[42] reject). ALL fallible reconstruction (restore, tool derivation, transcript repair) lives inside one pre-acquisition rejection boundary. try_explicit_resume now has a hard contract: once the trigger is recognized against preserved work it NEVER raises and NEVER returns None — an internal failure returns a notice tuple, so a recognized resume can never fall through into a fresh normal turn (Odin reproduced a fresh LLM response; pinned: zero fresh generation, TERMINAL_REJECTED, no SUSPENDED loop). 3 (monotonic session fence): SessionManager gains a per-channel mutation revision bumped on every semantic mutation (append, removal, compaction, fallback compaction, secret scrub, tombstone) — process-local by design, matching its only consumer. Auto-resume anchors on the revision instead of len(messages), closing the add-then-remove ABA (Odin resumed despite an intervening message; pinned: count restored, revision +2, stands down). 4 (key-aware redaction): _deep_scrub_strings adds recursive case-insensitive sensitive-KEY redaction (normalized exact match — auth redacts, author does not) so opaque credentials with no token-shaped signature never persist under Authorization/password/api-key/etc., at any nesting depth, in any persisted copy. Pinned with opaque values. Suite 7,921/5, lint/type gates new=0, coverage findings=0 at 88.4%.
1 (fall-through, final hole): _explicit_resume_recognized returning None when the second load comes back empty (corrupt payload self-healed to rejected / another resumer won / expired) now returns a notice tuple — once row_summary establishes preserved work, NO outcome runs the trigger as a fresh tool-capable turn. Odin's corrupt-JSON end-to-end repro pinned: notice delivered, zero fresh generation. 2 (exhaustive checkpoint schema): validate_payload now checks EVERY persisted field with exact types — guard flags via type(v) is bool (his hedging_retried=0 repro re-armed a consumed guard and made two calls; now: TERMINAL_REJECTED, zero generation, pinned end-to-end), non-boolean bounded integers, counter<=cap invariants (continuation, validation, iteration<=chat_cap), typed string/dict lists, the exact stuck-tracker export shape, and the trajectory envelope (iterations list-of-objects, non-negative iteration_revision, history list). The field SET is exact: unknown extras reject too (they used to reach _ChatTurn(**fields) after acquisition and loop back to SUSPENDED). Validation baseline in tests now derives from a REAL snapshot so it can never drift from the schema. 3 (monotonic lease expiry): renewals go through lease_expires_at=MAX(lease_expires_at, ?) in SQL — a delayed same-owner renewal carrying an earlier expiry (same generation/token/revision lineage passes the fence) can no longer move the expiry backward and expose a healthy owner to the expiry sweep. Pinned: the delayed smaller renewal never regresses, checkpoint path included, sweep untouched. Suite 7,923/5, lint/type gates new=0, coverage findings=0 at 88.4%.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LLM capacity recovery + SubsystemGuard fix + turn checkpoint/resume
One PR per Aaron's directive, implementing the design settled over three bridge rounds (your round-2 answer + round-3 clarifications on the plan gist, all binding). Field context: sol's 2026-07-29/30 degradation — 110
server_is_overloadederrors arriving inside HTTP 200 SSE streams, each killing a main-thread turn in ~1.5s and discarding every completed tool call.What ships (by commit)
src/llm/errors.py(LLMError(RuntimeError): Capacity/RateLimit/Transport/Auth/Request;.retry_afterkept as a plain attribute for the two duck-typed consumers). The Codex stream readers parse the SSE error object structurally; capacity markers (service_unavailable_error/server_is_overloaded/server_error, matched on type AND code,response.failednested shape handled) escape_send_with_retriesimmediately — no inner retry burn, no client-breaker count, never account rotation. Terminal raises typed with byte-identical message strings. ollama/kimi exhaustion wrapped (kimi 429 carries parsed Retry-After).src/llm/model_breaker.py, keyedprovider:effective-model(round-3 fix: shutdown cleanup for leaked resources #6), registry owned byBotServices(live reloads never reset it), counted once per failed logical generation, true single-probe half-open with token-attributed resolution, adaptive cooldown (doubles to cap, resets on success).src/llm/recovery.py: one policy for chat/agents/loops. Monotonic deadline (default 300s), full-jitter capped 45s,retry_afterhonoured as a clamped floor, waits bounded by remaining budget and raced against /stop. Fast-fail: Auth, Request, RateLimit (round-3 fix: codex auth races, session thread safety, agent lifecycle #2 — internal rotation already exhausted; quota semantics unchanged). Unclassified exceptions escape immediately (the agents bare-exceptdefect is fixed, not spread)._call_llmuses the shared recovery (CircuitOpen waited through); loops get in-iteration recovery withretry_circuit_open=False(CircuitOpenError still escapes to the loop manager — pinned); agents recover inside the iteration callbacks keyed on their effective model, and the manager ladder is REMOVED (lifetime exhaustion still wins as TIMEOUT). Gateway: capacity/CircuitOpen no longer feed the sticky guard counter (the double penalty that latchedllm_*UNAVAILABLE until restart); capacity marks a self-expiring transient DEGRADED; newnotify_generation_success(provenance_provider)clears a latched guard from bypass paths — provenance only, never the post-await active provider.src/turn_state/store.py: one SQLite DB (checkpoints + ledger + leases + tombstones, atomic transitions), content-addressed blobs, three-part fencing (generation, revision, lease token — round-3 fix: wire ghost tools and correct registry definitions #3), heartbeats never advancelast_progress_ator the revision, UTC-persisted recovery deadlines (round-3 fix: add Pydantic validators for numeric config bounds #5), boot sweep (stale ACTIVE → SUSPENDED, in-flight ops → OUTCOME_UNKNOWN), three TTL clocks with OUTCOME_UNKNOWN never expiring, terminal payload compaction. Fail-closed: init failure disables the feature loudly; mid-turn write failure raises and halts the turn (round-3 fix: SSRF URL validation + trajectory path traversal hardening #4 — proven by integration test: the second batch never starts after the store dies)._ChatTurnfield explicitly PERSISTED or RECONSTRUCTED with the reason;tests/test_turn_checkpoint_codec.pycensus-pins completeness so a new field fails until classified — resume can never silently re-arm the one-shot guard budgets (both findings from the design round, pinned forever). Stuck-tracker export/import, full trajectory with an iteration-revision guard (no double-append), image blocks blob-externalized.src/discord/turn_resume.py. Explicit:resume/continuefrom the original requester rides the intake pipeline; the trigger never enters the frozen transcript. Auto: in-process waiter (restart ⇒ explicit-only); it actively claims the breaker probe slot when the cooldown elapses — the resumed generation is the capacity probe, and a re-suspension re-arms the waiter with the breaker's escalating cooldown as pacing; stands down fail-safe if the session advanced or can't be read. Admission: re-fetch, same author, unchanged digest, deleted/edited ⇒ TERMINAL_REJECTED; tools re-derived from the CURRENT catalog+permissions (current policy wins). Replay: unmatched blocks repaired from the ledger (APPLIED replays its stored result; unknown/never-ran stated truthfully); nothing is ever re-executed. The resumed generation carries only its REMAINING budget from the persisted deadline.llm_recovery:andturn_state:(defaults active; NOT added to the tracked config.yml — six sections already live schema-only, so the deploy needs no skip-worktree dance). Housekeeping runs the three-clock sweep.shutdown_servicescloses the store.Deliberate characterization-pin amendments (called out per round-3 #7)
test_agent_lifecycle,test_recovery,test_agent_trajectory): EXECUTING→RECOVERING→EXECUTING is gone;recovery_attemptsstays 0 (field kept for API shape). Lifetime-wins-over-exception-class newly pinned.characterization/test_chat_tool_loop): the hardcoded single retry became the shared recovery; the(circuit breaker recovery failed)wording is gone._LoopTurncheckpointing in v1.v1 fences (from the approved plan)
Ledger boundary is
_run_one_toolONLY (round-3 #1); reconcilers ship as the conservative default (no receipt ⇒ OUTCOME_UNKNOWN/never-rerun;RECONCILED_*states exist in the schema); checkpointing covers Discord chat turns only (web shims excluded,_LoopTurnlater); 401/429 handling, ollama/kimi retry loops, the empty-200 asymmetry and the dead exhaustion line inopenai_codex.pyuntouched.Gates
Suite 7,866 passed / 5 skipped (+~75 net). ruff clean. lint gate new=0. type gate new=0 (the 2 pre-existing browser.py findings are in the merge-base too). Coverage findings=0, total 88.2% (was 87.7); baseline regenerated with all new files ratcheted, 26 files improved, 0 decreased.
Known operational notes
data/turn_state/(rides the existing data backup).resume(by design).