From 58485f05c565f87eb1a1ba418902b227c3a21e05 Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Wed, 15 Jul 2026 10:40:03 +0200 Subject: [PATCH] docs: separate implemented design from proposals --- AGENTS.md | 12 +- README.md | 14 +- design/decisions.md | 97 ++-- design/designs/documentation-site.md | 54 +-- design/designs/improvement-proposals.md | 441 ------------------ .../designs/long-running-loop-reliability.md | 387 +++++++++++++++ .../success-semantics-and-evaluation.md | 8 +- design/proposals/improvement-proposals.md | 193 ++++++++ docs/http-contract.md | 39 +- docs/session-layout.md | 92 ++-- skills/loopy-loop/SKILL.md | 37 +- src/loopy_loop/config.py | 2 +- src/loopy_loop/coordinator_app.py | 10 +- src/loopy_loop/models.py | 5 +- src/loopy_loop/recovery.py | 28 +- src/tests/test_worker_liveness_recovery.py | 7 +- website/src/app/docs/cli-reference/page.mdx | 32 +- website/src/app/docs/concepts/page.mdx | 8 +- website/src/app/docs/configuration/page.mdx | 17 +- website/src/app/docs/getting-started/page.mdx | 8 +- website/src/app/docs/http-contract/page.mdx | 99 ++-- website/src/app/docs/page.mdx | 2 +- website/src/app/docs/session-layout/page.mdx | 68 +-- website/src/app/docs/troubleshooting/page.mdx | 19 +- 24 files changed, 976 insertions(+), 703 deletions(-) delete mode 100644 design/designs/improvement-proposals.md create mode 100644 design/designs/long-running-loop-reliability.md create mode 100644 design/proposals/improvement-proposals.md diff --git a/AGENTS.md b/AGENTS.md index 69e22ae..79f37fb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,9 +54,8 @@ involvement is a last resort, never a normal step. run stops — state exactly what was missing and what was tried. - **Do NOT build or assume a preferred human gate.** No `paused` / `waiting_for_human` state, no `gate_request.json`, no external-action approval flow. That was considered and - rejected (D5; `design/designs/improvement-proposals.md` P0.2). Do not pause a run to ask - a question when you could either solve it autonomously or stop cleanly with - `unresolvable_error`. + rejected by D5. Do not pause a run to ask a question when you could either solve it + autonomously or stop cleanly with `unresolvable_error`. ## Rule 3 — Design and decision docs must be understandable cold, by future agents AND humans @@ -72,9 +71,10 @@ specialist. Write for them. companion design section must be self-contained. - **Anchor claims in the code.** Reference the file/function (e.g. `coordinator_app.py`, `harness_runner.py`) so a reader can verify — but lead with the plain-English meaning. -- **Keep `design/` honest about status.** `design/designs/` is binding; `design/analysis/` - is working notes (may be superseded); `improvement-proposals.md` is *proposed, not - decided*. Don't cite a proposal as if it were a decision. +- **Keep `design/` honest about status.** `design/designs/` is binding; + `design/proposals/` is unimplemented, non-binding work under consideration; and + `design/analysis/` is working notes (may be superseded). Don't cite a proposal as if + it were a decision or a description of current behavior. When in doubt on any rule, favor the version a stranger could read cold and fully understand. diff --git a/README.md b/README.md index fccb837..e5ab451 100644 --- a/README.md +++ b/README.md @@ -262,10 +262,11 @@ Important rules: team-harness API/network errors. - `recovery_policy` (`drain` by default, or `reap`) and `recovery_drain_timeout_s` control what crash recovery does with agent - processes a dead worker left running: drain lets them finish (one shared - bounded deadline) before the iteration is re-run; reap kills them - immediately. These are coordinator-side settings and are not part of the - config snapshot sent to the worker. + processes left by an interrupted worker task: drain lets them finish within + one shared bounded deadline; reap kills them immediately. The interrupted + task is recorded as abandoned, consumes a turn, and then normal scheduling + continues only if no stop condition fires. These are coordinator-side + settings and are not part of the config snapshot sent to the worker. - `workflow_consecutive_failures_cap` (default 5) is a per-workflow circuit breaker: that many consecutive failed iterations of the same workflow stop the loop with `stop_reason="workflow_failure_cap"` instead of retrying a @@ -429,8 +430,9 @@ consecutive failed iterations of any single workflow stop the loop with `stop_reason="workflow_failure_cap"` after `workflow_consecutive_failures_cap`. Failed iterations record a `failure_kind` in history — `transient` (provider said retry; team-harness's own retries were exhausted), `deterministic` -(auth/config errors retries cannot fix), `crash` (worker died mid-iteration), -or `unknown` — so a stopped run is legible without reading harness logs. +(auth/config errors retries cannot fix), `crash` (the task was abandoned by +worker-crash recovery; this does not prove an unverifiable worker died), or +`unknown` — so a stopped run is legible without reading harness logs. ## Workflow Sets and Child Sessions diff --git a/design/decisions.md b/design/decisions.md index 09b46e4..4cf772c 100644 --- a/design/decisions.md +++ b/design/decisions.md @@ -7,8 +7,11 @@ the conversation that produced them** — needs to understand each one cold. Companion docs: - `design/designs/success-semantics-and-evaluation.md` — the self-contained, detailed form of **D3** and **D4** (this log states the conclusion; that doc makes it complete). -- `design/designs/improvement-proposals.md` — forward-looking changes we are *considering* - (not decided; do not treat as binding). +- `design/designs/long-running-loop-reliability.md` — the self-contained description + of the crash recovery, telemetry, failure containment, and model-tier mechanisms + already implemented under several decisions below. +- `design/proposals/` — forward-looking changes we are *considering* (not decided; + do not treat them as binding or as descriptions of current behavior). - `design/analysis/` — the July 2026 review that produced several of these decisions (working notes; may be messy or superseded). - `README.md` — the user-facing behavior these decisions implement. @@ -46,7 +49,9 @@ This is the premise the rest of the log depends on. **Decision.** The coordinator drives exactly **one** worker at a time, over a two-endpoint ping-pong API (`POST /register`, `POST /finished`). v0.2.0 deliberately -removed leases, polling, and worker identity to get here. Several recovery paths in +removed leases, polling, and the multi-worker registry/scheduling model to get here. +v0.3.0 later added process identity solely to prove ownership and liveness during +crash recovery; it did not restore a worker pool. Several recovery paths in `coordinator_app.py` are correct **only** under this single-worker assumption, and say so in comments. @@ -57,9 +62,10 @@ parallelism already exists one layer down, inside `team-harness` (a coordinator spawn several worker CLIs at once). **Consequences.** Do **not** reintroduce parallel loopy workers as a "scaling" feature; -it is a redesign of the state machine, not a flag, and it is explicitly deferred (see -`improvement-proposals.md`). Recovery logic may rely on "at most one task is live." -Child sessions are depth-first, one at a time (see D6). +it is a redesign of the state machine, not a flag. The deliberate boundaries in +`designs/long-running-loop-reliability.md` record it as rejected, not pending work. +Recovery logic may rely on "at most one task is live." Child sessions are depth-first, +one at a time (see D6). ## D3. Iteration success means "the assignment ran," not "the work is good" @@ -137,9 +143,9 @@ only when a blocker is genuinely unavoidable without a human. When they do stop, recorded blocker must be specific enough that a human reading it later understands exactly what was missing and what was tried, because that report is the entire human-facing surface. Do **not** add a `paused` / `waiting_for_human` state, a -`gate_request.json` flow, or an external-action approval gate; that proposal was -considered and rejected (see `improvement-proposals.md` P0.2). This decision is the -mechanism `AGENTS.md` and `CLAUDE.md` point agents at. +`gate_request.json` flow, or an external-action approval gate; that alternative was +considered and rejected. This entry is the canonical disposition and the mechanism +`AGENTS.md` and `CLAUDE.md` point agents at. ## D6. Large multi-phase targets are driven by the planner/dispatcher double loop from the start @@ -157,13 +163,12 @@ the start avoids re-architecting mid-project. **Consequences.** The parent/child machinery is on the critical path from day one, so it must be hardened *first* — durable active-child crash recovery and a PM template that is -runnable from a clean init are prerequisites, not later polish (see -`improvement-proposals.md` P0). Note that this recovery is of session/child *state* from -files, not of running agent processes: re-adopting a crashed worker's agent subprocesses -is not feasible, so a hard worker crash is handled by cleanup, not adoption -(`improvement-proposals.md` P2.5). Child sessions remain depth-first and one-at-a-time -(consistent with D2). The planner drives the target's *own* authoritative plan; it does -not invent a parallel backlog. +runnable from a clean init are prerequisites, not later polish. Both are implemented and +described in `designs/long-running-loop-reliability.md`. Session-stack recovery +reconstructs session/child *state* from files; it does not re-adopt a crashed worker's +agent subprocesses. A hard worker crash is handled by the D7 drain/reap cleanup path. +Child sessions remain depth-first and one-at-a-time (consistent with D2). The planner +drives the target's *own* authoritative plan; it does not invent a parallel backlog. ## D7. Process-lifecycle ownership is split: team-harness owns agent processes, loopy-loop owns the worker @@ -174,35 +179,47 @@ own: provides a durable liveness check plus **drain / reap / ignore** policy operations over those groups. (team-harness `design/decisions.md` TH-D5; `design/designs/process-lifecycle-and-reaping.md`.) -- **loopy-loop owns its own `loopy worker` process.** It records the worker's pid and a - heartbeat in the session directory, and on crash recovery it (a) uses worker liveness to tell - "still running" from "dead" before reclaiming a task — closing the duplicate-work window on a - second `/register` — and (b) applies a policy per orphaned agent for the interrupted run. +- **loopy-loop owns its own `loopy worker` process.** It records the worker's hostname, + pid, and pid-reuse-proof start-time token on the live `CurrentTask`. On crash recovery it + (a) probes a verifiable same-host identity to tell "still running" from "dead" before + reclaiming a task — closing the local duplicate-work window on a second `/register` — and + (b) applies the configured policy to the interrupted run's orphaned agents. There is no + periodic heartbeat; liveness is checked directly against the stored identity. **Default: bounded drain** — let an in-flight agent finish within a timeout (fits loopy's git-is-truth, cost-conscious profile; no concurrent-writer problem because it runs during - recovery), with **reap** as the escape for force-stop / hung-past-timeout / unsafe-to-finish. - A drained iteration is **still re-run**: its `result.json` never existed and is never - synthesized from drained outputs (that would fabricate a result the dead coordinator never - produced — the false-closure trap D3 prevents). The drain's yield is the agents' completed - repo edits (preserved by git, D1) plus an explicit **salvage record** — `salvage.json` in the - interrupted iteration's directory and an `abandoned_after_drain` history code — so the - provenance of the surviving edits is auditable rather than a mystery diff. - -**Context.** loopy-loop runs the harness synchronously inside its worker; the agent CLIs are -children of that worker, not of loopy-loop's coordinator, and loopy-loop tracks no process -identity of its own today. A hard worker crash therefore orphans agent processes that keep -spending money and writing to the checkout, and loopy-loop cannot currently distinguish a live -worker from a dead one. Neither problem is solvable by re-adopting processes (impossible — see -D6 and team-harness TH-D2); both are solvable by *tracking identity, then draining or reaping*, -and the natural split is "each layer owns the processes it spawns." + recovery), with **reap** as the escape for hung-past-timeout or unsafe-to-finish work. A + future force-stop command must reuse this cleanup path rather than inventing another one. + A drained iteration is **never accepted or synthesized from drained outputs**: its + `result.json` never existed, and fabricating one would trigger the false-closure trap D3 + prevents. If stop conditions still allow the session to continue, the scheduler dispatches + another real iteration; a failure cap or `max_turns` may instead stop it. Recovery preserves + completed repo edits through git (D1). When it processes at least one tracked harness run, + it also writes `salvage.json` in the interrupted iteration's directory. When at least one + orphan reaches a settled outcome, history uses `abandoned_after_` instead of plain + `abandoned`. These records make the provenance of surviving edits auditable rather than a + mystery diff. + +**Context.** Before v0.3.0, loopy-loop ran the harness synchronously inside its worker +without persisting process identity. The agent CLIs are children of that worker, not of +loopy-loop's coordinator, so a hard worker crash could orphan processes that kept spending +money and writing to the checkout while the coordinator could not distinguish a live worker +from a dead one. Neither problem is solvable by re-adopting processes (impossible — see D6 +and team-harness TH-D2). For a verifiable same-host identity, both are solved by *tracking +identity, then draining or reaping*, with the natural split "each layer owns the processes it +spawns." A remote worker remains unverifiable and falls back to legacy abandonment because +its processes cannot be reached from the coordinator host. A missing start-time token or +identity provider prevents the worker-liveness proof; same-host agent recovery is still +attempted when team-harness has usable run records, and otherwise degrades to legacy +abandonment. Those fallback paths cannot prove the old writer is gone. **Consequences.** The process-lifecycle mechanism (liveness + drain/reap/ignore) is a team-harness feature (we own it — it is not a `/proc`-scraping hack bolted onto loopy-loop); -loopy-loop consumes it. loopy-loop's own new work is small: persist the worker pid + heartbeat, -and apply a recovery policy per orphan (default bounded drain, reap as escape). This makes D6's -state-recovery *safe* (verify-dead-before-reclaim) rather than optimistic. See -`improvement-proposals.md` P0.1 (worker liveness) and P2.5 (recovery policy), and team-harness -TH-D5. +loopy-loop consumes it. loopy-loop persists the worker's hostname/pid/start-time identity and +applies one configured recovery policy to the interrupted run's orphans (default bounded +drain, reap as escape). For identity-tracked same-host runs this makes D6's state recovery +verify-dead-before-reclaim rather than optimistic; legacy and remote identities retain the +documented limitation above. The implemented protocol and salvage boundary are described in +`designs/long-running-loop-reliability.md`; the process-group mechanism is team-harness TH-D5. ## D8. Constraints on agents are fail-closed detection with a repair path, never hard prevention diff --git a/design/designs/documentation-site.md b/design/designs/documentation-site.md index 1b12be6..37043fa 100644 --- a/design/designs/documentation-site.md +++ b/design/designs/documentation-site.md @@ -1,14 +1,14 @@ # Design: Public Documentation Site -**Status:** Proposed (plan for a new deliverable; nothing built yet) +**Status:** Accepted and implemented (`website/` plus `.github/workflows/docs-deploy.yml`) **Date recorded:** 2026-07-12 -**Applies to:** a new, self-contained documentation site for `loopy-loop`, to live -in this repository and be published on a `writeit.ai` subdomain. +**Applies to:** the self-contained `loopy-loop` documentation site in `website/` +and its GitHub Pages deployment target on a `writeit.ai` subdomain. -This document records the plan and the decisions behind it, so that when the site is -built the choices are already argued and a later reader understands *why* the stack -looks the way it does. It is written before implementation on purpose: the "what to -build" is settled here first. +This document was written before implementation and now records the decisions behind +the shipped site, so a later reader understands *why* the stack looks the way it does. +The implementation landed in `website/`; its build and deployment workflow lives at +`.github/workflows/docs-deploy.yml`. The shared principle: @@ -40,7 +40,7 @@ The shared principle: Three sibling frontends were reviewed before choosing an approach: - **`orchestra/public_and_admin_app/fe`** (the `247agents.io` site) — **the model we - will follow.** It is a Next.js 16 App-Router site whose docs are native + followed.** It is a Next.js 16 App-Router site whose docs are native [`@next/mdx`](https://nextjs.org/docs/app/building-your-application/configuring/mdx) pages: each `src/app/(public)/docs/**/page.mdx` file *is* a route. It uses Tailwind v4 + shadcn/ui + `@tailwindcss/typography` (`prose`), `rehype-pretty-code` (Shiki) @@ -213,9 +213,9 @@ These become the shadcn/Tailwind CSS variables (`--background`, `--foreground`, - **Font:** `writeit.ai` uses **proxima-nova**, a paid Adobe Typekit font served from a domain-locked kit. It cannot ship in a self-hostable OSS module and will not render off `writeit.ai` domains. We therefore match the *colours* exactly and pick a close - open substitute — **Hanken Grotesk** or **Figtree** (both load via `next/font`, - self-host cleanly, and are visually adjacent to proxima-nova). This keeps the module - truly self-hostable while preserving brand feel. + open substitute. The implementation chose **Hanken Grotesk**, loaded through + `next/font`; it self-hosts cleanly and is visually adjacent to proxima-nova. This + keeps the module truly self-hostable while preserving brand feel. - **Light-first:** `writeit.ai` itself is light-only, so light mode is the faithful match. Dark mode is a cheap fast-follow (`next-themes` + the shadcn dark tokens orchestra already defines) and is deferred, not designed out. @@ -227,20 +227,20 @@ These become the shadcn/Tailwind CSS variables (`--background`, `--foreground`, ### Decision The first version restructures **existing** documentation into MDX pages rather than -writing net-new prose, and reconciles the drifted `skills/loopy-loop/SKILL.md` against -the current API as part of that pass. +writing net-new prose. The same release sequence also reconciled the drifted +`skills/loopy-loop/SKILL.md` against the current API. ### Context / why -The content is roughly 80% already written — the 466-line `README.md`, +The content was roughly 80% already written — the then-466-line `README.md`, `docs/session-layout.md`, `docs/http-contract.md`, the accepted `success-semantics-and-evaluation.md` design doc, the `CHANGELOG`, and `SKILL.md`. -This is primarily a re-homing and structuring job. `SKILL.md` currently references the -older `.loopy_loop/workflows/` layout and pre-0.2.0 polling language; the docs must -follow the README's current two-endpoint / `workflow_sets/` model, and the drift is -worth fixing while the content is being touched. +This was primarily a re-homing and structuring job. Before the implementation, +`SKILL.md` referenced the older `.loopy_loop/workflows/` layout and pre-0.2.0 polling +language. The site and corrected Skill now follow the README's two-endpoint / +`workflow_sets/` model. -### Proposed information architecture +### Implemented information architecture Each entry is one `page.mdx`, sourced from the material in parentheses. @@ -256,7 +256,7 @@ Each entry is one `page.mdx`, sourced from the material in parentheses. | `/docs/session-layout` | `docs/session-layout.md` | | `/docs/http-contract` | `docs/http-contract.md` | | `/docs/child-sessions` | `child_requests` contract, `pm_planner_dispatcher` | -| `/docs/cli-reference` | `init` / `coordinator` / `worker` / `status` / `stop` | +| `/docs/cli-reference` | `init` / `coordinator` / `worker` / `status` / `events` / `stop` | | `/docs/troubleshooting` | `SKILL.md` "Common Pitfalls" | The nav order and grouping live in one hand-maintained array (orchestra's @@ -269,16 +269,16 @@ The nav order and grouping live in one hand-maintained array (orchestra's `Next.js` (App Router) · `@next/mdx` · `output: 'export'` + `trailingSlash: true` · `Tailwind v4` + `shadcn/ui` + `@tailwindcss/typography` · `remark-gfm` + `rehype-slug` + `rehype-pretty-code` (Shiki) · `Pagefind` + `cmdk` for ⌘K search · -WriteIt palette with Hanken Grotesk / Figtree · deployed to GitHub Pages at -`loopy.writeit.ai`. +WriteIt palette with Hanken Grotesk · GitHub Pages deployment target at +`loopy.writeit.ai` (the external Pages/DNS provisioning remains an operational step). --- ## Follow-up (out of scope for this repo's PR) -After the site is live, add a "Docs" link on `writeit.ai` pointing to -`https://loopy.writeit.ai`. That is a separate change in the `writeit` repo and is -tracked as a follow-up, not part of building the site here. +The remaining cross-repository follow-up is to add a "Docs" link on `writeit.ai` +pointing to `https://loopy.writeit.ai`. That change belongs in the `writeit` repo and +is not part of this implementation. --- @@ -290,7 +290,7 @@ tracked as a follow-up, not part of building the site here. | 2. Location | `website/` in this repo | Self-hostable module; docs version with code | | 3. Hosting | GitHub Pages at `loopy.writeit.ai` via CNAME | Hosting stays with the OSS repo; static, no server | | 4. Search | Pagefind + `cmdk` (⌘K) | Static, self-hostable, closes orchestra's search gap | -| 5. Style | WriteIt palette + open font, light-first | Brand-continuous; font stays self-hostable; faithful to a light-only site | +| 5. Style | WriteIt palette + Hanken Grotesk, light-first | Brand-continuous; font stays self-hostable; faithful to a light-only site | | 6. Content | Restructure existing docs into MDX; fix `SKILL.md` drift | Content ~80% written; keep docs on the current API | ### When to revisit @@ -299,4 +299,4 @@ tracked as a follow-up, not part of building the site here. (search + dark mode built in). The content (MDX) ports with little change. - If org ops-consistency outweighs repo-locality, switch Decision 3 to **Firebase Hosting** (same CNAME flow, different deploy step, no code change). -- Add **dark mode** (`next-themes` + existing shadcn dark tokens) once light mode ships. +- **Dark mode** remains an optional follow-up (`next-themes` + shadcn dark tokens). diff --git a/design/designs/improvement-proposals.md b/design/designs/improvement-proposals.md deleted file mode 100644 index c879d84..0000000 --- a/design/designs/improvement-proposals.md +++ /dev/null @@ -1,441 +0,0 @@ -# Design: Improvement Proposals - -**Status:** Living — per-proposal statuses below (last status pass: 2026-07-13) -**Date:** 2026-07-12 -**Relationship to other docs:** Companion to -[`success-semantics-and-evaluation.md`](./success-semantics-and-evaluation.md) -(which records decisions that are *already made* and should not be reverted). This -doc is a tracked ledger of changes we are considering or have since shipped. It is -derived from the July 2026 review under [`../analysis/`](../analysis/), curated and -re-prioritized. - -**Framing decision that drives priority:** we intend to drive a large, multi-phase -target project with the **planner/dispatcher double loop from the very beginning** — -not after a single-loop pilot. That decision moves the parent/child machinery from -"harden it later" to "harden it first," because it is exercised on day one of the -flagship use case. The ordering below reflects that. - -Each proposal has: **what**, **why**, **sketch**, **effort** (S≈1–2 days, M≈1 week, -L≈multi-week), and **status**. The **Status** line is authoritative — read it first. -For proposals marked implemented, the What/Why text is preserved as the *original -problem statement*: it describes the pre-implementation state that motivated the -work, not the current behavior. - ---- - -## Priority 0 — Foundations for the double loop - -These are prerequisites for pointing the double loop at any real, multi-phase target. -Do them first. - -### P0.1 — Durable session-stack / active-child recovery - -**What.** Make "parent suspended on child X; child X active" a durable, -crash-recoverable fact in root state, and make coordinator startup reconstruct and -validate the session stack. - -**Why.** This is the concrete gap that makes the current double loop unsafe. Today, -while a child runs, the only "active" pointer is `CoordinatorService.state_store` -aimed at the child's `state.json`; there is no durable `active_child_session_id` in -root state. After a coordinator crash, a fresh `StateStore` resolves -`latest_top_level_state_path()` → reopens the **parent**, not the running child. The -child-request file that spawned it was already deleted at dispatch, so the child is -orphaned: it may sit marked `running` forever while the parent re-dispatches new -work, and its evidence linkage is lost. **There is no test for "crash while a child -is active."** For a double loop used from day one, this is the first thing to fix. - -**Sketch.** -- Persist the session stack in root state (or a small `runtime.json`): the ordered - chain of suspended parents + the single active session id. -- A state transition atomically means "parent suspended on child X; X active." -- On startup: finalize terminal children, resume their parents; leave non-terminal - children active. `children.json` becomes an index/projection, not the only bridge. -- Add an `attempt_id` (UUID) per dispatch so a legitimate retry is distinguishable - from the original execution (the current key `(session_id, workflow_id, iteration)` - cannot tell them apart). -- Write `children.json`, `result.json`, `pending_finished_request.json`, and control/ - eval artifacts atomically (temp-file + `os.replace`), as `state.json` already is. -- **Worker liveness (makes reclaim *safe*, not optimistic).** Have the `loopy worker` - record its pid + a periodic heartbeat in the session dir. Then a second `/register` - while a `current_task` exists can *verify* whether the worker is actually dead before - reclaiming — closing the duplicate-work window — instead of assuming abandonment. On a - confirmed-dead worker, apply the recovery policy to its orphaned agent processes — by - default **bounded drain**, reap as the escape — before fresh work starts (see P2.5; - cross-repo split in D7). -- **Salvage record for drained iterations.** A drained iteration is still re-run — its - `result.json` never existed (the dead worker would have written it), and we deliberately do - **not** synthesize one from the drained agents' outputs: that would fabricate an iteration - result the coordinator never produced, exactly the false-closure trap D3 exists to prevent. - Instead, make the salvage explicit: write `salvage.json` into the interrupted iteration's - directory (drained agent ids, exit codes, pointers to their harness output dirs, a diffstat - of what they changed in the working tree) and record the iteration in history as - `abandoned_after_drain` (distinct from plain `abandoned`). The working tree preserves the - drained agents' edits (D1 — the next iteration builds on git state); the salvage record - preserves the *provenance*, so an auditor never meets a mystery diff, and the failure - taxonomy (P2.3) can treat "crashed but salvaged" differently from "crashed, lost everything." -- Tests: restart mid-child; restart after child terminal but before parent resume; - double `/register` while a task is live; late `/finished` from an old attempt. - -**Scope — this recovers *state*, not *processes*.** P0.1 makes a restarted coordinator -reconstruct the session/child *state machine* from files. It deliberately does **not** -try to re-adopt any running agent processes, because that is a different problem with a -hard OS ceiling (see P2.5). This is the right split, because the two crash cases differ: -- **Coordinator crash** (what P0.1 covers): the agent CLIs are children of the *worker*, - not the coordinator, so they are unaffected. The worker either finishes and writes its - artifacts (recovered from files), or — if the coordinator is down at `/finished` time — - exits *after* its agents already completed. No orphaned processes arise here; file-based - state recovery is sufficient and correct. -- **Worker crash mid-run** (NOT covered by P0.1): the agent CLIs can orphan and keep - running. That is a process-cleanup problem, handled separately in P2.5 — do not fold it - into P0.1. - -**Effort.** M–L. **Status.** **Implemented in 0.4.0** — `active_child_session_id` -pointer, startup session-stack walk with adoption/finalization, attempt ids, -request-file tombstones, atomic artifact writes. - -### P0.2 — Keep the last-resort human escape hatch; do NOT build a preferred human-gate - -**Design stance.** The goal is **full autonomy**. Human involvement is a *last -resort*, not a step in the normal flow. An earlier draft of this doc proposed a -first-class, resumable "pause and wait for a human to answer a gate" state; that is -**rejected** — it promotes human involvement from last-resort to a preferred path, -which is the opposite of the goal. - -**What already exists (and is enough).** The escape hatch the autonomy model needs is -already wired end to end: -- `ControlSignal.stop_reason` is `Literal["goal_met", "unresolvable_error"]` - (`models.py`). -- A workflow writes `control.json` with `stop_reason: "unresolvable_error"` → - `_apply_session_control` sets the flag (`coordinator_app.py`) → - `_apply_stop_precedence` stops the loop as terminal. -- The stock planner prompt already instructs it: *"If no useful work remains and the - blocker is genuinely terminal, update the Session control path with stop_reason - `unresolvable_error` and record the exact blocker in current_state.md."* - -So the loop can already run unattended and, only when genuinely stuck, stop and say -"I can't finish this without a human." No new state, no `gate_request.json`, no -resume-after-human flow, no external-action approval gate. - -**The only work worth doing here (small, prompt-level, low priority):** -- **Make it a genuine last resort in prompt guidance.** Ensure workflow prompts try - hard before giving up — exhaust re-scoping, retrying with a better child goal, and - routing around the blocker — so `unresolvable_error` fires rarely and only when - truly unavoidable. This is prompt tuning, not code. -- **Make the give-up legible.** When it does stop, the blocker report a human reads - should be clear and specific (the exact decision or capability that was missing, and - what was tried). Today that lives in `current_state.md` / the control `reason`; keep - it high-quality. A structured blocker summary is a nice-to-have, not required. -- **Optional, probably unnecessary:** distinguish a clean "gave up, a human decision - is required" from a crash-style `failed`. Both are terminal; the single - `unresolvable_error` reason with a good message is likely sufficient. Only split it - if operational reports actually need to tell the two apart. - -**Effort.** S (prompt guidance only) — or none. **Status.** **Resolved by decision -(D5), no code** — do not build the pause/resume gate. The autonomy escape hatch -already exists; only optional prompt/report polish remains. - -### P0.3 — Per-child budgets *(withdrawn)* - -**Withdrawn.** Considered and dropped. The idea was to let a `ChildSessionRequest` -carry its own `max_turns`/cost/model-profile/retry policy instead of inheriting the -parent's root config. Rejected as not important enough to justify the trouble it -brings: it expands the child-request schema and the config-resolution surface, adds -more ways for a misconfigured child to behave surprisingly, and the simpler default — -a child inherits root config, bounded by the root `max_turns` — is adequate. If a -concrete need for a smaller/different child budget actually shows up, revisit then; -until it does, keeping child config simple is the better default. (The one small, -unrelated piece of hygiene worth keeping in mind separately: give an invalid -child-request file a terminal rejection record rather than leaving it silently in the -directory — fold that into P0.1's atomic-artifact work if convenient.) - -### P0.4 — Make the PM template runnable from a clean init - -**What.** `loopy init --template pm_planner_dispatcher` must produce a repo that can -actually run — today it copies only `planner/` and `dispatcher/`, but the dispatcher -spawns child sessions running `inner_outer_eval`, which is **absent** from a clean -init. - -**Why.** The double loop's child set is a hard dependency of the parent template. -Right now a fresh init of the PM template cannot execute a single child. The preflight -test only checks the parent, so this is invisible in CI. - -**Sketch.** Either bundle the `inner_outer_eval` (or a purpose-built child) set into -the PM template, or make workflow-set dependencies declarative and installable -(`requires_workflow_sets: [...]`) with preflight validation and a clean-install -template smoke test in CI. - -**Effort.** S–M. **Status.** **Implemented in 0.4.0** — the PM template ships its -child workflow set via packaged-template extra sources, with an end-to-end -clean-init smoke test. - ---- - -## Priority 1 — Legibility and safety - -### P1.1 — `events.jsonl` + usage/cost ledger, then `loopy status --watch` - -**What.** Fill the reserved-but-empty `events.jsonl` with one versioned event per -significant transition, capture per-iteration token/cost/duration, aggregate into -`session.json`, and add a `max_cost_usd` stop condition. Then build a modest live -view on top (`loopy status --watch`, `loopy events --follow --json`). - -**Why.** `events.jsonl` is documented as reserved but nothing writes to it. For a -long-horizon double loop the only way to see what's happening today is to tail files -by hand, and there is no record of what a session *cost*. Build the canonical event/ -cost stream before any dashboard, so operations, debugging, budgets, and any future -TUI all read one truth rather than inventing separate ones. A local TUI (rich is -already a dependency) fits the solo-maintainer deployment; a web dashboard is not -justified yet. - -**Sketch.** Events carry `event_id`, `schema_version`, UTC ts, root/child session ids, -task + attempt ids, a small typed payload; appended under the session lock; tolerate a -truncated final line. team-harness already records coordinator usage per turn — add a -usage adapter with explicit `known`/`unknown` fields (don't pretend worker token -costs are always measurable). Budgets enforceable at session/child/workflow/wall-clock -levels, checked before dispatch and after each result. - -**Effort.** M–L. **Status.** **Implemented (unreleased)** — versioned events -appended after each committing mutation (best-effort delivery by design — -state.json stays the durable truth; torn-tail-tolerant reader); worker-read token usage from team-harness `run.json` with explicit -unknown-usage accounting; durable per-session ledger + children.json roll-up; -`model_prices`-derived cost with the `max_cost_usd` stop condition -(coordinator-side, requires prices — preflight-enforced); `loopy status` -(session stack + usage + cost, `--watch`) and `loopy events [--follow] -[--json]`. Scope notes: budget enforcement is per session subtree (a running -child enforces against its own spend until finalized — bounded by the -two-level depth limit); only session-level budgets shipped (per-workflow and -wall-clock budgets were not needed yet). - -### P1.2 — Deterministic backstop under the judge (per target, for high-stakes work) - -**What.** For target repos that own a trustworthy contract-test suite, add an -eval-banana **deterministic check that shells out to the repo's own commands** -(`uv run pytest`, `import-linter`, `alembic upgrade`, `make test`) as a hard gate -*underneath* the LLM judge — and have the eval runner parse `report.json` to derive -`goal_check.json` itself rather than trusting an agent to paraphrase console output. - -**Why.** This is the one change that removes the single-point-of-failure noted in the -success-semantics doc, **without reverting either deliberate decision** there. It does -not reintroduce the agent-authoring problem, because the checks are repo-owned, not -agent-invented. Keep the LLM judge for the qualitative "did this achieve the -outcome"; add the deterministic suite as the objective floor so a mis-judged -`goal_met` cannot pass a red suite. - -> **Boundary (from the success-semantics doc):** do **not** globally drop the stock -> `inner_outer_eval` "deterministic forbidden" rule. That rule is correct for generic -> repos where the only deterministic checks would be agent-authored. This proposal -> applies **only** to targets with a real suite, via a dedicated child workflow set -> that overrides the stock policy. - -**Effort.** S–M (mostly the child workflow set + runner-side report parsing). -**Status.** **Deferred (2026-07-13)** — out of scope for the pre-target milestone. -It only pays off against a target repo that owns a trustworthy suite, so build it -as part of that target's dedicated child workflow set when the double loop is -pointed at one, not generically now. The boundary above still binds when it lands. - -### P1.3 — Fix the Agent Skill and stale "multi-worker" claims - -**What.** Rewrite `skills/loopy-loop/SKILL.md` (and any README lines) that describe -the removed pre-0.2.0 API, and test the skill against a clean generated repo. - -**Why.** The skill still documents top-level `.loopy_loop/state.json`, "one or more -blocking workers" polling over HTTP, inline `goal`, and the removed -`.loopy_loop/workflows//` layout. v0.2.0 made state session-local, made the worker -a single ping-pong worker (not polling), and stopped loading the old workflows layout. -A skill that confidently generates the *old* setup is worse than no skill — it teaches -agents to build broken configurations. This is a pre-launch blocker. Similarly, the -README's "one or more workers" wording contradicts the single-worker model the -coordinator is explicitly safe under. - -**Effort.** S. **Status.** Proposed. - ---- - -## Priority 2 — Hardening and DX - -### P2.1 — Named model/execution profiles; pin the trio; declare eval-banana - -**What.** Replace hardcoded model IDs (repeated across `config.py`, `cli.py`, packaged -YAML, README, `.team-harness/config.toml`, `.eval-banana/config.toml`, and sibling -repos) with named profiles (`balanced`, `high_assurance`, `cheap`) resolved once in a -project-local file. Consider bounding the `team-harness` dependency once its contract -churns independently of loopy releases. ~~Make eval-banana a declared extra -(`loopy-loop[eval]`) or fail preflight with a clear message~~ — superseded, see Status. - -**Why.** Model IDs churn (git history is full of "bump gpt-5.4 → 5.5"); duplicating a -model string across N files and sibling repos guarantees drift and a launch-day -footgun (install today, hit a decommissioned model). team-harness has already shipped -breaking pre-1.0 changes, so an unbounded `>=` dependency is a standing risk. -eval-banana was required by the recommended template without being a dependency, so a -template could require a tool that wasn't installed. - -**Sketch.** One resolved profile → exact provider/model/effort, snapshotted into the -session. `loopy doctor` validates the profile against the actually-installed CLIs. -Consider a small compatibility matrix (loopy-loop × team-harness × eval-banana) tested -in CI: "treat the trio as one release train" until the contracts have survived real -use. - -**Effort.** M. **Status.** **Partially resolved (2026-07-13)** — the eval-banana -piece landed the simple way: it is now a plain hard dependency (chosen over the -optional-extra design; one less install mode to document), and the worker appends -the scripts directory that actually holds the installed `eval-banana` script -(derived from the package's install record, `sysconfig` as fallback) to `PATH` so -the CLI resolves for spawned agents across supported install modes (`uv tool -install` and pipx expose only the primary package's entry points). - -The named-profiles piece landed in a deliberately smaller form (2026-07-14): -**named model tiers** (`model_tiers` + `default_tier` in the root config, see -D9). Tiers address the drift concern *within a project's config* — a repo that -opts in declares each model id once and its workflow prompts refer to tier -names only — but they scope the vocabulary to *worker* models chosen per -spawn, not full per-session execution profiles (provider/API/prices), because -coordinators stay uniformly strong by policy (D9) and so never need -per-session differentiation. The other half of the original concern — the -model ids hardcoded in loopy's own shipped defaults (`config.py`/`cli.py` -constants, packaged template YAML, README examples) — is NOT resolved: the -stock templates still carry explicit ids and only include tiers as a commented -example. If per-session coordinator profiles ever become a real need, they -compose with tiers rather than replacing them. Trio pinning remains -**Proposed**. - -### P2.2 — `_advance()` refactor, folded into the P0.1 state-machine work - -**What.** Extract the repeated "apply stop → maybe dispatch child → choose next → set -`current_task` → build run response" block (currently duplicated three times across -`register_worker()` and both branches of `finish_assignment()`) into a single -`_advance(state, cause, now) -> TaskResponse`. - -**Why.** It's the one spot in `coordinator_app.py` (~706 lines) where drift will cause -a real bug, because the three copies must stay behaviorally identical. But do it *with* -P0.1, not as a cosmetic pass first — encode the new session-stack invariants and -table-driven transition tests at the same time, or the refactor just centralizes the -current ambiguity. - -**Effort.** S (as part of P0.1). **Status.** **Implemented in 0.4.0** — folded into -P0.1 as `_advance()`. - -### P2.3 — Richer failure taxonomy + per-workflow failure cap - -**What.** Distinguish transient (retryable: coordinator 429/5xx, worker API/network) -from deterministic (a prompt that always fails) failures, each with retryability, -backoff, max attempts, and whether it consumes the workflow's turn budget. Add a -consecutive-failure circuit breaker per workflow (analogous to -`goal_check_consecutive_failures_cap`) → `stop_reason: "workflow_failure_cap"`. - -**Why.** Failure currently collapses to a few strings, and a generic harness exception -becomes a failed iteration the scheduler can retry until `max_turns` — spending money -without distinguishing a transient 429 from a deterministic test failure. A wedged -`inner` shouldn't be able to burn the whole turn budget. - -**Effort.** S–M. **Status.** **Implemented (unreleased)** — `failure_kind` -(`transient`/`deterministic`/`crash`/`unknown`, derived from team-harness's -structured `retryable` signal) recorded on results, `/finished`, and history; -per-workflow consecutive-failure counter in `LoopState` with coordinator-side -`workflow_consecutive_failures_cap` (default 5) stopping the loop with -`stop_reason="workflow_failure_cap"`. Retry/backoff at the loopy level was -deliberately NOT added: team-harness already retries transient API errors with -backoff (`team_harness_max_retries` etc.); a second retry layer would multiply -delays without new information. - -### P2.4 — Operator experience: `doctor`, `validate`, session-aware `status`/`stop` - -**What.** -- `loopy doctor`: Python/package versions, configured worker binaries + auth, - eval-banana availability, git status, writable session dir, port availability, - config/profile validity. -- `loopy validate`: workflow-graph referential integrity (`must_follow` / - `run_after_successes` already checked at preflight — surface it standalone), - template dependencies, signal schemas, required external tools — without starting a - session. -- Make `status`/`stop` session-stack aware (today `status` can show the suspended - parent while a child runs; `stop` sets `stop_requested` on the parent and only takes - effect after the child terminates). Add cooperative `pause`/`resume` and a - `stop --force` that records an outcome. - -**Why.** The current CLI can silently operate the wrong session when a child is active, -and cannot interrupt a running harness except between iterations. For multi-day -double-loop runs this is a real safety and ergonomics gap. - -**Effort.** M. **Status.** Proposed. - -### P2.5 — Orphan agent-process cleanup on restart (team-harness level) - -**What.** On a hard *worker* crash mid-run (Ctrl-C, OOM, SIGKILL, panic), the agent CLI -subprocesses (codex/claude/gemini) it spawned can orphan and keep running — still -spending money, still writing to the target checkout — with nothing tracking them. -Prevent and clean up those orphans. - -**Why.** Neither loopy-loop nor team-harness persists agent PIDs. loopy-loop runs the -harness synchronously and tracks zero processes; team-harness holds the subprocess -handle only **in memory** (`AgentManager._agents`), and spawns with no -`start_new_session`/process-group, so a crashed worker's children are reparented to init -and run on unsupervised. There is no startup reaping anywhere. **Re-adopting a running -orphan is not portably possible** — process control is tied to the dead parent's asyncio -transport; a new process cannot reattach and re-drive it. So the achievable fix is -*prevent + clean up*, not adopt. - -**We own team-harness, so this is a designed feature there, not a `/proc`-scraping hack -here.** The full design is written up in team-harness -`design/designs/process-lifecycle-and-reaping.md` (decision TH-D5); loopy-loop's D7 records -the ownership split. Summary of the split: -- **team-harness side (the bulk):** spawn each worker with `start_new_session=True` (own - process group), persist `pid`/`pgid`/`starttime` into its existing per-run worker-session - manifest, and expose a durable liveness check plus **drain / reap / ignore** policy ops - (`reap_run(manifest, policy=..., drain_timeout_s=...)` / `th reap`) — verifying `starttime` - so a recycled id is never killed. -- **loopy-loop side (small):** on crash recovery, pick a policy per orphan for the interrupted - run before starting fresh. Default to **bounded drain** — let an in-flight agent finish within - a timeout, write the salvage record (P0.1), then re-run the iteration from that completed - state; this fits loopy's cost-conscious, git-is-truth profile (D1), avoids wasting - near-complete API spend and half-applied edits, and the "two writers" objection is moot - because draining happens during recovery before any new work is dispatched. Drain yields a - finalized worker record + preserved repo edits — never a synthesized `result.json` (see the - P0.1 salvage bullet). **Reap** (kill, then re-run) is the escape: for `stop --force` (P2.4), - a hung orphan past the drain timeout, or a crash cause that makes finishing unsafe. - team-harness provides the liveness check + policy ops + timeout (TH-D5); the logical-session - resume path is a separate, larger option and is **not** proposed here. - -**Effort.** M (team-harness, tracked there) + S (loopy surfacing). **Status.** -**Implemented** — team-harness 0.3.0 (TH-D5: process-group identity + `th reap`) -plus loopy-loop 0.3.0 (worker identity on `/register`, bounded-drain recovery of -orphaned agents, `salvage.json`). - ---- - -## Explicitly deferred (not now) - -- **Parallel loopy workers.** Both review passes independently argued against the - obvious "scale = more workers" answer. Parallelism already lives inside team-harness; - two loopy assignments editing one checkout buy nondeterministic corruption and merge - conflicts, not throughput. Revisit only behind worktree/branch isolation with a merge - coordinator, and only once tasks are explicitly read-only or isolated. The early work - of a large multi-phase target is typically highly serial anyway. -- **Web dashboard.** A local TUI over `events.jsonl` (P1.1) covers the solo-maintainer - deployment without another service + auth surface. -- **Breadth-first / arbitrary-depth child sessions.** Depth-first, one-child-at-a-time - is the right v1 and matches a shared checkout. Harden the single-child path (P0.1) - before adding breadth. - ---- - -## Suggested sequencing - -1. ~~**P0.1 + P2.2 together**~~ — **done (0.4.0)**: session-stack/state-machine - hardening plus the `_advance()` refactor. (P0.2 needed no code — the autonomy - escape hatch already exists.) -2. ~~**P0.4**~~ — **done (0.4.0)**: PM template runnable from a clean init; the - double loop is executable end to end. (P2.5 also **done**: team-harness 0.3.0 + - loopy-loop 0.3.0.) -3. **P1.3** — fix the Agent Skill and stale claims (now also stale against - 0.3.0/0.4.0: worker identity, attempt ids, workflow sets). -4. **P2.3** — failure taxonomy + per-workflow failure cap. -5. **P1.1** — events/cost ledger + `status --watch`, so double-loop runs are legible. -6. **P2.1 remainder, P2.4** — profiles/pins and operator UX (rolling). **P1.2** is - deferred into the flagship target's own workflow-set design. - -After (3)–(5), the double loop is ready to drive a large, multi-phase target as a -sequence of narrow work packages that runs unattended — falling back to the -`unresolvable_error` escape hatch only when a blocker is genuinely unresolvable -without a human. Deterministic backstops (P1.2) arrive with that target's own -workflow-set design. That target-execution design is the subject of a separate -doc, not this one. diff --git a/design/designs/long-running-loop-reliability.md b/design/designs/long-running-loop-reliability.md new file mode 100644 index 0000000..d450ce1 --- /dev/null +++ b/design/designs/long-running-loop-reliability.md @@ -0,0 +1,387 @@ +# Design: Reliability and Operations for Long-Running Loops + +**Status:** Accepted and implemented in releases 0.3.0–0.6.0 +**Date consolidated:** 2026-07-15 +**Applies to:** the coordinator/worker runtime, planner/dispatcher child sessions, +crash recovery, operational telemetry, failure containment, and model selection. + +This document describes current behavior. It consolidates the shipped parts of +the July 2026 improvement review, whose historical identifiers (P0.1, P1.1, +and so on) still appear in code comments, tests, and the changelog. The active +work that did not ship is kept separately in +[`../proposals/improvement-proposals.md`](../proposals/improvement-proposals.md). + +The design is driven by three existing decisions: + +- files and git are the durable source of truth (D1); +- exactly one loopy worker owns an assignment at a time (D2); and +- evaluation artifacts decide work quality; transport and recovery code must + not manufacture semantic success (D3/D4). + +Together, the mechanisms below let the planner/dispatcher double loop run for a +long time without losing its place, duplicating verifiably live local work after +a crash, or hiding cost and failure state from an operator. Remote or otherwise +unverifiable worker identity has a narrower guarantee, documented below. + +--- + +## Durable parent→child session state + +### The parent records which child suspends it + +When a top-level workflow dispatches a child, the parent's `LoopState` records +`active_child_session_id`. The child records `parent_session_id`, while +`children.json` is an audit index of every dispatched child and its outcome. The +pointer in `state.json`, not the presence of a request file, determines which +session is active. + +On `--resume`, `CoordinatorService._reconstruct_session_stack()` in +`src/loopy_loop/coordinator_app.py` walks parent→child pointers to the deepest +live session. It finalizes a child already terminal on disk and resumes its +parent. It also repairs interrupted handoffs: a dangling pointer is cleared, +and a fully-created running child whose final parent-pointer write did not land +can be adopted from `children.json` by `_adoptable_child_id()`. + +This solves a specific crash case. Before the pointer existed, restarting while +a child ran reopened the latest top-level session; the child stayed marked +running but became unreachable, and the parent could dispatch duplicate work. +Now the relationship is reconstructable from files alone. + +`src/tests/test_session_stack_recovery.py` covers restart during a child, +restart after child termination but before parent resume, dangling-pointer +repair, child adoption, and terminal-child finalization. + +### The handoff is staged and recoverable, not a multi-file transaction + +The filesystem cannot atomically commit parent `state.json`, child `state.json`, +`children.json`, and the consumed request as one unit. Instead, +`_dispatch_child_session_if_requested()` writes them in a discoverable order and +startup reconciliation handles every crash window. + +The originating request filename is stored on the child record. While that +record is running, `_dispatched_request_files()` treats a leftover request file +as a tombstone, so a crash between recording and unlinking cannot dispatch it +twice. Invalid or unusable child requests are renamed to `*.json.rejected` with +the reason logged; the rejected file itself does not gain a structured reason. +They do not remain in the scan loop forever. A completed child's old filename +may be reused for genuinely new work. + +The implementation intentionally supports a depth-first, two-level tree: only a +top-level session dispatches one child at a time. This matches the single shared +checkout and D2/D6; it is not an accidental recursion limit. + +### One transition function enforces the suspended-parent invariant + +`CoordinatorService._advance()` is the common transition used after register, +normal completion, and recovered completion. Its fixed order is: + +1. if the state is a suspended parent, replay the live child's task or finalize + a terminal child and continue on the parent; +2. apply session control and stop precedence; +3. dispatch a pending child request after a successful parent workflow; +4. choose the next eligible workflow; +5. stamp and persist the new `CurrentTask`; and +6. return the corresponding run/stop response. + +Centralizing this sequence prevents formerly duplicated register/finish paths +from drifting. In particular, a parent with a live child cannot acquire its own +task; a duplicate completion retry receives the child's live task instead. + +--- + +## Dispatch identity, ownership, and crash-consistent artifacts + +Every dispatch has a fresh `attempt_id` on `CurrentTask` and `TaskResponse`. The +worker echoes it in `/finished` and records it in `result.json`. Coordinates such +as `(session_id, workflow_id, iteration)` identify the logical slot but do not +prove that a completion or artifact came from the live dispatch rather than a +superseded one. Attempt matching in `CoordinatorService._finish_assignment_locked()` and +`_read_recoverable_finished_request()` fences stale requests and stale local +artifacts without turning them into current results. + +The worker also sends `{hostname, pid, starttime}` on `/register` and every +`/finished`. The coordinator stores this `WorkerIdentity` on the live +`CurrentTask`, making ownership and same-host liveness verifiable. `starttime` +is a pid-reuse-proof process token supplied by team-harness. There is no periodic +heartbeat file: `src/loopy_loop/worker_identity.py` probes the recorded process +identity directly. + +Recovery-relevant artifact mutations are individually atomic. The helpers +`write_text_atomic()` and `write_json_atomic()` in `src/loopy_loop/sessions.py` +write a unique temporary file, flush it, and replace the destination. They are +used when publishing iteration results and text, prompts, harness ids, +`pending_finished_request.json`, later `children.json` mutations, and +`salvage.json`. Initial session scaffolding includes direct writes, so this is +not a blanket guarantee for every engine-created file. For the recovery-relevant +publications above, a process crash may leave an earlier complete version or +omit a later file, but it cannot leave a truncated artifact that recovery +mistakes for valid evidence. + +This guarantee does not extend magically to workflow-authored files such as +`control.json`, `goal_check.json`, or child requests. Packaged prompts instruct +workflows to publish those via temporary-file-plus-rename. The engine validates +them and never trusts torn/invalid content as evidence: invalid child requests +are rejected, invalid expected goal checks count toward `goal_check_broken`, and +invalid control stops with `invalid_control_output`. The crash model is process +crash consistency, not power-loss durability across every file (there is no +multi-file fsync transaction). + +--- + +## Worker liveness and orphaned-agent recovery + +Coordinator recovery and worker recovery are different problems: + +- A **coordinator crash** does not kill agent CLIs, because they are children of + the separate worker process. The durable session stack and result artifacts + are sufficient to resume. +- A **worker crash** can orphan agent CLIs which keep spending money and writing + to the checkout. State reconstruction alone would make immediate redispatch + unsafe because two writers could overlap. + +On a second `/register`, `CoordinatorService._raise_if_worker_alive()` first +checks the recorded worker identity. If that exact local process is alive, the +coordinator refuses the second worker with HTTP 409. If liveness is false or +unverifiable and a complete `pending_finished_request.json` or `result.json` +exists, the coordinator records that real result. Otherwise it performs orphan +recovery outside the state lock, so `loopy status` and `loopy stop` remain +responsive during a potentially long drain. + +The liveness guarantee is deliberately same-host. A remote hostname, a missing +start-time token, or an unavailable process-identity provider produces +"unknown," not "dead"; registration falls through rather than treating the +worker as verifiably alive. Same-host orphan reaping is still attempted when +team-harness has usable run records, but process reaping itself is same-host. A +remote worker is skipped and receives legacy abandonment because its processes +cannot be reached from the coordinator host. Deployments that separate +coordinator and worker hosts therefore do not get the local duplicate-worker +proof and must account for +that limitation operationally. + +D7 assigns the actual agent-process mechanism to team-harness, which owns the +processes it spawns. `recover_interrupted_iteration()` in +`src/loopy_loop/recovery.py` discovers the interrupted harness runs and invokes +team-harness's process-group operation: + +- `drain` (the default) lets in-flight agents finish within one shared bounded + timeout, preserving near-complete edits; team-harness reaps any process that + remains past that timeout; or +- `reap` terminates the tracked processes immediately. + +For an identity-tracked run, any reaper report that says a process may still be +writing makes recovery refuse replacement work. This fail-closed rule cannot be +claimed for the legacy/remote cases described above, where identity or reaping +is unavailable and registration falls back to abandonment. + +Whenever at least one harness run was processed, recovery atomically writes +`salvage.json` with the policy, counters, and team-harness reaper reports — even +if every report is unsettled and registration is then refused. The history code +changes from `abandoned` to `abandoned_after_` only when at least one +orphan reached a settled outcome. The salvage file does not contain a +loopy-generated diffstat. + +Drained agents may have left useful edits in the git working tree, but the task +which entered recovery has no coordinator-owned iteration result. Synthesizing +`result.json` from agent output would falsely close an iteration and violate D3. +Git preserves the work and `salvage.json` preserves its provenance. If the crash +abandonment does not itself trip `workflow_consecutive_failures_cap`, +`max_turns`, or another stop condition, normal scheduling continues and a later +real iteration can evaluate those edits; otherwise the session stops without +claiming the drained work succeeded. Recovery does not guarantee that the same +workflow is selected next. + +`src/tests/test_worker_liveness_recovery.py` covers live-worker refusal, +dead-worker reclaim, stale-owner completions, bounded recovery, unsettled-writer +refusal, and salvage records. + +--- + +## The planner/dispatcher template is executable from a clean init + +The `pm_planner_dispatcher` workflow set is a parent template whose dispatcher +creates child sessions using `inner_outer_eval`. That child workflow set is a +hard dependency of the template, not an optional example. + +`PACKAGED_TEMPLATE_EXTRA_SOURCES` in `src/loopy_loop/cli.py` therefore copies the +canonical `inner_outer_eval` workflow files whenever +`loopy init --template pm_planner_dispatcher` runs. The files are sourced from +the child template itself rather than duplicated under the PM template, so the +two cannot drift. + +The acceptance contract is more than file presence: +`test_clean_pm_init_can_dispatch_an_inner_outer_eval_child()` in +`src/tests/test_cli.py` starts from an empty directory, initializes the PM +template, dispatches a child, and runs its first assignment through the worker +path with the child's own goal. + +--- + +## Operational events, usage, and cost + +### `state.json` is truth; `events.jsonl` is a legibility projection + +`src/loopy_loop/events.py` writes one schema-versioned JSON line for significant +transitions such as session start/stop, task dispatch/finish, abandonment, +goal-check evidence, and child start/finish. Each session, including a child, +has its own stream. + +Events are appended **after** the state mutation commits. This ordering protects +correctness: an append failure or crash can create a gap, but it cannot roll back +the durable state. Replayed finalization can also duplicate an event. Consumers +must key on `event_id`, tolerate gaps/duplicates, and ignore a torn final line. +No scheduler or recovery decision may depend on the event stream. + +The CLI makes that projection useful without adding a dashboard: + +- `loopy status` walks the session stack and reports each session plus subtree + totals; `--watch` re-renders it; +- `loopy events` prints the deepest active session stream; `--follow` switches + streams as children start or finish, and `--json` preserves raw envelopes. + +These commands live in `src/loopy_loop/cli.py`; stream and CLI behavior are +covered by `src/tests/test_events_and_usage.py`. + +### Unknown usage stays unknown + +The worker's `_read_harness_usage()` reads coordinator-model turn usage from +team-harness's `run.json`. It cannot measure the separate Codex/Claude/Gemini CLI +accounts. Missing records return `None`, not zero; partially measured runs keep +their known token subtotal but count as an unknown iteration. + +`LoopState.usage_totals` is the durable per-session ledger. When a child +finalizes, its totals are copied to its `children.json` record; +`session_tree_usage_totals()` derives a parent's subtree total without +double-storing it. A resumed pre-ledger session reconciles historical iterations +as usage-unknown rather than pretending they cost nothing. + +When `model_prices` is configured, `estimate_cost_usd()` prices the known +coordinator tokens. `max_cost_usd` stops a session when its subtree estimate +reaches the configured budget, and preflight rejects a budget without prices. +This is an estimate with an explicit blind spot, not a claim to total agent spend. +The shipped budget is session-subtree-wide; per-workflow and wall-clock budgets +were deliberately left out until a concrete need appears. The budget and prices +are coordinator-side root settings loaded when the coordinator starts (and +reloaded on resume), not fields frozen in the worker config snapshot. While a +child is running, the coordinator checks that currently loaded shared threshold +against the child's known measured subtree only; the suspended parent can +include child spend only after finalization and the child cannot see prior +parent/sibling spend. The threshold is checked after results and can overshoot by +an iteration; it is not a single global hard cap across an in-flight parent and +child. + +--- + +## Failure containment + +Failed iterations record one `FailureKind` from `src/loopy_loop/models.py`: + +- `transient`: team-harness/provider marked the failure retryable and its own + retries were exhausted; +- `deterministic`: a confirmed auth/config/request failure that retrying cannot + fix; +- `crash`: the iteration was abandoned by the worker-crash recovery path (this + label does not prove that an unverifiable remote worker actually died); or +- `unknown`: the available evidence cannot support a stronger classification. + +`classify_failure_detail()` in `src/loopy_loop/harness_runner.py` derives the +classification from team-harness's structured detail where available. The kind +is carried through `result.json`, `/finished`, and session history; normal +`task_finished` events include it as well. Crash abandonment is explicit in +history and the `iteration_abandoned` event even though that event's payload +does not repeat the `failure_kind` field. A stopped run is therefore diagnosable +without scraping prose logs. + +`CoordinatorService._track_workflow_failure_cap()` maintains a durable +consecutive-failure counter per workflow. A success resets that workflow's +counter; reaching `workflow_consecutive_failures_cap` (default 5) stops the +session with `workflow_failure_cap`, including when crash-abandoned iterations +reach the cap. This prevents one wedged workflow from consuming all remaining +turns. + +loopy-loop does not add a second retry/backoff loop. Provider retry policy and +backoff remain in team-harness and are configurable through the +`team_harness_*retry*` fields. Repeating retries at both layers would multiply +latency without adding evidence. `src/tests/test_failure_taxonomy.py` covers the +classifier, durable counters, precedence, child behavior, and wire compatibility. + +--- + +## Dependency, documentation, and model-tier readiness + +### Eval tooling is part of the runnable product + +The packaged eval workflows invoke `eval-banana`, so it is a normal dependency +in `pyproject.toml`, not an optional install mode. Tool installers such as +`uv tool install` and pipx may expose only the primary package's entry points; +`ensure_interpreter_scripts_on_path()` in `src/loopy_loop/worker.py` finds the +directory containing the installed eval-banana script and appends it to `PATH`. +Appending preserves any target-repo or operator-selected executable that is +already earlier on `PATH`. + +The repository's Agent Skill was rewritten with the 0.5.0 release to describe +the session-local layout, workflow sets, single identity-verified ping-pong +worker, child sessions, attempts, and crash recovery. It lives at +`skills/loopy-loop/SKILL.md`; the README uses the same single-worker contract. +This was necessary because agent-facing setup instructions are effectively an +API surface: confidently generating a removed layout is worse than omitting a +feature. The clean-init tests validate the templates themselves, although the +prose Skill is not mechanically compared with them. + +### Named model tiers centralize project-local worker choice + +D9 keeps every harness coordinator in a session tree on the same strong +`team_harness_model`. Cost control happens per spawned worker. A root config may +declare `model_tiers` as tier name → agent → `{model, effort}` and an optional +`default_tier`. + +`load_root_config()` in `src/loopy_loop/config.py` rejects a `default_tier` +combined with duplicate explicit mappings. `resolve_model_tiers()` then derives +the concrete per-agent defaults from the named default, and +`render_model_tier_guidance()` appends the tier table to every harness +coordinator's system prompt. Workflow prompts can request `economy` or `strong` +without embedding model ids. Children inherit the parent's frozen resolved +config snapshot; editing root YAML mid-session does not silently change their +model policy. + +Tier selection is guidance plus audit evidence, never an engine veto (D8). The +harness records requested and effective model/effort per spawn; review or eval +can detect a poor choice and repair it. The engine does not enforce a tier +allowlist or weaken child coordinators by depth. + +--- + +## Deliberate boundaries + +The shipped hardening does not imply the following features: + +- no parallel loopy workers or concurrent child sessions (D2); +- no human-answer gate or `waiting_for_human` state (D5); +- no independently configured per-child budgets (the running child still + enforces the session tree's shared `max_cost_usd`; child-specific limits were + withdrawn as needless complexity); +- no per-depth coordinator profiles (D9); +- no agent-authored deterministic checks in the stock eval template (D4); +- no breadth-first child scheduling (D2/D6), and no grandchildren in the current + implementation even though a deeper depth-first chain is not forbidden by + those decisions; +- no synthetic success result for drained work (D3/D7); and +- no claim that coordinator-token cost includes agent-CLI spend. + +Changing a D-numbered boundary requires amending that decision. Changing another +current scope choice, such as child depth or child-specific budgets, requires +an explicit design update rather than treating the missing feature as a bug. + +--- + +## Historical proposal map + +| Review id | Shipped behavior | Release | Primary implementation/evidence | +|---|---|---:|---| +| P0.1 | Durable child pointer, stack reconstruction, attempts, request idempotence, atomic artifacts | 0.4.0 | `coordinator_app.py`, `sessions.py`, `test_session_stack_recovery.py` | +| P0.4 | PM template includes its canonical child workflow set | 0.4.0 | `cli.py`, `test_cli.py` | +| P1.1 | Events, usage/cost ledger, budget, stack-aware status/events CLI | 0.5.0 | `events.py`, `models.py`, `test_events_and_usage.py` | +| P1.3 | Agent Skill and README aligned with the current API | 0.5.0 | `skills/loopy-loop/SKILL.md`, `README.md` | +| P2.1 (part) | Eval dependency/PATH readiness and named worker model tiers | 0.5.0/0.6.0 | `pyproject.toml`, `worker.py`, `config.py`, `test_config.py` | +| P2.2 | One `_advance()` transition | 0.4.0 | `coordinator_app.py`, `test_session_stack_recovery.py` | +| P2.3 | Failure taxonomy and per-workflow failure cap | 0.5.0 | `harness_runner.py`, `models.py`, `test_failure_taxonomy.py` | +| P2.5 | Worker liveness, drain/reap recovery, salvage provenance | 0.3.0 | `worker_identity.py`, `recovery.py`, `test_worker_liveness_recovery.py` | diff --git a/design/designs/success-semantics-and-evaluation.md b/design/designs/success-semantics-and-evaluation.md index 8a2a3b6..e336b40 100644 --- a/design/designs/success-semantics-and-evaluation.md +++ b/design/designs/success-semantics-and-evaluation.md @@ -77,7 +77,8 @@ and that layer is LLM-as-judge by design (Decision 2), a whole iteration's "success" can rest on a single model judgment with no deterministic backstop. For low-stakes goals this is an acceptable, conscious trade. For high-stakes work it should be **backstopped**, not reverted — see the note in Decision 2 about -repo-owned deterministic checks, and the separate improvement-proposals doc. +repo-owned deterministic checks and the active +[`P1.2` proposal](../proposals/improvement-proposals.md#p12--target-owned-deterministic-evaluation-backstop). ### Alternatives considered and rejected @@ -185,5 +186,6 @@ than abandoning the approach. Both decisions are sound. Neither should be reverted. The one thing worth adding — for high-stakes targets only — is a deterministic backstop built from the target repo's **own** contract tests, which strengthens both decisions without undoing -either. That, and other forward-looking proposals, are covered separately in the -improvement-proposals design doc. +either. That conditional follow-up is tracked as +[`P1.2`](../proposals/improvement-proposals.md#p12--target-owned-deterministic-evaluation-backstop), +not as part of this implemented design. diff --git a/design/proposals/improvement-proposals.md b/design/proposals/improvement-proposals.md new file mode 100644 index 0000000..38d11fa --- /dev/null +++ b/design/proposals/improvement-proposals.md @@ -0,0 +1,193 @@ +# Improvement Proposals + +**Status:** Proposed — non-binding, unimplemented work only +**Date reviewed:** 2026-07-15 +**Applies to:** possible follow-up work after the reliability and operations +features described in +[`../designs/long-running-loop-reliability.md`](../designs/long-running-loop-reliability.md). + +This file contains only proposals that remain relevant and are not implemented. +It is not a description of current behavior. Binding choices live in +[`../decisions.md`](../decisions.md) and implemented designs live in +[`../designs/`](../designs/). + +The proposal identifiers are retained from the July 2026 improvement review so +historical commits, tests, and changelog entries remain traceable. Implemented +items from that review moved to the reliability design linked above. Rejected, +withdrawn, or now-unnecessary ideas are intentionally absent; their lasting +dispositions live in the decision log and the implemented design's boundaries. + +--- + +## P1.2 — Target-owned deterministic evaluation backstop + +### Current state + +The packaged `inner_outer_eval` workflow set uses `harness_judge` checks only. +Its `eval_reviewer` prompt forbids agents from authoring deterministic checks, +because experience showed that implementers invent brittle or gameable checks. +This is the binding D4 policy and remains correct for a generic target repo. + +The boundary in D4 is equally important: a deterministic contract that the +target repo already owns is not agent-authored. Examples include the target's +existing `pytest` suite, migration check, import boundary check, or `make test` +target. Such a contract can catch a bad LLM judgment without recreating the +self-authored-check failure mode. + +### Proposal + +When a target repo has a trustworthy contract suite and the cost of a false +`goal_met` is material, give that target a dedicated child workflow set which: + +- keeps the LLM judge for qualitative outcome assessment; +- runs the target's pre-existing contract command as an additional hard gate; +- records the command, exit status, and report as session evidence; and +- derives the deterministic portion of `goal_check.json` from the report rather + than asking an agent to paraphrase console output. + +Do not loosen the stock `inner_outer_eval` rule. The dedicated workflow set owns +the command and its parsing contract; an implementation agent must not create or +rewrite the gate during the run. + +### Why it remains conditional + +There is no honest generic command loopy-loop can run for every repository. +Adding this to the stock template would either assume a toolchain or return to +agent-authored checks. Implement it with the first high-stakes target that has a +real suite, and test the workflow set against that target's actual report format. + +**Effort:** S–M per target workflow set. **Status:** Deferred until a suitable +target exists. + +--- + +## P2.1 — Centralize shipped model defaults and define dependency compatibility + +### Current state + +Two parts of the original P2.1 are complete: + +- `eval-banana` is a normal loopy-loop dependency and the worker makes its + installed CLI visible to spawned agents; and +- root configs can define named worker `model_tiers`, with D9 fixing the policy + of one strong coordinator model and prompt-guided, audited per-spawn worker + choice. + +Full per-session execution profiles are not missing work. D9 deliberately +rejects coordinator-model differentiation by parent/child depth for now. + +Two narrower maintenance risks remain. First, loopy-loop's own stock model ids +are repeated across `src/loopy_loop/config.py`, the CLI scaffold, packaged YAML, +README examples, and repository-local harness/eval configuration. Model churn +therefore still requires a coordinated edit. Second, published dependencies on +`team-harness` and `eval-banana` have minimum versions but no documented upper +compatibility policy; a breaking pre-1.0 release can be selected by a fresh +install. + +### Proposal + +1. Make one repository-owned definition the source for model ids used by shipped + defaults and generated templates. Generate or validate the duplicated + examples from that definition so drift fails CI. This concerns loopy-loop's + product defaults; project-local workflow prompts should continue to name + tiers, not model ids. +2. Define and test the supported loopy-loop × team-harness × eval-banana version + range. Choose exact pins, compatible upper bounds, or a tested compatibility + matrix based on observed release discipline, then encode the choice in + `pyproject.toml` and CI. The goal is reproducible compatibility, not exact + pinning for its own sake. + +Do not revive per-child budgets, per-session/per-child provider profiles, or +weaker child coordinator models as part of this work. Per-child budgets were +withdrawn as needless complexity; per-depth coordinator changes would require +amending D9. A repo-global profile shared uniformly by the whole session tree is +a separate idea and is not ruled out by D9. + +**Effort:** S–M. **Status:** Proposed. + +--- + +## P2.4 — Complete operator preflight and active-session control + +### Current state + +`loopy status` already walks the durable parent→child stack, reports usage and +estimated cost, and supports `--watch`. `loopy events --follow` likewise follows +the deepest active session. Those parts of the original proposal shipped in +0.5.0. + +The remaining gap is command-side validation and control. There is no `doctor` +or standalone `validate` command. `loopy stop` mutates only the latest top-level +`StateStore`; while a child is active, that child does not see the request and +the parent cannot honor it until the child terminates. There is also no recorded +force-stop path for a hung worker or orphaned agent processes. + +### Proposal + +- **`loopy validate`:** expose the existing `run_preflight()` and workflow-graph + validation without starting a session. Report all actionable configuration, + prompt, workflow-reference, and required-tool errors in one run. +- **`loopy doctor`:** add environment diagnostics that preflight cannot infer + from config alone: Python/package versions, installed agent CLIs and auth, + writable session paths, git state, port availability, and the effective + model/dependency compatibility information from P2.1. +- **Active-session-aware `loopy stop`:** resolve the durable session stack and + record a tree-level stop intent that the live child observes at its next + coordinator check-in, while ensuring the suspended parent cannot dispatch more + work afterward. This is cooperative and does not interrupt the harness already + running. Define the multi-file transition and its restart behavior before + implementing it; do not merely set two independent flags and hope both writes + land. +- **`loopy stop --force`:** for an explicit operator intervention, terminate the + loopy-owned worker process, then use team-harness's D7 drain/reap machinery for + the agent process groups that worker spawned. Finally write a durable terminal + outcome explaining what was stopped and what cleanup occurred. It must never + leave a possibly-live writer while dispatching replacement work. + +This proposal does **not** add a `paused` or `waiting_for_human` state. Terminating +the coordinator and later using `--resume` can prevent new dispatch after the +in-flight worker finishes or fails its handoff, but it does not pause the running +harness. D5 rejects a preferred human gate and D8 rejects arbitrary preventive +mid-run approval flows. + +**Effort:** M. **Status:** Proposed. + +--- + +## Future direction — Deeper depth-first child chains + +### Current state + +The shipped planner/dispatcher runtime supports a depth-first, two-level tree: +only a top-level session can dispatch a child, and that parent stays suspended +until the child terminates. P0.1 removed the original durability prerequisite +by making the active-child pointer, staged dispatch, stack reconstruction, and +child finalization crash-recoverable. D9 also defines model policy for "any +deeper level," so a deeper sequential chain is not rejected architecture. + +The common parent→child machinery already walks durable pointers and rolls a +finished child's whole-subtree usage into its parent. The explicit guard in +`CoordinatorService._dispatch_child_session_if_requested()` still prevents a +session with `parent_session_id` from dispatching a grandchild. + +### Proposal + +If a concrete target needs hierarchical decomposition beyond one planner and +one implementation loop, generalize the existing depth-first stack without +adding concurrency: + +- allow the deepest active session to dispatch one child and suspend until it + finishes; +- preserve one live task and one live child edge across the entire checkout; +- make restart reconstruction, terminal-child unwinding, request idempotence, + status/events, stop handling, and usage aggregation work at every depth; and +- add crash-window tests at multiple nested edges before calling the deeper + form supported. + +Do not combine this with breadth-first children or parallel loopy workers; D2 +and D6 keep execution sequential on the shared checkout. Do not add nesting +only for abstraction's sake: the existing two-level double loop is simpler and +already adequate for most targets. + +**Effort:** M–L. **Status:** Deferred until a target demonstrates a real +three-level decomposition need. diff --git a/docs/http-contract.md b/docs/http-contract.md index cf49889..07a2cb7 100644 --- a/docs/http-contract.md +++ b/docs/http-contract.md @@ -80,8 +80,8 @@ Rules: - `workflow_set` tells the worker which `.loopy_loop/workflow_sets//workflows//` directory to load. -- If `current_task` is already set (previous worker crashed without calling - `/finished`), `/register` proceeds in three steps: +- If `current_task` is already set (the prior assignment remains + unacknowledged), `/register` proceeds in three steps: 1. **Liveness check.** If the recorded worker identity is *verifiably still alive* (same host, matching pid + starttime), the call is refused with **HTTP 409** and no state is mutated — the task is not abandoned and no @@ -92,18 +92,18 @@ Rules: file proves the task completed, the completed task is recorded in history before checking stop conditions. 3. **Orphan recovery.** With nothing recoverable, the coordinator applies the - configured recovery policy (`recovery_policy`, default `drain`; ONE - `recovery_drain_timeout_s` deadline shared across all of the iteration's - interrupted runs) to any agent processes the dead worker's harness run - left behind, writes a `salvage.json` into the interrupted iteration - directory when something was handled, and records the iteration as failed - with `error="abandoned_after_"` (or plain `"abandoned"` when - nothing settled). Requires team-harness with the process reaper; older - versions skip this step. Recovery refuses to dispatch replacement work — - **HTTP 409** — when team-harness's guard reports the run's owner still - alive, or when any orphan's state after recovery is "may still be - running" (unverifiable identity, probe failure, or a kill that did not - land); the salvage record documents the unresolved processes. + configured recovery policy (`recovery_policy`, default `drain`; one + `recovery_drain_timeout_s` deadline shared across all interrupted harness + runs) to agent processes left by the interrupted worker task. It writes + `salvage.json` whenever at least one tracked harness run is processed, and + records the task as failed with `error="abandoned_after_"` when at + least one orphan settled (or plain `"abandoned"` otherwise). Recovery + refuses replacement work with **HTTP 409** when team-harness's guard says + the run owner is still alive, or when a processed orphan may still be + running because its identity was unverifiable, a probe failed, or a kill + did not land. The salvage record documents those unresolved processes. + Older team-harness versions without the process reaper degrade to plain + abandonment. The recovery settings are coordinator-side configuration only — they are **not** part of the wire `config_snapshot` (released workers reject unknown snapshot fields). @@ -111,8 +111,9 @@ Rules: usable while it drains; `/register` can still block roughly up to the drain deadline (plus kill grace periods), and the bundled worker uses an unbounded read timeout on `/register` only. Process recovery is same-host: a worker - identity from another hostname skips reaping (its processes cannot be - reached from here). A **hung-but-alive** worker keeps its task (409); the + identity from another hostname skips reaping and falls back to plain + abandonment because its processes cannot be reached from here. A + **hung-but-alive** worker keeps its task (409); the escape hatch is to kill that process and register again — the 409 message names its pid. - If the loop is in a terminal state, `/register` immediately returns `action=stop`. @@ -153,8 +154,10 @@ run the next dispatched task, so its identity is stamped onto that task. `failure_kind` is optional and classifies a failed iteration for the history record: `transient` (provider said retry — the harness's own retries were already exhausted), `deterministic` (auth/config error retries cannot fix), or -`unknown`. The coordinator itself records `crash` for iterations abandoned by -a dead worker. Consecutive failures of the same workflow are counted +`unknown`. The coordinator itself records `crash` for tasks abandoned by +worker-crash recovery; this label records the recovery path, not proof that a +remote or otherwise unverifiable worker was dead. Consecutive failures of the +same workflow are counted (coordinator-side `workflow_consecutive_failures_cap`, default 5; any success of that workflow resets its counter) and stop the loop terminally with `stop_reason="workflow_failure_cap"` at the cap. diff --git a/docs/session-layout.md b/docs/session-layout.md index 6bea8a5..f0c7e83 100644 --- a/docs/session-layout.md +++ b/docs/session-layout.md @@ -6,10 +6,15 @@ One session directory is created per fresh coordinator run and reused for all it .loopy_loop/ └── sessions/ └── / + ├── goal.md ├── session.json + ├── state.json ├── events.jsonl ├── control.json ├── updates_from_user.md + ├── children.json + ├── child_requests/ + ├── children/ ├── project_state/ │ └── finished.md ├── eval_checks/ @@ -36,8 +41,19 @@ One session directory is created per fresh coordinator run and reused for all it └── goal_check.json ``` +`children/` appears after the session dispatches its first child; the +`children.json` ledger and `child_requests/` inbox are scaffolded at session +creation. + ## Session Files +`goal.md` + +- Exact resolved goal text for this session, including a child request's goal + or a `--goal-file` override +- The rendered Goal and this session-local copy are canonical for active work; + the repo-root goal file may describe a different session + `session.json` - Session metadata written once when the session directory is created @@ -45,6 +61,13 @@ One session directory is created per fresh coordinator run and reused for all it - New session ids use `___` so session directories sort chronologically by name +`state.json` + +- Coordinator-owned durable dispatch state, history, stop state, active-child + pointer, and usage ledger +- This is the durable source of truth; `events.jsonl` is only an observability + projection + `events.jsonl` - Append-only event stream: one JSON object per line, one file per session @@ -77,8 +100,9 @@ One session directory is created per fresh coordinator run and reused for all it `project_state/` - Optional workflow-owned markdown state for reusable workflows -- `loopy_loop_goal.txt` is the source of truth for the target, constraints, and - completion intent; do not copy or restate the goal into `project_state/` +- The rendered Goal and session-local `goal.md` are the source of truth for the + target, constraints, and completion intent; do not copy or restate the goal + into `project_state/` - Common files include `README.md`, `memory.md`, `current_state.md`, `what_we_have.md`, `decisions.md`, `eval_results.md`, `finished.md`, and `what_we_should_do/plan.md` @@ -138,6 +162,21 @@ eval-banana run \ .loopy_loop/sessions//harness_outputs/_// ``` +`children.json`, `child_requests/`, and `children/` + +- `child_requests/` is the inbox for pending child-session requests, and + `children/` contains the child session directories once dispatched +- `children.json` indexes every dispatched child with its `session_id`, + `workflow_set`, status, timestamps, `stop_reason`, and originating + `request_file` +- A request filename suppresses redispatch only while its child record is + `running` or `dispatching`, closing the record-written/request-not-yet-removed + crash window; a completed child's old filename may be reused for new work +- The parent's `state.json` records `active_child_session_id` while a child runs, + which is the durable pointer followed during restart +- At finalization the child record captures whole-subtree usage so the parent's + tree-wide ledger and `max_cost_usd` include completed children + ## Iteration Files `prompt.txt` @@ -176,42 +215,31 @@ eval-banana run \ - If the file is missing but `result.json` exists for the active task, the coordinator can reconstruct the finished request from `result.json` -`children.json` (parent sessions) - -- Index of child sessions dispatched by this session; each record carries the - child `session_id`, `workflow_set`, `status`, timestamps, `stop_reason`, and - the originating `request_file` name (which makes the dispatch scan - idempotent: a request whose filename already appears here is never - dispatched twice, even if a crash left the file behind) -- The parent's `state.json` additionally records `active_child_session_id` - while a child runs — the durable session-stack pointer a restarted - coordinator follows to resume the child instead of orphaning it -- At finalization the record also captures the child's usage totals - (`usage`: tokens, known/unknown iteration counts, duration), which is how - a parent's tree-wide usage and `max_cost_usd` budget include finished - children without double-storing - -Atomicity and crash model: the coordinator- and worker-owned artifacts above -are written atomically (same-directory temp file + rename), so a **process -crash** never leaves them truncated — readers see either the previous or the -new complete file. This does not extend to power loss / kernel crashes (no -fsync), and it cannot be enforced for **workflow-written** signals -(`control.json`, `goal_check.json`, child requests): the packaged prompts -instruct agents to publish those via temp-file + rename, but a torn write by a -non-compliant agent is read as invalid output. +Atomicity and crash model: recovery-relevant mutations such as `state.json`, +later `children.json` updates, iteration results, handoff records, and +`salvage.json` use same-directory temporary files plus replace. Each published +file is atomic, but the parent/child transition is not one multi-file +transaction; restart reconciliation handles its staged crash windows. Initial +session scaffolding includes direct writes, so atomicity is not a blanket claim +for every coordinator-created file. Nor can the engine enforce atomic writes +for workflow-owned signals (`control.json`, `goal_check.json`, child requests): +packaged prompts require temp-file-plus-rename publication, and the coordinator +treats torn or invalid output as invalid rather than as evidence. There is no +fsync guarantee for power or kernel failure. `salvage.json` -- Written into the interrupted iteration's directory during crash recovery, - when the coordinator applied the recovery policy (`recovery_policy`, default - bounded drain) to agent processes a dead worker's harness run left behind +- Written into the interrupted iteration's directory when crash recovery + processes at least one tracked harness run using `recovery_policy` (bounded + `drain` by default, or immediate `reap`) - Records the reap reports: which orphaned agents were drained (allowed to finish), reaped (killed), or skipped, so the provenance of any surviving working-tree edits is auditable rather than a mystery diff -- The iteration is still re-run — its `result.json` never existed and is never - fabricated; when any orphan actually settled, the corresponding history entry - carries `error="abandoned_after_"` (e.g. `abandoned_after_drain`) - instead of plain `"abandoned"` +- Its `result.json` never existed and is never fabricated. The abandonment + consumes a turn; normal scheduling continues only if no stop condition fires, + and it need not select the same workflow next. When any orphan settled, the + history entry carries `error="abandoned_after_"` (for example, + `abandoned_after_drain`) instead of plain `"abandoned"` - Schema: `{"schema_version": 1, "recorded_at": ..., "policy": ..., "reaped_runs": N, "settled_workers": N, "unsettled_workers": N, "reports": [...]}`. A non-zero `unsettled_workers` means some orphan may diff --git a/skills/loopy-loop/SKILL.md b/skills/loopy-loop/SKILL.md index dac75ff..8c8b6f3 100644 --- a/skills/loopy-loop/SKILL.md +++ b/skills/loopy-loop/SKILL.md @@ -448,26 +448,29 @@ terminal state. Resume reconstructs the session stack: a running child session continues where it was; a child found terminal is finalized and its parent resumed. -### Crash recovery (worker died mid-iteration) +### Crash recovery (worker task interrupted) When a new worker registers while a task is still marked live, the coordinator: 1. **Liveness check** — refuses (409) if the recorded worker is verifiably still alive on this host. -2. **Result recovery** — if the dead worker already produced +2. **Result recovery** — if the prior worker already produced `pending_finished_request.json` or `result.json`, the completed task is recorded; no work is lost. 3. **Orphan recovery** — otherwise the recovery policy is applied to agent - processes the dead worker's harness run left behind: `drain` (default) + processes the prior worker's harness run left behind: `drain` (default) waits up to `recovery_drain_timeout_s` for them to finish; `reap` kills - them. When at least one orphaned run was actually handled, a - `salvage.json` in the interrupted iteration directory records what - happened to each orphan and the failed iteration carries - `error="abandoned_after_"` (plain `"abandoned"` when nothing - settled). The iteration is then re-dispatched unless a stop condition - fires first (the abandonment consumes a turn, so it can itself trigger - `max_turns`). If any orphan may still be running, the coordinator refuses - to dispatch (409) rather than risk duplicate work. + them. When at least one harness run is processed, `salvage.json` records + what happened; the failed iteration carries + `error="abandoned_after_"` only when something settled (plain + `"abandoned"` otherwise). The abandonment consumes a turn and normal + scheduling continues only if no stop condition fires; the same workflow is + not guaranteed to run next. For identity-tracked harness runs, an unsettled + reaper outcome refuses dispatch (409). A remote loopy worker skips reaping + and falls back to legacy abandonment; when local worker liveness is + unverifiable, same-host team-harness recovery is still attempted if its run + records support it. Those fallback paths cannot provide the local safety + proof. A hung-but-alive worker keeps its task (409 names its pid); the escape hatch is to kill that process and register again. @@ -488,15 +491,13 @@ live child under the suspended parent. Every session also has an append-only `child_started`, `child_finished`, `session_stopped`, ...) — the operational legibility stream (best-effort; the durable truth stays in state.json). -Both commands print a friendly error and exit if the coordinator holds the +The commands print a friendly error and exit if the coordinator holds the state lock mid-request — retry shortly. -**Child-session caveat:** both commands operate on the latest **top-level** -session state. While a child session runs, `loopy status` shows the suspended -parent (often `current_task: none`), not the live child; and `loopy stop` -sets `stop_requested` on the parent — the child does not see the flag and -keeps iterating until it reaches a terminal state, and only then does the -resumed parent honor the stop. +**Child-session caveat:** `status` and `events` resolve the durable stack and +show/follow the live child. `stop` still operates on the latest **top-level** +session state: the child does not see that flag and keeps iterating until it +reaches a terminal state, after which the resumed parent honors the stop. Per-iteration artifacts live at `.loopy_loop/sessions//iterations/_/` diff --git a/src/loopy_loop/config.py b/src/loopy_loop/config.py index c16b0bb..5396085 100644 --- a/src/loopy_loop/config.py +++ b/src/loopy_loop/config.py @@ -215,7 +215,7 @@ class RootConfig(BaseModel): description=( "What to do with agent processes orphaned by a crashed worker: " "'drain' lets in-flight agents finish (bounded by " - "recovery_drain_timeout_s) before re-running the iteration; " + "recovery_drain_timeout_s) before normal scheduling continues; " "'reap' kills them immediately." ), ) diff --git a/src/loopy_loop/coordinator_app.py b/src/loopy_loop/coordinator_app.py index 46cbcaa..0c5866a 100644 --- a/src/loopy_loop/coordinator_app.py +++ b/src/loopy_loop/coordinator_app.py @@ -201,8 +201,8 @@ def register_worker( self, *, request: RegisterRequest | None = None ) -> TaskResponse: caller = request.worker if request is not None else None - # Two-phase recovery: the potentially long drain of a dead worker's - # orphaned agents (up to recovery_drain_timeout_s) runs in phase A, + # Two-phase recovery: the potentially long drain of an interrupted + # task's agent processes (up to recovery_drain_timeout_s) runs in phase A, # OUTSIDE the state lock, so `loopy status`/`stop` and /finished stay # responsive. Phase B re-validates under the lock and retries from # phase A when the state moved in between. @@ -312,8 +312,8 @@ def mutator(state: LoopState | None) -> tuple[LoopState, TaskResponse | None]: now=now, ) elif recovery is not None and _same_task(orphaned, recovery[0]): - # The dead worker's orphans were handled in phase A - # (outside the lock); commit the abandonment. + # The interrupted task's agent processes were handled in + # phase A (outside the lock); commit the abandonment. outcome = recovery[1] error = "abandoned" if outcome.salvaged: @@ -714,7 +714,7 @@ def _track_workflow_failure_cap( state.stop_reason = "workflow_failure_cap" def _recover_orphaned_agents(self, *, current_task: CurrentTask) -> RecoveryOutcome: - """Apply the configured recovery policy to a dead worker's orphans. + """Apply the configured recovery policy to an interrupted task's agents. Runs in phase A, OUTSIDE the state lock — draining can take up to the configured timeout without blocking status/stop or /finished. diff --git a/src/loopy_loop/models.py b/src/loopy_loop/models.py index 1603b91..7d64074 100644 --- a/src/loopy_loop/models.py +++ b/src/loopy_loop/models.py @@ -23,8 +23,9 @@ # own retries were already exhausted — a later iteration may succeed. # - "deterministic": retrying the same thing cannot help (auth failure, # invalid config, 4xx). -# - "crash": the coordinator observed the worker die mid-iteration -# (abandoned / abandoned_after_ entries). +# - "crash": the task was abandoned by the worker-crash recovery path +# (abandoned / abandoned_after_ entries); for a remote or otherwise +# unverifiable identity, this does not prove the worker died. # - "unknown": no classification signal (agent-process failures, unexpected # exceptions, results from pre-taxonomy versions). FailureKind = Literal["transient", "deterministic", "crash", "unknown"] diff --git a/src/loopy_loop/recovery.py b/src/loopy_loop/recovery.py index 85ae247..a4f615a 100644 --- a/src/loopy_loop/recovery.py +++ b/src/loopy_loop/recovery.py @@ -1,10 +1,10 @@ -"""Crash recovery for a dead worker's orphaned agent processes (D7 / P2.5). +"""Recovery for an interrupted worker task's agent processes (D7 / P2.5). -When the coordinator confirms a worker is dead and its iteration produced no +When a prior task remains unacknowledged and its iteration produced no recoverable result, the agent CLIs that worker's harness spawned may still be -running — orphaned, spending money, writing to the checkout. team-harness owns -the mechanism (persisted process identity + drain/reap policies, TH-D5); this -module is the loopy-side trigger: +running, spending money and writing to the checkout. team-harness owns the +mechanism (persisted process identity + drain/reap policies, TH-D5); this module +is the loopy-side trigger: 1. **Discover** the interrupted harness run(s): team-harness routes each run's session output under the iteration's ``harness_outputs`` directory, named by @@ -15,12 +15,14 @@ timeout, preserving near-complete work and a clean tree) or ``reap`` (kill). 3. **Record the salvage**: a ``salvage.json`` in the interrupted iteration's directory carrying the reap reports, so the provenance of any surviving - working-tree edits is auditable (the iteration itself is still re-run — its - ``result.json`` never existed and is never fabricated; loopy `decisions.md` - D3/D7). + working-tree edits is auditable. The interrupted task is abandoned and + consumes a turn; its ``result.json`` never existed and is never fabricated, + and normal scheduling continues only if no stop condition fires (loopy + `decisions.md` D3/D7). team-harness versions without the reaper are tolerated: recovery degrades to -the pre-existing behavior (mark abandoned, re-run) with nothing reaped. +the pre-existing behavior (mark abandoned, then continue normal scheduling if +allowed) with nothing reaped. ``ReapRefusedError`` — team-harness's own parent-liveness guard — bubbles up so the coordinator can treat "the run's owner is still alive" as a busy signal. """ @@ -100,7 +102,7 @@ def _load_reaper() -> tuple[Any, Any] | None: @dataclass class RecoveryOutcome: - """What crash recovery did about a dead worker's orphaned agents.""" + """What crash recovery did about an interrupted task's agent processes.""" reaped_runs: int = 0 settled_workers: int = 0 @@ -238,9 +240,9 @@ def _write_salvage_record( ) -> None: """Make the salvage auditable: which orphans were handled, and how. - The iteration is still re-run (D3: its result.json never existed and is - never fabricated); surviving working-tree edits are explained by this - record instead of appearing as a mystery diff. + The interrupted task is abandoned rather than synthesized (D3: its + result.json never existed and is never fabricated); surviving working-tree + edits are explained by this record instead of appearing as a mystery diff. """ iteration_dir = iteration_dir_path( repo_root=repo_root, diff --git a/src/tests/test_worker_liveness_recovery.py b/src/tests/test_worker_liveness_recovery.py index e6fe678..42def23 100644 --- a/src/tests/test_worker_liveness_recovery.py +++ b/src/tests/test_worker_liveness_recovery.py @@ -2,9 +2,10 @@ The worker sends its process identity with /register and /finished; the coordinator stamps it onto the dispatched CurrentTask, refuses (409) to reclaim -a task whose worker is verifiably alive, and — on a confirmed-dead worker with -nothing recoverable — applies the recovery policy to orphaned agent processes -via team-harness before re-running the iteration. +a task whose worker is verifiably alive, and — when the prior worker is not +verifiably alive and nothing is recoverable — applies the recovery policy to +agent processes via team-harness before recording abandonment and resuming +normal scheduling. """ from __future__ import annotations diff --git a/website/src/app/docs/cli-reference/page.mdx b/website/src/app/docs/cli-reference/page.mdx index d0f114d..6c01696 100644 --- a/website/src/app/docs/cli-reference/page.mdx +++ b/website/src/app/docs/cli-reference/page.mdx @@ -1,12 +1,12 @@ export const metadata = { title: "CLI Reference", description: - "Every loopy-loop command: init, coordinator, worker, status, and stop, with their flags and usage examples.", + "Every loopy-loop command: init, coordinator, worker, status, events, and stop, with their flags and usage examples.", }; # CLI Reference -loopy-loop is driven by a single command-line tool with five subcommands: `init` scaffolds a repo, `coordinator` and `worker` run the loop, and `status` and `stop` control it. Every command operates on the current working directory, so run them from the root of your target repository. +loopy-loop is driven by a single command-line tool with six subcommands: `init` scaffolds a repo, `coordinator` and `worker` run the loop, `status` and `events` inspect it, and `stop` controls it. Every command operates on the current working directory, so run them from the root of your target repository. ## Command aliases @@ -57,7 +57,7 @@ Starting a coordinator against a still-running session without `--resume` is int ## loopy worker -Runs a blocking worker that executes assignments through `team-harness`. It calls `/register` once for its first task, then loops — running each workflow and reporting via `/finished` — until the coordinator returns a stop response. You can run several workers against one coordinator. +Runs the single blocking worker that executes assignments through `team-harness`. It calls `/register` once for its first task, then loops — running each workflow and reporting via `/finished` — until the coordinator returns a stop response. A second worker is refused while the recorded worker is verifiably alive. | Option | Default | Description | | --- | --- | --- | @@ -71,7 +71,7 @@ The worker reaches the model layer too, so if your provider needs an API key, ex ## loopy status -Prints the state of the latest session: overall status, session id, completed iteration count, the current task (workflow id, iteration, session, and start time), and the stop reason if any. If there is no state yet, it says so. +Prints the durable session stack: overall status, session id, completed iteration count, current task, stop reason, subtree token usage and duration, and estimated cost when `model_prices` is configured. While a child runs it appears below the suspended parent. If there is no state yet, the command says so. ```bash loopy status @@ -79,17 +79,29 @@ loopy status ```text status: running -session: 20260712T193000Z-a1b2c3d4e5f6 -iteration_count: 12 -current_task: inner (iteration 12, session 20260712T193000Z-a1b2c3d4e5f6, started 2026-07-12T19:41:07Z) +session: 20260712_193000_a1b2c3d4e5f6_7a8b9c0d +iteration_count: 11 +current_task: inner (iteration 12, session 20260712_193000_a1b2c3d4e5f6_7a8b9c0d, started 2026-07-12T19:41:07Z) stop_reason: none +subtree_usage: prompt_tokens=12000 completion_tokens=3000 (iterations fully measured: 11, unknown: 0) +subtree_harness_duration_s: 840 ``` -`status` takes no options. +`loopy status --watch` clears and re-renders the view every two seconds until interrupted. + +## loopy events + +Prints the deepest active session's versioned `events.jsonl` stream. `--follow` tails new events and switches streams when a child starts or finishes; `--json` emits raw JSON lines. + +```bash +loopy events +loopy events --follow +loopy events --follow --json +``` ## loopy stop -Requests a graceful stop by setting `stop_requested=true` in the latest session-local state. Running workers exit after their next check-in with the coordinator; the coordinator brings the session to a terminal state. It prints `stop requested` on success, and errors if there is no state to stop. +Requests a graceful stop by setting `stop_requested=true` in the latest top-level session state. When no child is active, the worker exits after its next check-in and the coordinator brings the session to a terminal state. It prints `stop requested` on success, and errors if there is no state to stop. ```bash loopy stop @@ -97,6 +109,8 @@ loopy stop `stop` takes no options. For how a *workflow* stops the loop on its own — versus this operator-initiated stop — see [Success & Control](/docs/success-and-control). +`stop` currently targets the latest top-level session. If a child is active, the child does not see the flag; it terminates normally, then the resumed parent honors the request. Child-aware and force-stop behavior remains proposed work. + ## Where to go next - [Getting Started](/docs/getting-started) — these commands in a full first-run walkthrough. diff --git a/website/src/app/docs/concepts/page.mdx b/website/src/app/docs/concepts/page.mdx index 9c3cc6c..24717c0 100644 --- a/website/src/app/docs/concepts/page.mdx +++ b/website/src/app/docs/concepts/page.mdx @@ -6,7 +6,7 @@ export const metadata = { # Concepts -This page explains how loopy-loop is put together and why. Once you have the mental model — a coordinator that decides, workers that execute, and a session directory that remembers — the configuration and workflow pages read as details rather than surprises. +This page explains how loopy-loop is put together and why. Once you have the mental model — a coordinator that decides, one worker that executes, and a session directory that remembers — the configuration and workflow pages read as details rather than surprises. ## Coordinator and worker @@ -14,13 +14,13 @@ loopy-loop separates *deciding what to do next* from *doing it*. Those are two l The **coordinator** is a small FastAPI server. It owns the loop state, loads your `loopy_loop_config.yaml`, resolves the goal, and creates a session. On each request it chooses the next workflow to run based on scheduling rules and the history so far, records results, checks the session's control and eval artifacts, and decides whether to continue or stop. The coordinator never runs agents itself. -A **worker** is a blocking executor. It asks the coordinator for a task, renders that workflow's prompt with the concrete session paths, runs it, writes the outputs into the iteration directory, and reports back. Then it asks for the next task. A worker holds no loop state of its own — if it crashes, the coordinator notices the orphaned task on the next check-in and dispatches fresh work. You can run several workers against one coordinator. +A **worker** is a blocking executor. It asks the coordinator for a task, renders that workflow's prompt with the concrete session paths, runs it, writes the outputs into the iteration directory, and reports back. Then it asks for the next task. Exactly one worker owns assignments for a coordinator. If its task is interrupted, the next registration first recovers any completed result or handles identity-tracked orphaned agent processes before recording an abandonment and returning to normal scheduling; a stop condition may end the session instead. A second worker is refused while the recorded local owner is verifiably alive. This split is what makes the loop durable and inspectable: the coordinator's decisions live in files, and the worker's work products live in files, so nothing important is trapped inside a process. ## Delegation to team-harness -Workers do not talk to model providers directly. They run each assignment through [`team-harness`](https://github.com/writeitai/team-harness), the orchestration layer that manages a coordinator model and can spawn external agent CLIs such as Codex, Claude Code, and Gemini as worker subprocesses. +The loopy worker does not talk to model providers directly. It runs each assignment through [`team-harness`](https://github.com/writeitai/team-harness), the orchestration layer that manages a coordinator model and can spawn external agent CLIs such as Codex, Claude Code, and Gemini as worker subprocesses. Concretely, the worker calls `TeamHarness.run(...)` with the rendered workflow prompt. team-harness runs its own coordinator model, optionally delegates to one or more agent CLIs, and returns a result. loopy-loop then normalizes and stores that result. Everything about *which* provider, model, and agents team-harness uses comes from the `team_harness_*` fields in your config — see [Configuration](/docs/configuration). @@ -52,7 +52,7 @@ Because state is files, you can read it, diff it, resume from it, and audit it. ## The two-endpoint model -At a conceptual level, the coordinator and worker communicate with a simple ping-pong: the worker asks once for a task, then reports each completed task and receives the next one, until it is told to stop. There is no polling, no leases, and no worker identity to track — every response tells the worker either to run a specific workflow or to stop. +At a conceptual level, the coordinator and worker communicate with a simple ping-pong: the worker asks once for a task, then reports each completed task and receives the next one, until it is told to stop. There is no polling, lease scheduler, or worker pool. The worker does send process identity, which the coordinator stores on the live task solely for ownership and crash-recovery liveness checks; every response still tells it either to run a specific workflow or to stop. That is all you need to hold in mind here. The exact JSON payloads, the two endpoints, and crash-recovery behavior are documented in the [HTTP Contract](/docs/http-contract). diff --git a/website/src/app/docs/configuration/page.mdx b/website/src/app/docs/configuration/page.mdx index 971f7bd..0341550 100644 --- a/website/src/app/docs/configuration/page.mdx +++ b/website/src/app/docs/configuration/page.mdx @@ -19,6 +19,7 @@ goal_file: loopy_loop_goal.txt workflow_set: inner_outer_eval max_turns: 160 goal_check_consecutive_failures_cap: 3 +workflow_consecutive_failures_cap: 5 team_harness_provider: "codex" team_harness_model: "gpt-5.5" team_harness_agents: @@ -33,6 +34,11 @@ team_harness_agent_reasoning_efforts: codex: "high" team_harness_api_base: "https://openrouter.ai/api/v1" team_harness_api_key_env: "OPENROUTER_API_KEY" +# Optional coordinator-model cost estimate and stop threshold: +# model_prices: +# prompt_usd_per_1m: 2.5 +# completion_usd_per_1m: 10.0 +# max_cost_usd: 50.0 ``` ## Core fields @@ -45,6 +51,7 @@ These fields control the goal, the workflow set, and how long the loop runs. | `workflow_set` | string | required | The workflow set used for new sessions when the coordinator is started without an override. | | `max_turns` | integer | required | Maximum number of completed workflow iterations before the loop stops. | | `goal_check_consecutive_failures_cap` | integer | `3` | How many consecutive invalid or missing goal-check outputs are tolerated before the loop stops with `goal_check_broken`. Must be at least `1`. | +| `workflow_consecutive_failures_cap` | integer | `5` | Consecutive failures allowed for one workflow before the coordinator stops with `workflow_failure_cap`. A success of that workflow resets its counter. | The goal must live in a file. See the [Success & Control](/docs/success-and-control) page for how `max_turns` and the failure cap participate in stopping the loop. @@ -59,6 +66,8 @@ Everything about *how* work is executed — which provider, coordinator model, a | `team_harness_agents` | list of strings | `["codex"]` | Agent names team-harness should make available as worker subprocesses, such as `codex`, `claude`, and `gemini`. | | `team_harness_agent_models` | map | `{}` | Per-agent default worker model overrides, keyed by agent name. | | `team_harness_agent_reasoning_efforts` | map | `{}` | Per-agent reasoning-effort overrides, keyed by agent name. Only agents whose templates support a reasoning-effort flag use this value. | +| `model_tiers` | map | `{}` | Coordinator-side tier name → agent → `{model, effort}` bundles rendered into the harness prompt for per-spawn worker choice. Tier selection is guidance and audit evidence, not enforcement. | +| `default_tier` | string | unset | A `model_tiers` key whose bundles derive the per-agent default model/effort maps. When set, remove explicit `team_harness_agent_models` and `team_harness_agent_reasoning_efforts`. | | `team_harness_api_base` | string | `"https://openrouter.ai/api/v1"` | The OpenAI-compatible API base URL passed to team-harness. Normalized on load (see below). | | `team_harness_api_key_env` | string | `"OPENROUTER_API_KEY"` | Name of the environment variable that holds the API key. | | `team_harness_system_prompt_extension` | string | `""` | Extra system-prompt text appended for every harness run. The templates use this to state session-state and PR/merge policy. | @@ -72,11 +81,15 @@ These optional fields tune how team-harness retries transient API and network er | `team_harness_max_retries` | integer | unset | Coordinator retry budget. Must be `0` or greater. | | `team_harness_retry_base_delay_s` | number | unset | Base delay in seconds for retry backoff. Must be greater than `0`. | | `team_harness_retry_max_delay_s` | number | unset | Maximum delay in seconds for retry backoff. Must be greater than `0`, and no less than `team_harness_retry_base_delay_s`. | -| `recovery_policy` | string | `drain` | Coordinator-side (not sent to workers). What crash recovery does with agent processes a dead worker left running: `drain` lets them finish (one shared `recovery_drain_timeout_s` deadline) before the iteration is re-run; `reap` kills them immediately. | +| `recovery_policy` | string | `drain` | Coordinator-side (not sent to workers). What crash recovery does with identity-tracked agent processes: `drain` lets them finish within one shared `recovery_drain_timeout_s` deadline; `reap` kills them immediately. Recovery records abandonment, then normal scheduling continues only if no stop condition fires. | | `recovery_drain_timeout_s` | number | `600` | Coordinator-side. Shared deadline in seconds for draining ALL of an iteration's orphaned agents during crash recovery. Must be `0` or greater. | The `inner_outer_eval` template ships these as commented-out examples so you can see the shape before enabling them. +### Usage and cost controls + +`model_prices` supplies `prompt_usd_per_1m` and `completion_usd_per_1m` for the harness coordinator model. With it configured, status can estimate cost from known coordinator-token usage. `max_cost_usd` is an optional positive stop threshold and is rejected unless prices are present. Agent-CLI subprocess spend is not measurable and is never included; missing or partial usage remains explicitly unknown. These fields are coordinator-side and reload when a coordinator resumes rather than entering the worker config snapshot. + ### Optional criteria fields The config also accepts two optional list fields, `completion_criteria` and `stop_criteria`, both defaulting to an empty list. They record observable criteria for reference by your workflows; the loop's actual stop decision is driven by `control.json`, not by these lists. Most projects keep completion criteria in the goal file instead. @@ -109,6 +122,8 @@ The root config is parsed strictly, which turns silent mistakes into loud, early - **`goal_file` must resolve to a non-empty file.** It is read relative to the config file, and an empty goal is rejected. - **`workflow_set` must not be empty.** - **Agent maps require non-empty keys and values.** Empty strings in `team_harness_agent_models` or `team_harness_agent_reasoning_efforts` are rejected. +- **Tier declarations are checked.** Tier/agent names and models must be non-empty single lines, tier agents must appear in `team_harness_agents`, and `default_tier` must cover every configured agent. A default tier conflicts with explicit per-agent model/effort maps. +- **Cost thresholds require prices.** `max_cost_usd` without `model_prices` is rejected because the threshold could never be evaluated honestly. - **Retry bounds are checked.** `team_harness_retry_max_delay_s` must be at least `team_harness_retry_base_delay_s`. ## Where to go next diff --git a/website/src/app/docs/getting-started/page.mdx b/website/src/app/docs/getting-started/page.mdx index 1e8223c..abee0d6 100644 --- a/website/src/app/docs/getting-started/page.mdx +++ b/website/src/app/docs/getting-started/page.mdx @@ -76,7 +76,7 @@ The goal must live in a file — inline `goal:` values in `loopy_loop_config.yam ## Run the loop -loopy-loop runs as two processes: a coordinator that owns the loop state and one or more workers that execute assignments. Start them in separate terminals. +loopy-loop runs as two processes: a coordinator that owns the loop state and one worker that executes assignments. Start them in separate terminals. The default templates use `team_harness_provider: "codex"`, which relies on your local Codex authentication. If you switch to an OpenAI-compatible provider, export the environment variable named in `team_harness_api_key_env` (usually `OPENROUTER_API_KEY`) in **both** the coordinator and worker shells. @@ -92,7 +92,7 @@ Start a worker in another terminal, pointing it at the coordinator: loopy worker --coordinator http://127.0.0.1:8080 ``` -The worker asks the coordinator for a task, runs it through `team-harness`, reports the result, and repeats until the coordinator tells it to stop. Multiple workers can share one coordinator. +The worker asks the coordinator for a task, runs it through `team-harness`, reports the result, and repeats until the coordinator tells it to stop. Exactly one worker owns assignments for a coordinator; a second worker is refused while the recorded worker is verifiably alive. ## Monitor, stop, and resume @@ -100,10 +100,12 @@ From the repo root, check on the run or ask it to wind down: ```bash loopy status +loopy status --watch +loopy events --follow loopy stop ``` -`loopy status` prints the latest session id, iteration count, current task, and stop reason. `loopy stop` requests a graceful stop; workers exit after their next check-in. +`loopy status` prints the active session stack, current tasks, usage, and estimated cost; `--watch` refreshes it every two seconds. `loopy events --follow` tails the deepest active session. `loopy stop` requests a graceful stop; while a child runs, the top-level stop takes effect after that child terminates and the parent resumes. If the coordinator process dies while a session is still running, the state file stays marked `running`. Reattach to it with `--resume`: diff --git a/website/src/app/docs/http-contract/page.mdx b/website/src/app/docs/http-contract/page.mdx index d55cef4..e862d77 100644 --- a/website/src/app/docs/http-contract/page.mdx +++ b/website/src/app/docs/http-contract/page.mdx @@ -18,15 +18,15 @@ Every response from either endpoint is a `TaskResponse` whose `action` is either `"run"` or `"stop"`. - A **run** response carries `workflow_set`, `workflow_id`, `session_id`, - `iteration`, and a `config_snapshot`. The worker loads the named workflow, renders - its prompt, and executes it. + `iteration`, `attempt_id`, and a `config_snapshot`. The worker loads the named + workflow, renders its prompt, and executes it. - A **stop** response carries only a `stop_reason`; the other fields are `null`. The worker exits. ## POST /register -A worker calls `/register` to receive its first assignment. The request body is -empty. +A worker calls `/register` with its process identity to receive its first +assignment. Request — the worker's process identity is **required** (a breaking change in 0.3; pre-0.3 workers are rejected with HTTP 400): @@ -53,6 +53,7 @@ Run response: "workflow_id": "planner", "session_id": "20260419_143022_71393ee22450_ab12cd34", "iteration": 3, + "attempt_id": "a1b2c3d4e5f6", "config_snapshot": { "goal": "Ship a minimal working landing page", "goal_hash": "71393ee22450", @@ -87,6 +88,7 @@ Stop response: "workflow_id": null, "session_id": null, "iteration": null, + "attempt_id": null, "config_snapshot": null } ``` @@ -99,20 +101,23 @@ Rules for `/register`: - `workflow_set` tells the worker which `.loopy_loop/workflow_sets//workflows//` directory to load. -- If a task is already current — meaning the previous worker crashed without calling - `/finished` — `/register` first verifies whether the recorded worker is *actually - dead*: a verifiably-alive worker (same host, matching pid + start-time token, and - not a zombie) gets **HTTP 409** instead of having its task reclaimed. If that - worker is hung, kill its process (the 409 names the pid) and register again. For a - confirmed-dead or unverifiable worker, the coordinator checks that iteration +- If a task is already current — meaning the prior assignment remains + unacknowledged — `/register` first checks the recorded worker's liveness. A + verifiably-alive worker (same host, matching pid + start-time token, and not a + zombie) gets **HTTP 409** instead of having its task reclaimed. If that worker is + hung, kill its process (the 409 names the pid) and register again. When the worker + is not verifiably alive, the coordinator checks that iteration directory for `pending_finished_request.json` or `result.json`. If either proves the task completed, the coordinator records the completed task in history before checking stop conditions. A task with no recoverable local completion has its orphaned agent processes drained or reaped per `recovery_policy` (a `salvage.json` - records what was handled) and is recorded as failed with - `error="abandoned_after_"` (or plain `"abandoned"` when nothing settled). - If any orphan may still be running after recovery, `/register` returns **HTTP - 409** rather than dispatching a second writer. See + records processed-run reports) and is recorded as failed with + `error="abandoned_after_"` when something settled (or plain + `"abandoned"` otherwise). For an identity-tracked harness run, an unsettled + reaper outcome makes `/register` return **HTTP 409** rather than dispatching a + second writer. A remote loopy worker cannot provide the same local proof + because the coordinator cannot reach its processes. + See [Crash recovery](#crash-recovery-and-stale-retries). - If the loop is already in a terminal state, `/register` immediately returns `action: "stop"`. @@ -148,17 +153,37 @@ Request: "iteration": 3, "success": true, "text": "done", - "error": null + "error": null, + "failure_kind": null, + "usage": { + "prompt_tokens": 5210, + "completion_tokens": 902, + "turns": 3, + "turns_without_usage": 0 + }, + "duration_s": 187.4, + "attempt_id": "a1b2c3d4e5f6", + "worker": { + "hostname": "buildbox", + "pid": 4242, + "starttime": "lstart:..." + } } ``` +`attempt_id` must echo the live task's value. `worker` identifies the caller +that will own the next task. `failure_kind`, `usage`, and `duration_s` are +optional; missing usage means unknown, never zero. + Response: same shape as the `/register` response (`action` is `"run"` or `"stop"`). Rules for `/finished`: - **Match required.** The report must match the current task on `session_id`, - `workflow_id`, and `iteration`. Only then does the coordinator record the result, - read the session artifacts, and dispatch the next task. + `workflow_id`, `iteration`, and the live `attempt_id`. Only then does the + coordinator record the result, read session artifacts, and dispatch the next + task. A stale call from a different identified worker gets HTTP 409 rather + than receiving the live owner's task. - **`control.json` and `goal_check.json`.** After a matched finish, the coordinator reads `control.json` only from the session directory, and reads `goal_check.json` only from the current iteration directory when the workflow is `goal_check` or has @@ -177,9 +202,10 @@ The contract is designed so that a killed worker, a duplicate retry, or a lost acknowledgment never corrupts loop state. **Stale finish (mismatch).** If a `/finished` call's `session_id` + -`workflow_id` + `iteration` does not match the current task, the call is stale. The -coordinator does **not** mutate state and does **not** change the current task; it -returns the current task's run response so the caller sees what is actually running. +`workflow_id` + `iteration` + live `attempt_id` does not match the current task, +the call is stale. The coordinator does **not** mutate state or change the +current task. It returns the current task only to that task's recorded owner (or +for a pre-identity legacy task); a different identified caller gets HTTP 409. **Attempt ids.** Every dispatched task carries a unique `attempt_id`, echoed by the worker on `/finished`. A call from a superseded attempt — same @@ -206,22 +232,21 @@ pending file is missing) and records the task as completed — rather than marki `abandoned`. A locally written result is treated as authoritative. The per-iteration files behind this are described in [Session Layout](/docs/session-layout). -**Worker crash mid-run.** With nothing recoverable, `/register` applies the -configured recovery policy to any agent processes the dead worker's harness run -left behind — by default **bounded drain**: let in-flight agents finish (their -completed repo edits survive in the working tree; git is the source of truth), -then re-run the iteration from that better starting point. One +**Worker task interrupted mid-run.** With nothing recoverable, `/register` +applies the configured recovery policy to identity-tracked agent processes the +prior worker's harness run left behind — by default **bounded drain**: let +in-flight agents finish while their edits survive in the working tree. One `recovery_drain_timeout_s` deadline is shared across all of the iteration's interrupted runs. A `salvage.json` in the interrupted iteration directory records -what was drained or reaped, so surviving edits are auditable rather than a mystery -diff. The iteration's own result is never fabricated. Recovery runs outside the -state lock (`loopy status`/`stop` stay usable), but `/register` itself can block -roughly up to the drain deadline plus kill grace periods — the bundled worker uses -an unbounded read timeout on `/register` only. Process recovery is same-host-only: -a worker identity from another hostname skips reaping. If any orphan may still be -running afterwards (unverifiable identity, probe failure, or a kill that did not -land), the coordinator refuses to dispatch replacement work (HTTP 409) and the -salvage record names the unresolved processes. This requires a team-harness version -with the process reaper — older versions skip orphan recovery and fall back to -plain abandonment. The recovery settings are coordinator-side only; they are not -part of the wire `config_snapshot`. +what the reaper reported, so surviving edits are auditable rather than a mystery +diff. The task's result is never fabricated: abandonment consumes a turn, after +which normal scheduling continues only if no stop condition fires. Recovery runs +outside the state lock (`loopy status`/`stop` stay usable), but `/register` itself +can block roughly up to the drain deadline plus kill grace periods — the bundled +worker uses an unbounded read timeout on `/register` only. Process recovery is +same-host-only. For an identity-tracked harness run, an unsettled reaper outcome +refuses replacement work (HTTP 409) and is captured in `salvage.json`. A remote +loopy worker skips reaping and falls back to legacy abandonment; when local +worker liveness is unverifiable, team-harness recovery is still attempted if its +run data supports it. The recovery settings are coordinator-side only and are +not part of the wire `config_snapshot`. diff --git a/website/src/app/docs/page.mdx b/website/src/app/docs/page.mdx index 786962d..1c12344 100644 --- a/website/src/app/docs/page.mdx +++ b/website/src/app/docs/page.mdx @@ -8,7 +8,7 @@ export const metadata = { `loopy-loop` runs long-running AI agent workflows inside your repository. It turns a goal file into an inspectable sequence of agent iterations — plan, implement, evaluate, record evidence, and continue — until the goal is met or the loop hits a terminal blocker. -It is a small Python CLI plus a [FastAPI](https://fastapi.tiangolo.com/) coordinator and one or more workers. The coordinator owns the loop state and decides the next workflow; the workers execute assignments through [`team-harness`](https://github.com/writeitai/team-harness), which can delegate to agent CLIs such as Codex, Claude Code, and Gemini. +It is a small Python CLI plus a [FastAPI](https://fastapi.tiangolo.com/) coordinator and one worker. The coordinator owns the loop state and decides the next workflow; the worker executes assignments through [`team-harness`](https://github.com/writeitai/team-harness), which can delegate to multiple agent CLIs such as Codex, Claude Code, and Gemini. ## Why loopy-loop diff --git a/website/src/app/docs/session-layout/page.mdx b/website/src/app/docs/session-layout/page.mdx index 9d1654d..3dfddf6 100644 --- a/website/src/app/docs/session-layout/page.mdx +++ b/website/src/app/docs/session-layout/page.mdx @@ -32,10 +32,15 @@ A session directory looks like this: .loopy_loop/ └── sessions/ └── / + ├── goal.md ├── session.json + ├── state.json ├── events.jsonl ├── control.json ├── updates_from_user.md + ├── children.json + ├── child_requests/ + ├── children/ ├── project_state/ │ └── finished.md ├── eval_checks/ @@ -62,16 +67,26 @@ A session directory looks like this: └── goal_check.json ``` -Sessions that spawn child work also grow a `child_requests/` inbox and a -`children/` directory; those are covered on [Child Sessions](/docs/child-sessions). +The `children/` directory is populated only when the session dispatches child +work; these files are covered on [Child Sessions](/docs/child-sessions). ## Session files +**`goal.md`** — The exact resolved goal text for this session. The rendered Goal +and this session-local copy are canonical for active work, including child goals +and `--goal-file` overrides. + **`session.json`** — Session metadata, written once when the directory is created. It contains `session_id`, `goal_hash`, and `created_at`. -**`events.jsonl`** — A reserved append-only event log for diagnostics, created at -session start in v1. +**`state.json`** — Coordinator-owned durable dispatch state, history, stop state, +active-child pointer, and usage ledger. This is the source of truth behind the +best-effort event stream. + +**`events.jsonl`** — A best-effort, append-only stream of versioned lifecycle +events. It is written after the durable state mutation, so consumers must +tolerate gaps, duplicates, and a torn final line; `state.json` remains the source +of truth. Tail it with `loopy events --follow`. **`control.json`** — The session-scoped workflow stop switch, created with `state: "running"` when the session starts. Workflows update it only when they want @@ -85,9 +100,9 @@ planning input, and clears the file only after reflecting the request into `project_state/`. **`project_state/`** — Optional workflow-owned durable markdown state for reusable -workflows. `loopy_loop_goal.txt` remains the source of truth for the target, -constraints, and completion intent; do not copy or restate the goal into -`project_state/`. Common files include `README.md`, `memory.md`, +workflows. The rendered Goal and session-local `goal.md` are canonical for the +active session (especially for a child or `--goal-file` override); do not copy or +restate the goal into `project_state/`. Common files include `README.md`, `memory.md`, `current_state.md`, `what_we_have.md`, `decisions.md`, `eval_results.md`, `finished.md`, and `what_we_should_do/plan.md`. The `README.md` should document ownership rules — `memory.md` for essential durable facts only, `finished.md` for @@ -135,6 +150,15 @@ team-harness then creates a child directory named with its own run id: .loopy_loop/sessions//harness_outputs/_// ``` +**`children.json`, `child_requests/`, and `children/`** — `child_requests/` is +the inbox for pending requests, `children/` holds dispatched child sessions, +and `children.json` indexes their durable status and outcomes. While a child is +active, the parent's `state.json` carries `active_child_session_id`, which is +the pointer followed during restart. A request filename suppresses redispatch +only while its child record is `running` or `dispatching`; once that child is +complete, the filename may be reused for new work. Final child records include +whole-subtree usage for the parent's ledger and cost budget. + ## Iteration files Each iteration gets its own directory under `iterations/`, named @@ -175,25 +199,17 @@ task instead of marking it `abandoned`; if it is missing but `result.json` exist for the active task, the coordinator reconstructs the finished request from `result.json`. See [HTTP Contract](/docs/http-contract) for the recovery rules. -**`children.json`** (parent sessions) — Index of child sessions dispatched by -this session. Each record carries the child `session_id`, `workflow_set`, -`status`, timestamps, `stop_reason`, and the originating `request_file` name, -which makes the dispatch scan idempotent — a request whose filename already -appears here is never dispatched twice, even if a crash left the file behind. -While a child runs, the parent's `state.json` records -`active_child_session_id`: the durable session-stack pointer a restarted -coordinator follows to resume the child instead of orphaning it. - -**`salvage.json`** — Written into the interrupted iteration's directory during -crash recovery, when the coordinator drained or reaped agent processes a dead -worker's harness run left behind (per `recovery_policy`). It records the reap -reports so the provenance of any surviving working-tree edits is auditable rather -than a mystery diff. The iteration is still re-run — its `result.json` never -existed and is never fabricated — and when any orphan actually settled, its -history entry carries `error="abandoned_after_"` (e.g. -`abandoned_after_drain`) instead of plain `"abandoned"`. A non-zero -`unsettled_workers` in the record means some orphan may still be running; the -coordinator refuses to dispatch replacement work (HTTP 409) until it is resolved. +**`salvage.json`** — Written into the interrupted iteration's directory when +crash recovery processes at least one tracked harness run. It records reaper +reports so the provenance of surviving working-tree edits is auditable rather +than a mystery diff. The task's `result.json` never existed and is never +fabricated; abandonment consumes a turn, then normal scheduling continues only +if no stop condition fires. When any orphan settled, history carries +`error="abandoned_after_"` instead of plain `"abandoned"`. For an +identity-tracked run, a non-zero `unsettled_workers` means some orphan may still +be running and replacement dispatch is refused (HTTP 409). Remote or otherwise +unreachable processes can fall back to legacy abandonment without that local +proof. **`goal_check.json`** — A per-iteration eval artifact, present only for the reserved `goal_check` workflow or a workflow configured with diff --git a/website/src/app/docs/troubleshooting/page.mdx b/website/src/app/docs/troubleshooting/page.mdx index e39ed95..bc47764 100644 --- a/website/src/app/docs/troubleshooting/page.mdx +++ b/website/src/app/docs/troubleshooting/page.mdx @@ -37,7 +37,7 @@ output path, **and** it updates `control.json` when the goal is met. The eval ve is evidence; the stop is a separate, explicit decision. See [Success & Control](/docs/success-and-control) for the full model. -You can always force a stop by hand with `loopy stop`, which the loop records as +You can request a graceful stop by hand with `loopy stop`, which the loop records as `stop_reason: "stop_requested"`. ## The loop stopped with goal_check_broken @@ -141,15 +141,19 @@ or run `loopy stop` first to reach a terminal state before a fresh start. **Symptom.** A worker process died in the middle of an assignment, and you are not sure whether that iteration's work was lost or will be double-counted. -**Cause.** The worker crashed inside the handoff window — after writing its result -but before the coordinator acknowledged `/finished`. +**Cause.** The worker may have stopped before publishing a result, or inside the +handoff window after writing its result but before the coordinator acknowledged +`/finished`. **Fix.** Usually nothing. The loop is built to recover: when the next `/register` arrives, the coordinator looks in the current iteration directory for `pending_finished_request.json` (or, failing that, `result.json`). If either proves -the task completed, it records the completed result instead of marking the task -`abandoned`. Only a task with no recoverable local result is recorded as failed with -`error="abandoned"`. Just start a worker again and let it re-register. The +the task completed, it records the completed result. A task with no recoverable +local result enters agent-process recovery: settled cleanup records +`abandoned_after_`, no settled process records plain `abandoned`, and an +unsettled identity-tracked process makes registration return HTTP 409 instead of +risking a second writer. Start the worker again and let it re-register; if it gets +409, resolve the process named by the recovery report before trying again. The [HTTP Contract](/docs/http-contract) documents the recovery rules in full. ## The worker cannot reach the coordinator @@ -170,7 +174,8 @@ loopy worker --coordinator http://127.0.0.1:8080 Also export the `team_harness_api_key_env` variable in the worker shell, not just the coordinator shell — each process resolves the key independently. The worker calls `/register` once for its first task, then loops on `/finished` until it -receives `action: "stop"`; multiple workers may share one coordinator. +receives `action: "stop"`. Run exactly one worker per coordinator; a second is +refused while the recorded worker is verifiably alive. ## A workflow ignores project_state