Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 46 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,11 @@ Useful control commands:

```bash
loopy status
loopy status --json
loopy update Prioritize the failing integration test
loopy stop
loopy stop --force
loopy reload
loopy traces list
```

Expand Down Expand Up @@ -406,6 +409,21 @@ Important rules:
COORDINATOR model only — agent-CLI subprocesses (codex, claude, gemini)
bill through their own accounts and are not measurable here.

An explicit `loopy reload` refreshes the running coordinator at the next task
boundary. It reloads workflow `prompt.txt` contents and these non-frozen,
coordinator-operational root settings: `recovery_policy`,
`recovery_drain_timeout_s`, `workflow_consecutive_failures_cap`,
`max_cost_usd`, and `model_prices`. The update is atomic across the
coordinator's cached workflow sets.

Reload deliberately does **not** change a live session's frozen goal or
`config_snapshot` (including harness provider/model/families, retry settings,
system prompt extension, criteria, and turn limit), its workflow membership or
`config.yaml` cadence, its workflow contract/roster, or its capability roster.
Those changes require a coordinator restart and a new session. A later child
workflow set that has never been loaded still freezes its on-disk definition
when that child session is created.

Workflow config lives beside each workflow prompt:

```yaml
Expand Down Expand Up @@ -651,25 +669,47 @@ loopy worker --coordinator http://127.0.0.1:8080
Runs a blocking worker until the coordinator returns `action: "stop"`.

```bash
loopy status # session stack, usage totals, estimated cost
loopy status # stack, liveness, family health, usage, estimated cost
loopy status --json # the same status as machine-readable JSON
loopy status --watch # re-render every 2 seconds
loopy events # the active session's event stream
loopy events --follow # tail it live (--json for raw lines)
loopy update TEXT... # append to the deepest active layer
loopy update --session SESSION_ID TEXT...
loopy stop # tree-wide stop at the next safe boundary
loopy stop --force # also reap the active iteration's tracked agent CLIs
loopy reload # refresh prompts/operational config at the next boundary
loopy traces list
loopy traces inspect MANIFEST_OR_ID
```

`status` prints the latest session state — the whole session stack while a
child runs (the live child is shown under its suspended parent), each
session's subtree token usage, and (with `model_prices` configured) estimated
cost. `update` appends the input record exactly as supplied; without `--session` it is
routed to the deepest active layer and later assignments append delivery and
acknowledgement records rather than editing history. `stop` projects root stop
intent through the whole active path and takes effect at the next register or
finish boundary; it does not invent a mid-harness interruption mechanism.
cost. For an active task it also reports the newest mtime anywhere in that
attempt's raw/output tree as `last activity: Ns ago`, plus unexpired
`rate_limited_families` found in the current team-harness `run.json` as
`model families rate-limited: FAMILY until TIME`. Missing, incomplete, or
older run records are tolerated. `status --json` exposes per-session
`last_activity_at`, `last_activity_age_s`, `rate_limit_data_available`, and
`rate_limited_families` fields.

`update` appends the input record exactly as supplied; without `--session` it
is routed to the deepest active layer and later assignments append delivery
and acknowledgement records rather than editing history. `stop` projects root
stop intent through the whole active path and takes effect at the next
register or finish boundary. `stop --force` first records that same durable
tree-wide intent, then invokes the existing recovery reaper with immediate
`reap` policy and reports harness-run, settled-agent, and unsettled-agent
counts. Force reaping is limited to tracked process groups on the local host;
a remote active worker is reported as unreachable rather than pretending its
agents were stopped.

`reload` writes an ignored, atomic reload generation under the sessions root.
At the next task boundary, the coordinator reruns preflight and atomically
refreshes only the prompt/operational fields described in Configuration above.
It does not mutate an already-frozen attempt; the following attempt receives
the refreshed prompt in its own immutable workflow snapshot.

Trace commands accept a manifest ID, a trace root, or a manifest path confined
to this repository's `.loopy_loop/traces/`. `inspect` prints the manifest plus
Expand Down
9 changes: 7 additions & 2 deletions design/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,8 +234,10 @@ own:
periodic heartbeat; liveness is checked directly against the stored identity.
**Default: bounded drain** — let an in-flight agent finish within a timeout (fits loopy's
git-is-truth, cost-conscious profile; no concurrent-writer problem because it runs during
recovery), with **reap** as the escape for hung-past-timeout or unsafe-to-finish work. A
future force-stop command must reuse this cleanup path rather than inventing another one.
recovery), with **reap** as the escape for hung-past-timeout or unsafe-to-finish work.
`loopy stop --force` reuses this exact cleanup path with team-harness's explicit
live-parent override after recording tree-wide stop intent; it does not implement a
second process killer.
A drained iteration is **never accepted or synthesized from drained outputs**: its
`result.json` never existed, and fabricating one would trigger the false-closure trap D3
prevents. If stop conditions still allow the session to continue, the scheduler dispatches
Expand Down Expand Up @@ -267,6 +269,9 @@ drain, reap as escape). For identity-tracked same-host runs this makes D6's stat
verify-dead-before-reclaim rather than optimistic; legacy and remote identities retain the
documented limitation above. The implemented protocol and salvage boundary are described in
`designs/long-running-loop-reliability.md`; the process-group mechanism is team-harness TH-D5.
Force-stop has the same host and durable-run-record boundary: it reports an unreachable remote
worker or zero discovered runs honestly instead of claiming that untracked processes were
terminated.

## D8. Semantic constraints are visible detection with accountable repair or disposition, never hard prevention

Expand Down
38 changes: 38 additions & 0 deletions design/designs/long-running-loop-reliability.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,16 @@ workflow is selected next.
dead-worker reclaim, stale-owner completions, bounded recovery, unsettled-writer
refusal, and salvage records.

`loopy stop --force` is the operator-triggered form of the same mechanism. It
first projects durable stop intent through the active session stack, then calls
`recover_interrupted_iteration()` with immediate `reap` policy and
team-harness's explicit live-parent override. This terminates the current
iteration's tracked same-host agent process groups without teaching Loopy a
second process-killing implementation. The command prints processed-run and
settled/unsettled-agent counts. A remote worker cannot be signalled from the
coordinator host and is reported as unreachable; absent durable run records are
reported as zero processed runs rather than a false guarantee.

---

## The planner/dispatcher template is executable from a clean init
Expand Down Expand Up @@ -276,6 +286,34 @@ The CLI makes that projection useful without adding a dashboard:
These commands live in `src/loopy_loop/cli.py`; stream and CLI behavior are
covered by `src/tests/test_events_and_usage.py`.

For the deepest active task, `loopy status` also scans the current folded raw
directory (or the legacy mirror/output location) for the newest mtime. The age
is an observation, not a heartbeat guarantee: a quiet but computing process can
have an old timestamp, while a recently written file proves only recent I/O.
The same best-effort scan reads unexpired `rate_limited_families` entries from
caller-owned team-harness `run.json` files. Missing or partially written data is
treated as unavailable, never as a coordinator failure. `status --json`
publishes the underlying timestamp, age, data-availability flag, and family
records.

### Explicit preflight-cache reload keeps frozen session policy intact

`loopy reload` atomically writes a generation marker below the ignored sessions
root. `CoordinatorService._reload_preflights_if_requested()` notices a new
generation at the next task boundary, validates fresh preflight data for every
cached workflow set, and swaps all refreshed caches together. The next attempt
therefore freezes the current workflow prompt instead of the startup copy.

The reload boundary follows the existing durable contracts. It may refresh
workflow prompt text and the root settings used only by the coordinator:
recovery policy/timeout, workflow failure cap, cost budget, and model prices.
It preserves workflow membership and `config.yaml` cadence because
`workflow_roster.json` is session-frozen. It also preserves the session's goal,
workflow contract, worker `config_snapshot`, model/retry/system-prompt policy,
and tree capability roster. Those values change only after restart/new-session
creation; descendants continue inheriting the parent's frozen execution
snapshot.

### Unknown usage stays unknown

The worker's `_read_harness_usage()` reads coordinator-model turn usage from
Expand Down
14 changes: 12 additions & 2 deletions skills/loopy-loop/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -418,17 +418,27 @@ Useful operations:

```bash
loopy status
loopy status --json
loopy status --watch
loopy events --follow
loopy update TEXT...
loopy stop
loopy stop --force
loopy reload
loopy traces list
loopy traces inspect MANIFEST_OR_ID
loopy coordinator --resume
```

`status` and `events` walk the active session stack. `loopy stop` is the
tree-wide stop request and applies at a safe register/finish boundary.
`status` and `events` walk the active session stack. Status includes the active
attempt's last raw/output write and any unexpired model-family rate-limit
circuits; `--json` exposes the same observations as structured fields.
`loopy stop` is the tree-wide stop request and applies at a safe
register/finish boundary. `loopy stop --force` also invokes the existing
same-host process-group reaper for the active iteration. `loopy reload`
refreshes workflow prompts and coordinator-operational recovery/failure/cost
settings at the next task boundary; it never changes a session-frozen config
snapshot, workflow roster/contract, or capability roster.
`--resume` reconstructs the active
path and continues the deepest live session. Exactly one worker is deliberate;
a second verifiably live worker is refused.
Expand Down
Loading
Loading