Skip to content

Commit cae28a8

Browse files
committed
The sweep names the runs holding a project's slots, and the shutdown names what it stops (#1646)
A drain fanned out two agents under a cap of three while the Agents panel showed nothing running: the daemon counted one live process, and a bare count could not be questioned. The process was a card-started triage that had done its work an hour earlier and never exited; the daemon's shutdown line said only "stopped 1 agent(s)". The runtime now reports the slots it holds — run id and pid, or "starting" — instead of a number. The sweep measures the cap against their number as before, and says their names: a stand-down at the cap reads "2 runs are already going (<id> (pid N), …)", a fan-out that came out short reads "started 2 agents alongside 1 already going (<id> (pid N))", and the shutdown line lists the ids it stopped. The "a `starting` key that is never removed" hypothesis in the issue is ruled out by the spawn's `finally`; the next occurrence names itself.
1 parent d874944 commit cae28a8

14 files changed

Lines changed: 184 additions & 40 deletions

FEATURES-SPEC.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ happens while nobody is at the keyboard.
122122
- Auto PM: drain the confirmed queue, refill it when empty
123123
- The routine rotation — update tickets from GitHub, triage quick wins, triage consensual work, plan tickets — plus a calendar-paced maintenance sweep
124124
- Each routine individually switchable off
125-
- Every stand-down reported with its reason ("it is a setting, not a bug")
125+
- Every stand-down reported with its reason ("it is a setting, not a bug"); a stand-down at the concurrency cap, or a fan-out that came out short, names the runs holding the slots
126126
- Concurrency cap: how many unattended agents per project
127127
- Fan-out planning: several agents, one ticket each
128128
- Cross-machine ticket claims so two agents never double-work; a claim whose agent ended with nothing to hand off is freed by the daemon

packages/the-framework/src/auto-pm.SPEC.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Auto PM spends leftover subscription quota on the product's own roadmap: while t
1616
- A claim whose agent settled with nothing to hand off is released by the sweep: the pull request that normally lifts the lock is never coming, and without the release the queue would jam forever on a dead claim. The freed work is not respawned by this daemon — one commitless run is evidence for a human, not an invitation to repeat it every cooldown. A claim whose agent never even started (a refused spawn, a stop mid-batch) is freed the same way.
1717
- The queue coordinates a ticketless entry only once its check-off is on the data branch, and that leaves a window: an agent handed off to a cloud session settles locally before its published work is adopted, so until the check-off lands the entry still reads open, and past the cooldown it can be fanned out to a second agent. The same window opens when the daemon restarts, since only its in-memory pin covered the wait.
1818
- The two triage routines each work on one fixed branch, and their agent refuses to start when that branch already exists, so two triages never run at once. Before firing either — from the schedule or from its "Run now" — the daemon deletes a leftover copy of that branch whose pull request is closed or merged; a branch with an open pull request, or with no pull request history at all, is kept, since that may be a triage still under way.
19-
- Each routine can be switched off individually, and every stand-down is reported with its reason: a wedged sweep must not look like a healthy idle one.
19+
- Each routine can be switched off individually, and every stand-down is reported with its reason: a wedged sweep must not look like a healthy idle one. A pass held back by the concurrency cap, or a fan-out that came out smaller than the cap, names the runs already occupying the slots — run id and process id — because the daemon counts live processes, while the dashboard shows each run's own status, and a process that outlives its finished run is visible only to the count.
2020
- Switching the draining routine off means "do not *work* the queue", not "do nothing": the pass falls through to the rotation, which puts entries *on* the queue rather than taking them off. The one exception is a click that asked for the queue by name: a drain-only sweep says why it cannot, rather than borrowing the click.
2121

2222
## Rationales

packages/the-framework/src/auto-pm.test.SPEC.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Covers auto PM's decision policy — quota and queue both failing closed when unreadable, cooldowns, concurrency caps — the drain-before-refill cycle, rotation order and per-routine opt-outs (an unticked drain routine falls through to the rotation; a drain-only click still stands down), the calendar-paced maintenance sweep, fan-out with pinned entries and locked tickets, drains claiming their entry's ticket before starting (ticketless entries exempt; a lost claim drops the entry, not the batch), durable claims surviving restarts and hand-offs, a claim released when its agent settles with nothing to hand off and only then (the freed work not respawned; refused-spawn claims freed too; failed releases retried under a bound), on-demand and drain-only passes, a pass narrowed to a triage routine releasing its leftover branch before its one start (standing down when the routine is off, and never falling through to the queue or another routine), and the report the dashboard shows.
1+
Covers auto PM's decision policy — quota and queue both failing closed when unreadable, cooldowns, concurrency caps — the drain-before-refill cycle, rotation order and per-routine opt-outs (an unticked drain routine falls through to the rotation; a drain-only click still stands down), the calendar-paced maintenance sweep, fan-out with pinned entries and locked tickets, drains claiming their entry's ticket before starting (ticketless entries exempt; a lost claim drops the entry, not the batch), durable claims surviving restarts and hand-offs, a claim released when its agent settles with nothing to hand off and only then (the freed work not respawned; refused-spawn claims freed too; failed releases retried under a bound), on-demand and drain-only passes, a cap stand-down and a short fan-out naming the runs holding the slots, a pass narrowed to a triage routine releasing its leftover branch before its one start (standing down when the routine is off, and never falling through to the queue or another routine), and the report the dashboard shows.
22

33
## Before modifying/creating SPEC.md files
44

packages/the-framework/src/auto-pm.test.ts

Lines changed: 40 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -169,7 +169,7 @@ function harness(overrides: Partial<AutoPmDeps> = {}) {
169169
// Pinned at one so every test written before #1204 keeps asserting against the behaviour it
170170
// was written for; the fan-out tests set it explicitly.
171171
concurrency: async () => 1,
172-
activeAgents: () => 0,
172+
activeAgents: () => [],
173173
quota: async () => status(1),
174174
start: async (p, job) => {
175175
started.push(p.id)
@@ -210,7 +210,7 @@ test('an on-demand tick sweeps with the preference off: the click is the ask (#1
210210
})
211211

212212
test('on demand skips the master switch and the cooldown: every other stand-down still holds (#1210/#1642)', async () => {
213-
const { loop, started } = harness({ enabled: async () => false, activeAgents: () => 1 })
213+
const { loop, started } = harness({ enabled: async () => false, activeAgents: () => ['run-live (pid 111)'] })
214214
await loop.tick({ onDemand: true })
215215
loop.stop()
216216
assert.deepEqual(started, [])
@@ -547,7 +547,7 @@ test('the report names what a sweep started (#1161)', async () => {
547547
test('the report carries the reason a sweep stood down (#1161)', async () => {
548548
// The whole point: standing down for a reason must not look like quietly working. The reason
549549
// was already logged, but the log is the daemon's stdout and the toggle is in a browser.
550-
const { loop } = harness({ activeAgents: () => 2 })
550+
const { loop } = harness({ activeAgents: () => ['run-a (pid 111)', 'run-b (pid 222)'] })
551551
await loop.tick()
552552
loop.stop()
553553
const [outcome] = loop.report().outcomes
@@ -758,6 +758,41 @@ test('a standing queue fans out to the concurrency in one tick, one entry per ag
758758
assert.equal(new Set(prompts).size, 3)
759759
})
760760

761+
// #1646: the live-agent reading names what it counted. The one time it came out one too high, the
762+
// Agents panel showed nothing running and the number could not be questioned — the run holding the
763+
// slot was a process that had outlived its finished run, visible only to the daemon's own table.
764+
765+
test('a cap stand-down names the runs holding the slots (#1646)', () => {
766+
const capped = autoPmDecision({ ...IDLE, activeAgents: 2, concurrency: 2, running: ['run-a (pid 111)', 'run-b (pid 222)'] })
767+
assert.equal(capped.start, false)
768+
assert.equal(
769+
capped.start === false ? capped.reason : '',
770+
'2 runs are already going (run-a (pid 111), run-b (pid 222)), and the routine keeps at most 2 at once',
771+
)
772+
// Unnamed stays as it was: the count alone is still a complete sentence.
773+
const unnamed = autoPmDecision({ ...IDLE, activeAgents: 1, concurrency: 1 })
774+
assert.equal(unnamed.start === false ? unnamed.reason : '', '1 run is already going')
775+
})
776+
777+
test('a fan-out that came out short says what it was short by, by name (#1646)', async () => {
778+
// Three allowed, one slot held by a run the sweep did not start: two go out, and the card says
779+
// alongside whom, so a held slot nobody can see on the dashboard is named rather than silent.
780+
const { loop, ran } = harness({
781+
cooldownMs: 0,
782+
concurrency: async () => 3,
783+
activeAgents: () => ['2026-08-22T22-06-41-065Z (pid 4242)'],
784+
queue: async () => ['entry a', 'entry b', 'entry c'],
785+
})
786+
await loop.tick()
787+
loop.stop()
788+
assert.equal(ran.length, 2, 'the batch is the cap minus the held slot')
789+
assert.equal(
790+
loop.report().outcomes[0]?.message,
791+
'started 2 agents alongside 1 already going (2026-08-22T22-06-41-065Z (pid 4242)): ' +
792+
'draining the queue entry "entry a"; draining the queue entry "entry b"',
793+
)
794+
})
795+
761796
test('an entry a live run was pinned to is not handed out twice (#1204)', async () => {
762797
// The assignment outlives the tick that made it: the first agent is still working entry a when
763798
// the next sweep comes round, and its queue has not landed yet, so the checkout still shows the
@@ -802,7 +837,7 @@ test('live runs count against the concurrency, so the sweep tops up rather than
802837
const { loop, started } = harness({
803838
cooldownMs: 0,
804839
concurrency: async () => 3,
805-
activeAgents: () => 2,
840+
activeAgents: () => ['run-a (pid 111)', 'run-b (pid 222)'],
806841
queue: async () => ['entry a', 'entry b', 'entry c'],
807842
})
808843
await loop.tick()
@@ -1531,7 +1566,7 @@ test('a switched-off pinned routine stands the click down, and so does every oth
15311566
assert.equal(unknown.loop.report().outcomes[0]?.message, 'no routine is pinned to the-framework/nobody')
15321567

15331568
// The click skips the master switch and the cooldown, not the cap (#1204/#1642).
1534-
const capped = harness({ jobs: [PINNED_JOB], cooldownMs: 0, activeAgents: () => 1 })
1569+
const capped = harness({ jobs: [PINNED_JOB], cooldownMs: 0, activeAgents: () => ['run-live (pid 111)'] })
15351570
await capped.loop.tick({ onDemand: true, only: { pinned: 'the-framework/triage-quick' }, projectId: 'p1' })
15361571
capped.loop.stop()
15371572
assert.deepEqual(capped.ran, [], 'a live agent at the cap holds the click like it holds the sweep')

packages/the-framework/src/auto-pm.ts

Lines changed: 29 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,12 @@ export interface AutoPmInputs {
4040
backlogEmpty: boolean | undefined
4141
/** Live agents on this project, measured against {@link AutoPmInputs.concurrency}. */
4242
activeAgents: number
43+
/**
44+
* What holds each of those slots (#1646), one label per run, for the stand-down's wording. A
45+
* cap reached by a process the Agents panel no longer shows looked exactly like one reached by
46+
* real work; named, the reader can go and look at the run it names.
47+
*/
48+
running?: readonly string[]
4349
/**
4450
* How many agents the routine may keep going on this project at once (#1204);
4551
* {@link DEFAULT_AUTO_PM_CONCURRENCY} when unset. Floored at one, because zero concurrent
@@ -125,7 +131,10 @@ export function autoPmDecision(input: AutoPmInputs): AutoPmDecision {
125131
// The cap is named, for the same reason the quota refusal names its line (#960): with the
126132
// setting raised or lowered, "already going" on its own reads as a bug rather than a setting.
127133
// At one — what this was before #1204 — the old wording is kept exactly.
128-
const going = `${input.activeAgents} run${input.activeAgents === 1 ? ' is' : 's are'} already going`
134+
// Which runs, when the reading says (#1646): the one time this wording mattered, the runs it
135+
// counted were nowhere on the dashboard, and a number could not be questioned.
136+
const named = input.running?.length ? ` (${input.running.join(', ')})` : ''
137+
const going = `${input.activeAgents} run${input.activeAgents === 1 ? ' is' : 's are'} already going${named}`
129138
return { start: false, reason: concurrency === 1 ? going : `${going}, and the routine keeps at most ${concurrency} at once` }
130139
}
131140
const cooldownMs = input.cooldownMs ?? DEFAULT_AUTO_PM_COOLDOWN_MS
@@ -482,8 +491,13 @@ export interface AutoPmDeps {
482491
* entry each (#1204) and the assignment has to come from the read the decision was made on.
483492
*/
484493
queue(project: AutoPmProject): Promise<readonly string[]>
485-
/** How many agents are live on a project. */
486-
activeAgents(project: AutoPmProject): number
494+
/**
495+
* The agents live on a project, one label each (#1646) — the run's id and pid, as the daemon
496+
* holds them. Their number is what the cap is measured against; their names are what the
497+
* stand-down and the fan-out say, so a slot held by a process the dashboard no longer shows
498+
* names itself rather than reading as a bug in the count.
499+
*/
500+
activeAgents(project: AutoPmProject): readonly string[]
487501
/**
488502
* How many agents the routine may keep going per project (#1204). Re-read per tick like
489503
* {@link AutoPmDeps.enabled}, so the setting takes effect without a restart. Unset or unreadable
@@ -807,12 +821,14 @@ export function startAutoPm(deps: AutoPmDeps): AutoPmLoop {
807821
// Per project, because the model the work would run on is (#1619). It costs no reading:
808822
// the meter is polled elsewhere and this only measures the last one against the boundary.
809823
const quota = await deps.quota(project).catch(() => undefined)
810-
const activeAgents = deps.activeAgents(project)
824+
const running = deps.activeAgents(project)
825+
const activeAgents = running.length
811826
const since = lastStart.get(project.id)
812827
const decision = autoPmDecision({
813828
enabled: true,
814829
backlogEmpty: entries === undefined ? undefined : entries.length === 0,
815-
activeAgents: activeAgents,
830+
activeAgents,
831+
running,
816832
concurrency,
817833
quota,
818834
...(since !== undefined ? { sinceLastStartMs: now() - since } : {}),
@@ -1082,7 +1098,14 @@ export function startAutoPm(deps: AutoPmDeps): AutoPmLoop {
10821098
// One line per project however many agents went out, and a single start keeps the old
10831099
// wording exactly.
10841100
const described = started.map(item => doing(item)).join('; ')
1085-
note(project, true, started.length === 1 ? described : `started ${started.length} agents: ${described}`)
1101+
// A fan-out that came out short says what it was short by (#1646): a batch of two under
1102+
// a cap of three, with the panel showing nothing running, used to be unexplainable.
1103+
const alongside = running.length ? ` alongside ${running.length} already going (${running.join(', ')})` : ''
1104+
note(
1105+
project,
1106+
true,
1107+
started.length === 1 ? `${described}${alongside}` : `started ${started.length} agents${alongside}: ${described}`,
1108+
)
10861109
} else if (!stopped) {
10871110
// Nothing took, so the cooldown armed above is given back: a batch that started nothing
10881111
// spent nothing, and holding it would strand the project for a whole cooldown.

packages/the-framework/src/daemon-runtime.SPEC.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ What the daemon does for a project: starting agents in isolated checkouts, retir
1515
- The agents the daemon is still responsible for — spawning, running, or mid-retirement — are named for the background sweep, so it never reclaims a checkout out from under a teardown.
1616
- A finished agent's history is archived onto the data branch (the dedicated branch the framework's shared records live on) — committed and pushed the moment the session settles — and its checkout is reclaimed once the work reaches the remote: the one retention rule, applied whatever the agent did. A push that cannot land keeps the checkout, and the background sweep retries it later.
1717
- An agent killed by a transient connection error is continued automatically (at most twice), and a child that died before booting is marked failed with the cause surfaced — never left "waiting to start" forever.
18-
- On shutdown, live agents are stopped rather than orphaned; each keeps its worktree and branch, so the dashboard can continue the same conversation in the same checkout when asked. A start that lands while the daemon is shutting down is refused rather than spawned into the gap between the stop pass and the server closing, where nothing would ever stop it. The refusal takes back the fresh worktree and branch it had already allocated, which no agent ever owned. An agent can also be forwarded to a connected device, its events relayed back.
18+
- On shutdown, live agents are stopped rather than orphaned, and named in the log as they are stopped — a run the dashboard had long shown as finished turning up here is how a process that outlived its work is found; each keeps its worktree and branch, so the dashboard can continue the same conversation in the same checkout when asked. A start that lands while the daemon is shutting down is refused rather than spawned into the gap between the stop pass and the server closing, where nothing would ever stop it. The refusal takes back the fresh worktree and branch it had already allocated, which no agent ever owned. An agent can also be forwarded to a connected device, its events relayed back.
1919
- Stopping resolves when the daemon has let go of the repo, not when the processes die: a child's exit event lands after its pid disappears, and the teardown that event starts — archive the agent, commit its work, keep or remove its checkout — runs well past that. A teardown that wedges costs the shutdown at most its bounded grace period — it never blocks the exit for good.
2020

2121
## Before modifying/creating SPEC.md files

0 commit comments

Comments
 (0)