Repository navigation
[Feature]: heartbeats wake the coordinator to say nothing changed — make them state, not mail #14910
Description
Problem or use case
Heartbeats are delivered as mail, so a message whose entire content is "nothing has changed" wakes the coordinator.
The wake path. A heartbeat is an ordinary orchestration send, so it goes through notifyMessageArrived. In mailbox-notification-coordinator.ts, waiters are filtered by type; when no waiter matches the arriving type, the code falls into consumers.length === 0 and schedules deliverForHandle, which is the mail-pointer push into the coordinator's terminal. There is no type gate on that branch.
So a heartbeat pushes a pointer in every case except while a check --wait with no type filter is blocking. Notably it still pushes when the coordinator is blocked in check --wait --types worker_done, because the heartbeat matches no waiter and therefore takes the push branch.
The arithmetic. The preamble instructs every worker to heartbeat every 5 minutes (HEARTBEAT_INTERVAL_MIN in preamble.ts). Six lanes is roughly 72 coordinator wakes per hour, each one a turn, to convey that nothing happened. Lanes scale that linearly, which is the opposite of what you want from a fleet.
This was not an oversight in the original design — it was out of scope. The heartbeat was introduced in ea8ea08116 (#1403), which squashed the "preamble rules + heartbeat schema" and "coordinator heartbeat + stale detector" work together. The stated purpose is narrow and still correct: "The coordinator uses this to distinguish 'still thinking' from 'hung / crashed.'" The cadence comment weighs detection latency against inbox spam — volume in a mailbox. What it does not consider is that arriving mail also pushes, because at the time the question was "how often should a worker report" and not "what should wake a coordinator".
The state half already exists and is already correct: dispatch_contexts.last_heartbeat_at, written by recordHeartbeat, read by getStaleDispatches. The message is the part that costs.
Proposed solution
Make the heartbeat state rather than mail. The worker still reports; the runtime still records last_heartbeat_at; the coordinator reads it when it is already talking to Orca instead of being interrupted to hear it.
The read surface already exists in review: #14907 / #14891 return a per-lane block on the orchestration commands a coordinator already runs, carrying delivery state and quiet time. Heartbeat freshness is the natural third field, and no new command or polling loop is needed.
Keep every other type pushing. worker_done, escalation, question and decision_gate are events — something happened and the coordinator should hear it now. Only the heartbeat, whose payload is by definition "no change", stops waking anyone. That preserves the property that matters: the coordinator wakes when something happened, not when nothing did.
Then add back the one wake that is worth it. There is a stall that nothing currently detects: work remains, every lane is idle, and the coordinator is idle too. Nobody is blocked on anything, nobody is working, and no message will ever arrive to break it — precisely because everyone is quiet. Today that Run sits still until a human notices.
That is the wake worth spending, and it is only computable centrally, because no single participant can see it: a lane knows only itself, and the coordinator is asleep. The runtime holds all three facts already — pending or ready tasks, per-lane idle state, and the coordinator's own state.
This has a pleasing symmetry with the first half: remove the wakes that mean nothing, add the one that means everything.
Alternatives or additional context
Coalesced heartbeat at a lower cadence. Instead of removing the message, have the runtime aggregate: one message per Run at a longer interval summarising every lane, rather than one per worker per 5 minutes. This keeps heartbeat-as-mail — so anything consuming those messages keeps working — while cutting wake count by roughly the lane count. Strictly less good than state (it still wakes for "no change"), but a much smaller change and a reasonable landing spot if removing the message is judged too invasive.
Type-gate the push only. Smallest possible version: leave everything as it is and skip the pointer push for heartbeat in the consumers.length === 0 branch. Heartbeats keep arriving as mail and are read on the next check, but stop interrupting. This is close to a one-place change and would capture most of the benefit; the downside is that the mailbox still fills, which is what the original cadence comment was trying to avoid.
Risks worth checking before implementing any of these:
check --waitwaits for messages. Heartbeats are not what it waits for, so removing them should not strand a waiter — but that deserves a test rather than an argument.- Heartbeats cross to federated peers as messages today. Turning them into state is a behaviour change that reaches older clients even with no wire change, which is the case
docs/reference/remote-wire-compatibility.mdwarns about. - [Feature]: nudge a silent dispatch from the runtime, without taking completion authority #14897 (nudging a silent dispatch) needs to know a worker is alive. If heartbeats become state, that supervisor reads state — which is actually better than depending on a message having arrived, but the two designs should be settled together rather than in sequence.
Related: #14907 and #14891 (the read surface this would use), #14897 (acting on the signal), #14832 (the mail-pointer push path, and its interaction with a user typing), #9228 (event-driven coordinator lifecycle — this issue is effectively an argument about which events deserve to be one).
Update (2026-08-26)
The read-surface PRs referenced in the proposal (#14891/#14892) closed unmerged on 2026-08-22, so the paragraphs that lean on them no longer describe anything in review. The problem statement is unchanged on current main: the mailbox push path has no type gate — when no waiter matches, mailbox-notification-coordinator.ts schedules delivery regardless of message type, so a heartbeat still wakes a coordinator that was explicitly waiting for worker_done. The state half also still exists (last_heartbeat_at on dispatch_contexts), so the proposal's substance — stop pushing heartbeats, read them as state — is intact; only the "read surface already in review" framing is stale.
Activity
Tauri-EPO commentedon Aug 20, 2026
Additional measured reproduction from a multi-hour supervised run with five workers on Windows / Orca 1.4.185:
This interrupted the coordinator more than a dozen times. Nearly every check returned a routine
heartbeat, so the observed cost scaled asworkers × heartbeat cadence: five lanes produced more coordinator turns for liveness telemetry than for actionable work.I also tried suppressing this entirely on the consumer side:
orca orchestration check --wait, discardedheartbeat, and surfaced onlyworker_done,escalation, andquestion.check --ack <deliveryId>immediately after each result, before starting the next wait.The prompt still appeared. Immediate acknowledgment therefore is not a reliable policy-level way to suppress heartbeat interruptions.
Current-main source check
Verified against
upstream/main@0f26ff4ad83e9ca736f6ad3bae6937cd0cdab7fc:notifyMessageArrivedfor it.read = 0anddelivered_at IS NULL.last_heartbeat_atand uses it for stale-dispatch detection.One correction to the initial interpretation: the code does re-check unread state before staging a pointer, so the evidence does not prove that notification is wholly independent of acknowledgment. It is consistent with two narrower races/asymmetries:
worker_done,escalation,questiondoes not matchheartbeat;consumers.length === 0therefore sends the heartbeat down the pointer path.Either way, ack timing is the wrong suppression contract.
Correction options
A — filter at pointer origin. Do not pointer-push valid
heartbeatmessages (and considerstatusseparately; some status/rejection messages may be actionable). This is the smallest compatible fix. Heartbeats remain auditable inbox rows, but the inbox still grows.B — make the consumer's interest filter govern pointer wakeups too. Conceptually clean while a waiter exists, but an active waiter is ephemeral: process restart gaps still have no durable declaration of interest. A durable per-Run subscription would be more complete, but materially larger.
C — aggregate by Run and time window. Preserves every signal and reduces volume, but still spends coordinator turns on “nothing changed” and adds batching/timer state.
D — stop creating heartbeat inbox deliveries. Record
last_heartbeat_atonly and expose freshness through Run/fleet state. This removes the duplicate representation and scales best, but changes behavior seen by independently updated/federated clients and any consumer that currently audits heartbeat messages.Recommendation
Land A plus the liveness read surface first, which is exactly the bounded direction already implemented in open PR #15173: preserve the message for compatibility, suppress routine heartbeat pointer delivery, and expose heartbeat age in fleet state. Then evaluate D as a separately negotiated compatibility change once consumers can rely on the state surface. B alone does not close waiter-registration gaps, and C reduces rather than removes the meaningless wake cost.
worker_done,escalation,question, anddecision_gateshould continue to wake the coordinator.statusshould not be blanket-classified as non-interactive until its rejection/failure variants are separated.Tauri-EPO commentedon Sep 14, 2026
Measurement from one coordinator day on Orca 1.4.201 (Windows 11, 2026-09-14, about 10 Cursor lanes supervised through
worker-start, coordinator draining withcheck --run … --jsonandcheck --ack <deliveryId>). Counted from the savedcheckreceipts, deduplicated by message id:heartbeatstatus/worker_done/questioncheckcheck --ackcalls issuedSo more than half of the acks the coordinator issued carried no information beyond "still alive", and each one was preceded by a
You have N orchestration message(s)pointer typed into the coordinator pane.Where this sits in the 1.4.201 source (
mainatfe4237cd41):src/main/runtime/rpc/methods/orchestration/messaging/send-methods.ts:110-115).last_heartbeat_atbut only marks the row read when the Dispatch is no longerdispatched(src/main/runtime/orchestration/lifecycle-reconciliation.ts:150-158); for a live worker the row stays unread and lands in the next FIFO batch.src/main/runtime/orca-runtime-deliver-pending-messages.ts:47-50), andcheckhas no--exclude-typesor heartbeat-specific ack (src/cli/specs/orchestration.ts:106-119;--typesis only the--waitwake condition, per the spec note at:122).worker-list'sprojection.livenessis computed from agent-status evidence, not fromlast_heartbeat_at(src/shared/orchestration-fleet-worker-projection.ts:45-60); the column is read only byworker-showand the stale-dispatch detector ([Feature]: orchestration worker-list carries no heartbeat information, so the fleet view cannot show which lane went quiet #15951).PR #15173 would move this to recorded state; it is currently marked conflicting against
main.Philipp-Kreutzmann-DC commentedon Sep 17, 2026
One more data point, on the prompt side. The dispatch preamble tells every worker to "send a heartbeat every 5 minutes while actively working on the task", so a brief that says "Do NOT send heartbeat or status messages" contradicts its own preamble. Measured on our side: two workers whose brief opened its task section with that line made 65 and 56 tool calls and sent no heartbeat, while a worker whose brief did not mention heartbeats spent 30 of its 346 tool calls on them. An option on the dispatch that leaves the heartbeat rule out of the preamble would remove the contradiction.
IsidroLOlguin commentedon Sep 19, 2026
One more measurement, and a small patch that gets most of this without changing the protocol or
the preamble.
Measured cost per heartbeat, on the coordinator's context rather than on turn counts.
Windows 11, Orca v1.4.205 (
11aba8bdc5; every file quoted below is unchanged onmainat50f507b731, 2026-09-19), one Run with one supervised worker, 2026-09-18. The worker heartbeatarrived every ~25-30 s (much faster than the 5-minute cadence the preamble asks for). Six of them
in under 5 minutes took the coordinator's context from 151,207 to 161,573 tokens — about 1,700
tokens per heartbeat, all of it overhead, and enough to trip that session's "wrap up and
restart" threshold with no work done. Every pointer said the same thing: "still alive, phase
reviewing".
The half that is already state is written before the pointer exists. Following the native
path in v1.4.205:
send-point-to-point.ts:109-111reconciles the heartbeat at send time:reconcileLifecycleMessage→reconcileHeartbeatMessage→db.recordHeartbeat(dispatchId, …)(
lifecycle-reconciliation.ts:171).runtime.notifyMessageArrived(to_handle, type).case 'heartbeat': handleLifecycleMessage(
coordinator.ts:185) — re-reconciling what was already reconciled at step 1.worker_donearrives(
lifecycle-reconciliation.ts:338-348), i.e. after the per-turn cost has been paid.So by the time the pointer reaches the pane, the row carries nothing the coordinator needs.
A trap worth naming: excluding
heartbeatfrom pointer eligibility would be a bug. That wasthe obvious one-line version, and it is wrong: a heartbeat sent from a pane that is not the
assignee is converted into a rejection that keeps
type: 'heartbeat', and the send re-notifiesit on purpose, because the coordinator's re-read is the only place that anomaly surfaces. A type
gate on the delivery branch would silence the anomaly along with the routine traffic.
What worked instead: close the row where its information is consumed. Mark it read and
delivered next to
recordHeartbeat. A closed row is invisible to both delivery lanes, since bothread
getUndeliveredUnreadMessages, and the rejected heartbeat returns earlier and stillsurfaces. Three sites, because three of them write liveness and none routes through another:
and the same two lines in the federated relay import (
db/federation/federation-relay-import.ts,after
this.recordHeartbeat(...)) and in the legacy lifecycle commit(
db/legacy/legacy-lifecycle-operation.ts), each followed by a re-read of the row(
message = this.getMessageById(message.id) ?? message) so the caller'sread === 0wake guardsees the close.
Closing the row is only half of it — all three paths still woke a
checkwaiter, which thenresolves to an empty result:
One follow-on: with reconciled heartbeats already closed,
suppressEarlierHeartbeats(
lifecycle-reconciliation.ts:338-348) can only ever meet rejected rows, whose whole point isto be seen — so it has to skip them, or the bulk sweep at
worker_donehides the anomaly theparagraph above is protecting:
Not covered, so none of it is taken as covered:
(a per-lane block on the commands a coordinator already runs) is still the other half, and
last_heartbeat_atremains the only place the information lives.checkas well, so a coordinator runningcheck --types heartbeaton purpose now sees anempty mailbox. There is a test in the tree that asserts the opposite —
rpc/methods/orchestration/messaging/check.test.ts:498, "records heartbeat returned by unreadmanual check", asserting
count === 1— and this change inverts it to0. That test has tobe rewritten deliberately, or the change looks like a regression when CI goes red.
worker-stopneverlanded ([Bug]: a worker-stop that closed nothing still revokes the capability, muting a worker that is still running #21688) typically announces itself with a heartbeat; that wake-up is currently the earliest sign
the stop was unproven. Worth keeping some read surface for it before this lands.
visible.
a live relay. The diffs are written to be read, not to
git apply— hunk headers are real,some context lines are elided.
fabkho commentedon Oct 3, 2026
Data point from 1.4.218: about 20 mailbox nudges in one coordinator session, about 15 of them heartbeats, and each one cost a coordinator turn. #15173 would fix this. Could it get a review?
yoyofreeman commentedon Oct 14, 2026
+1 — additional measurement from a large-context coordinator (Linux).
Environment: Ubuntu 24.04.5, Linux 6.8, pi 1.2.0, orca 1.4.224.
We ran one supervised Orca worker (injected preamble) while the coordinator session was also doing unrelated work. Measured 2026-10-14:
cacheRead ≈ 161,000tokens (in≈200 / out≈100 / cacheRead≈161k) — i.e. the entire coordinator context is re-read on every heartbeat wake. Cadence was ~4–5 min → ~12 wakes/hour ≈ $0.03/hour for a single lane.Our-side mitigation (no upstream change required): we now run deterministic long shell commands (
gates/pytest/builds/downloads) in-place undernohupinstead of spawning an LLM worker — no worker ⇒ no heartbeat ⇒ no wake — and we release/close a settled worker immediately. The upstream asks remain: stop pushing heartbeat to the coordinator (#14910) and/or expose the interval (#23894).changl commentedon Oct 23, 2026
Still present in 1.4.237 (macOS). Data point: an Opus coordinator carrying ~670k tokens of context, supervising ~8 workers at the 5-minute preamble cadence, was woken about once a minute by heartbeat-only mail. Each wake is a full model turn that re-sends the whole context to conclude nothing changed (roughly 40M cached tokens an hour). Confirming the analysis above: a
check --wait --types worker_done,escalation,questionwaiter does not suppress these; only an unfiltered waiter does.Our stopgap is a small wrapper that keeps one unfiltered
check --waitarmed, acks heartbeat-only deliveries itself, and returns only on other message types. It works, but every coordinator has to remember to run it. A review of #15173 (it needs a rebase) would make it unnecessary.MihokTim commentedon Oct 23, 2026
Another data point, still present on Orca 1.4.237 (Windows 11, 2026-10-24).
Setup: a Claude Native (structured chat) coordinator bound to one Run, supervising 1–3 Codex workers started with
worker-start(structured), coordinator draining withcheck --run … --json→check --ack <deliveryId>.Over ~45 minutes (20:41–21:26 UTC) the coordinator conversation received 15 separate "You have N orchestration message" turns. 6 of the 15 (40%) were heartbeat-only deliveries; the rest carried
worker_done,question/escalationorstatus. Several more heartbeat pointers arrived mid-turn on top of that. Each heartbeat-only turn re-sent the coordinator's whole context (about 190k–230k tokens at that point) just to ACK "alive".Two things we checked while looking for a workaround on 1.4.237, in case they help:
worker-start --help,send --help,check --helpnor the version-matchedskills get orchestrationguide exposes a way to change the heartbeat cadence or to exclude heartbeats from the coordinator pointer;check --typesonly affects--waitwake-up, andcheckstill returns the whole FIFO delivery.check --wait --types heartbeatwaiter does not look like a safe workaround: the delivery it receives is the whole FIFO batch, so it can pick upworker_done/questionmail together with the heartbeats, and an unacknowledged delivery appears to hold back the coordinator pointer — i.e. the workaround risks muting the actionable mail it was meant to protect.So from the coordinator side we have no safe local mitigation; #15173 (heartbeat as recorded state instead of mail) or a per-Run notify-type filter as proposed in #26767 would solve this for us.
MihokTim commentedon Oct 23, 2026
Follow-up to my comment above: we checked whether a local workaround is safe — a separate long-running
check --wait --types heartbeatwaiter next to a Native coordinator — and concluded it is not safe on 1.4.237. Source references are pinned to thev1.4.237tag (0717478):checkreturns the whole FIFO delivery regardless of type (up to 50 unread, or a replay of an existing un-ACKed delivery), so it can pick upquestion/worker_donetogether with heartbeats (role-mailbox-delivery.ts L44–104), and ACK marks the whole delivery read (L111–169).exclusiveper Run (waiter_exists) and share the Run'sconsumer_generation, so a helper is not an isolated consumer next to the coordinator (check-run.ts L174–205).The liveness update itself happens at send time (
recordHeartbeat→last_heartbeat_at) and does not depend on the coordinator reading the mail (lifecycle-reconciliation.ts L124–171), so excluding accepted heartbeats from the pointer batch on the Orca side (the direction of #15173) looks like the minimal safe fix. These three files are byte-identical inv1.4.238. This is from reading the source; we did not reproduce the mixed-delivery case live.iampaycheck commentedon Oct 25, 2026
Another data point for this, from Windows 11 with Claude Code coordinators, on Orca 1.4.232 (we updated to 1.4.240 today and have not re-measured on it yet).
We run long-lived coordinator Runs with 5-15 dispatched workers each. Over about 20 hours, two coordinator Runs received:
Workers send one heartbeat roughly every 5 minutes by design, so the wake rate scales with the number of workers rather than with real work. In a one-hour read-only sample of two idle coordinators we counted about 10 heartbeat or "alive" rows per coordinator-hour, each of which is a full model turn if the pointer is the only wake path.
To avoid paying a turn for these we built our own mail waiters that acknowledge heartbeat-only batches silently, plus a watchdog and a stop-guard. All of that exists only to filter rows that Orca already types.
So the narrow fix described in the title (a heartbeat-only batch never pointer-wakes an idle coordinator; it stays in the inbox and is acknowledged with the next real batch) would remove most of our machinery. Happy to test a build on Windows if that helps.
danielchanintereras commentedon Oct 26, 2026
Another data point on Orca 1.4.237 (macOS, Claude Code coordinator, 2–4 Codex workers per Run). Every worker heartbeat still reaches the coordinator as a pointer, so a long-context coordinator turn runs just to read "alive". We now route bounded reviews/workers outside supervised Dispatches purely to avoid this, which loses Orca's liveness / ask-reply benefits.
Request: a user-controllable setting, e.g.
orchestration.heartbeat.wakeCoordinator = false(or per-Run), where valid heartbeats only update liveness + audit and never create a pointer/turn, whileworker_done/question/escalation/decision_gatestill wake normally. Happy to test a build.