Skip to content

feat(orchestration): make a recorded heartbeat state the coordinator reads, not mail that wakes it - #15173

Open
Tauri-EPO wants to merge 2 commits into
stablyai:mainfrom
Tauri-EPO:Tauri-EPO/heartbeat-state
Open

Tauri-EPO wants to merge 2 commits into
stablyai:mainfrom
Tauri-EPO:Tauri-EPO/heartbeat-state

Conversation

@Tauri-EPO

@Tauri-EPO Tauri-EPO commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

ELI5

A dispatched worker sends a heartbeat on a fixed cadence to say "still alive". Today that heartbeat
is pushed to the coordinator like any other mail, so the coordinator spends a turn reading a message
whose entire content is "nothing changed".

The liveness it carries was already written to the database before the push ever ran. So this stops
pushing it and leaves it where the coordinator can read it on demand.

What Changed

  • getUndeliveredUnreadMessages takes a new excludeRecordedHeartbeats option that narrows the
    read to exclude a recorded heartbeat.
  • selectOrchestrationPointerBatch passes it. That function is the one batch selector both delivery
    lanes share — the PTY lane in mailbox-pointer-delivery.ts and the structured-session lane in
    structured-mailbox-pointer-delivery.ts — so a single flag covers both, including a worker running
    over SSH or imported through federation, because all of those paths land rows in the same
    messages table and push them through the same selector.
  • A rejected heartbeat still delivers. It is not liveness: it means someone other than the
    assignee claimed it, which would otherwise mask a hung assignee. The exclusion tests for the
    _orcaLifecycleRejection marker that convertLifecycleMessageToRejection writes, and a false hit
    on that test only over-delivers.
  • Nothing is hidden. check, inbox, and worker-show read through different queries; the row
    stays read = 0 and keeps showing up there.
  • Two guide sentences say heartbeats are state the coordinator reads, not mail that nudges it — one
    in the always-loaded kernel and one in the worker-contract reference.
    src/cli/bundled-skill-guides.ts is regenerated, not hand-edited.
  • New orchestration-heartbeat-push-silence.test.ts, five cases against a real SQLite database.

The filter is applied in SQL rather than after the read on purpose. A recorded heartbeat is never
delivered and never read, so heartbeats accumulate at the head of sequence; a filter applied after
LIMIT would eventually return a page that is entirely heartbeats and push nothing while a
worker_done waited behind them. One of the five tests is exactly that case.

Why

#14910. HEARTBEAT_INTERVAL_MIN in preamble.ts
is 5, so each worker sends 12 heartbeats an hour; a six-lane fleet is 72 pushes an hour that convey
nothing the coordinator could not already read. Each one costs a coordinator turn and displaces
attention from mail that does carry a decision.

The heartbeat is not lost by not being pushed. reconcileLifecycleMessage writes
dispatch_contexts.last_heartbeat_at on the send path, before any pointer runs, and
getStaleDispatches reads that column. Stale-worker detection is untouched — a test asserts the
timestamp is still written when the push is silenced.

Prior art, for the record: #13235 proposed the same
wake half and was closed as a draft on 31 Aug without review. This is written against current main
rather than derived from it.

Linked Issue

Fixes #14910

Visual Proof

N/A — no UI surface. The change is which rows a mailbox pointer batch contains; nothing renders
differently.

Testing

Windows 11, Node 24.19.0, pnpm 12.0.0, rebased onto upstream/main at 9d29e6878e.

New test, red then green — with excludeRecordedHeartbeats left off, 3 of the 5 cases fail:

$ node node_modules/vitest/vitest.mjs run --config config/vitest.config.ts \
    src/main/runtime/orchestration/orchestration-heartbeat-push-silence.test.ts
 Test Files  1 passed (1)
      Tests  5 passed (5)

Suites around the change:

$ ... run src/main/runtime/orchestration
 Test Files  110 passed | 3 skipped (113)
      Tests  888 passed | 11 skipped (899)

$ ... run src/main/runtime/rpc/methods/orchestration \
    src/main/runtime/orchestration-mailbox-notification-consistency.test.ts \
    src/main/runtime/orchestration-mailbox-cold-park-idle.test.ts \
    src/main/runtime/orchestration-structured-chat-lease.test.ts
 Test Files  78 passed (78)
      Tests  729 passed (729)

Gates:

$ pnpm tc:node                      -> clean
$ pnpm tc:cli                       -> clean
$ pnpm verify:bundled-skill-guides  -> clean
$ node config/scripts/check-changed-code-quality.mjs upstream/main
code quality: 0 new finding(s) across 4 changed file(s).
type-aware code quality: 0 new finding(s) across 4 changed file(s).
React Doctor: 0 new finding(s) across 4 changed file(s).
Changed-code quality gate passed since 9d29e6878e70.

config/scripts/orchestration-skill-guidance.test.mjs is 28 passed / 1 failed here. That one failure
("keeps heartbeat and worker_done recipes bound to the injected capability") reproduces unmodified on
upstream/main on this machine: its fenced-block regex requires a bare LF after the fence marker, so
it cannot match a Windows checkout with core.autocrlf=true. Unrelated to this change and untouched.

pnpm build and the full pnpm test were not run locally; CI covers them.

  • I manually tested these changes locally
  • Automated tests added/updated, or explained why not below

The manual box is unchecked because I did not exercise this in a live multi-agent run. The evidence
above is the test suites, not an in-app session.

AI Disclosure

Written with Claude Code (Claude Opus). I reviewed the diff and ran every command quoted above.

Review

Cross-platform. No platform-dependent code. The predicate uses the SQLite core scalar functions
instr and COALESCE; no path, shell, or process handling is involved.

SSH and remote. The execution host owns the mailbox and the pointer push, and nothing here reads
or reports liveness across a connection. A heartbeat that arrives from a peer through
federation-relay-import is inserted into the same messages table and selected by the same
function, so remote workers get the same behavior rather than a second code path. No wire format,
RPC parameter, or stream opcode changes, so a mixed-version client/host pair is unaffected — this is
entirely host-side row selection.

Agent and integration compatibility. Both delivery lanes go through the shared selector, so PTY
agents and structured-session agents behave identically and no agent-specific branch is added.
Nothing here touches source control, so it is provider-neutral by construction.

Performance. One extra constant predicate on an already-filtered query, and the read returns
fewer rows than before. The starvation test covers the pathological case the SQL placement exists to
prevent.

Security. No privilege boundary is crossed and no new data reaches a caller. The rejected
heartbeat — what a wrong sender produces — is precisely the case that still delivers, so this cannot
be used to hide a spoofed liveness claim.

UI quality. No renderer change.

Folder workspaces. Not workspace-shaped; mailboxes are addressed by run or dispatch handle, not
by worktree.

Considered and left out. The e2e fake in orca-runtime-test-orchestration-messages.spec.ts
implements its own getUndeliveredUnreadMessages and does not model the new option. No spec seeds a
heartbeat row into it, so the difference is unobservable there; I left the fake alone rather than
grow the diff, and I am happy to mirror the option if you would rather it stay in step.

Agent skill upstream boundary

  • Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Ensure no issues in: Security, Cross-platoform support (Linux, Windows, Mac), Remote SSH, Mobile, general backwards compatibility, performance

Scope note: this is only the wake half of
#14910. The read half — reporting heartbeat
freshness so a coordinator can see the age it is no longer being told — is
#15963 and is deliberately not here.

Checklist

  • This PR is small and focused
  • I explained what changed and why (including ELI5)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred) — see Testing for exactly what ran and what did not

Author

  • X / Twitter: @EnricoPin

@Tauri-EPO
Tauri-EPO force-pushed the Tauri-EPO/heartbeat-state branch from b0a97e6 to cf04fcb Compare August 17, 2026 21:19
@coderabbitai

coderabbitai Bot commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

Review skipped

We couldn't safely recover the incremental review. No full review was started, and the last reviewed checkpoint was preserved. Retry later, or explicitly request a full review by commenting @coderabbitai full review.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Added Fleet Echo snapshots for orchestration runs. The snapshots report active lanes, delivery state, process state, heartbeat age, quiet duration, run identity, and truncation. RPC methods and CLI commands support fleet data by default and allow --no-fleet suppression. The CLI renders text output while preserving JSON responses. Added terminal-summary lookup and run-ownership checks. Ordinary heartbeats no longer create coordinator pointers, while rejected heartbeats and later lifecycle messages remain deliverable. Added tests for formatting, runtime sources, RPC propagation, dispatch liveness, and timestamp handling.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR satisfies the heartbeat-state, wake-suppression, visibility, and event-delivery requirements, but omits the additional idle-work wake described in #14910. Implement the additional wake for pending work with idle lanes and coordinator, or split and explicitly defer that requirement in #14910.
Docstring Coverage ⚠️ Warning Docstring coverage is 30.30% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Out of Scope Changes check ✅ Passed The changes support the heartbeat read and delivery behavior, while the broader Fleet Echo work is identified as preceding stacked scope.
Title check ✅ Passed The title clearly identifies the main change: valid recorded heartbeats no longer wake the coordinator and are read as state instead.
Description check ✅ Passed The description is complete and follows the required template. It explains the change and rationale, links issue #14910, documents the lack of UI changes, lists testing and environment details, record…

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
src/main/runtime/orchestration/db-dispatch-liveness.test.ts (1)

32-63: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Test the oldest-first contract with multiple active rows.

The first test has only one active row. It cannot detect loss of dc.rowid ASC ordering. Create multiple active rows and assert their ordered IDs. Assert the selected IDs in the limit test too.

src/main/runtime/orchestration/mailbox-pointer-eligibility.ts (1)

7-11: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Reduce these comments to one concise sentence each.

Keep the delivery invariant and remove the detailed rationale and implementation walkthrough.

As per coding guidelines: “Comments must be concise, non-obvious, and brief—prefer one line.”

Also applies to: 65-67

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d162d69e-e7ab-4471-93f6-6c64f4cf96cf

📥 Commits

Reviewing files that changed from the base of the PR and between 2aebcfe and b0a97e6da9f7386f48b574afba3819424924524b.

📒 Files selected for processing (21)
  • skill-guides/orchestration.md
  • src/cli/bundled-skill-guides.ts
  • src/cli/handlers/orchestration-fleet-echo-format.test.ts
  • src/cli/handlers/orchestration-fleet-echo-format.ts
  • src/cli/handlers/orchestration.ts
  • src/cli/specs/orchestration.ts
  • src/main/runtime/orca-runtime.ts
  • src/main/runtime/orchestration-heartbeat-push-silence.test.ts
  • src/main/runtime/orchestration/db-dispatch-liveness.test.ts
  • src/main/runtime/orchestration/db.test.ts
  • src/main/runtime/orchestration/db/dispatch-context/dispatch-lookup.ts
  • src/main/runtime/orchestration/mailbox-pointer-delivery.ts
  • src/main/runtime/orchestration/mailbox-pointer-eligibility.ts
  • src/main/runtime/rpc/methods/orchestration-check.test.ts
  • src/main/runtime/rpc/methods/orchestration-fleet-echo-sources.test.ts
  • src/main/runtime/rpc/methods/orchestration-fleet-echo-sources.ts
  • src/main/runtime/rpc/methods/orchestration-send-fleet-echo.test.ts
  • src/main/runtime/rpc/methods/orchestration-tasks-dispatch.test.ts
  • src/main/runtime/rpc/methods/orchestration.ts
  • src/shared/orchestration-fleet-echo.test.ts
  • src/shared/orchestration-fleet-echo.ts
💤 Files with no reviewable changes (1)
  • src/main/runtime/orchestration/db.test.ts

Included review availability: Your plan includes up to 10 reviews per rolling hour; 8 remain after this review.

Comment thread src/cli/handlers/orchestration.ts Outdated
Comment thread src/main/runtime/orchestration/mailbox-pointer-eligibility.ts Outdated
Comment thread src/main/runtime/rpc/methods/orchestration-tasks-dispatch.test.ts Outdated
@Tauri-EPO
Tauri-EPO force-pushed the Tauri-EPO/heartbeat-state branch 2 times, most recently from 53984ac to e04b8b1 Compare August 17, 2026 22:58

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
src/shared/orchestration-fleet-echo.ts (1)

53-56: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Shorten the multi-line rationale comments.

The comments state useful policy, but each is four lines long. Reduce each comment to one concise, non-obvious sentence.

  • src/shared/orchestration-fleet-echo.ts#L53-L56: Replace the scan-limit rationale with one short sentence.
  • src/shared/orchestration-fleet-echo.ts#L99-L102: Replace the ranking rationale with one short sentence.

As per coding guidelines: “Comments must be concise, non-obvious, and brief—prefer one line; do not explain obvious behavior or walk through code.”

Source: Coding guidelines

src/cli/handlers/orchestration-fleet-echo-format.ts (1)

32-34: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Reduce explanatory comments. The code and test names already express most of this behavior. Keep only short comments that explain a non-obvious constraint.

  • src/cli/handlers/orchestration-fleet-echo-format.ts#L32-L34: reduce this to one short rationale, or remove it.
  • src/cli/handlers/orchestration-fleet-echo-format.ts#L62-L63: reduce this to one short rationale.
  • src/cli/handlers/orchestration-fleet-echo-format.ts#L66-L68: reduce this to one short rationale.
  • src/cli/handlers/orchestration-fleet-echo-format.test.ts#L17-L18: remove or reduce this setup explanation.
  • src/cli/handlers/orchestration-fleet-echo-format.test.ts#L73-L74: remove or reduce this assertion explanation.
  • src/cli/handlers/orchestration-fleet-echo-format.test.ts#L108-L109: remove or reduce this assertion explanation.
  • src/cli/handlers/orchestration-fleet-echo-format.test.ts#L198-L199: remove or reduce this setup explanation.

As per coding guidelines, comments must be concise, non-obvious, and brief—prefer one line.

Source: Coding guidelines


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8db926b7-6d90-42ea-81b2-16df4a94f952

📥 Commits

Reviewing files that changed from the base of the PR and between 53984acb88a876e2e1f4a80529bcff9374ae9540 and e04b8b1f257997051632646b5b7687bff61fcfd7.

📒 Files selected for processing (7)
  • skill-guides/orchestration.md
  • src/cli/bundled-skill-guides.ts
  • src/cli/handlers/orchestration-fleet-echo-format.test.ts
  • src/cli/handlers/orchestration-fleet-echo-format.ts
  • src/main/runtime/rpc/methods/orchestration-fleet-echo-sources.ts
  • src/shared/orchestration-fleet-echo.test.ts
  • src/shared/orchestration-fleet-echo.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • skill-guides/orchestration.md
  • src/main/runtime/rpc/methods/orchestration-fleet-echo-sources.ts

Included review availability: Your plan includes up to 10 reviews per rolling hour; 8 remain after this review.

@Tauri-EPO

Copy link
Copy Markdown
Contributor Author

Thanks — the mailbox-pointer-eligibility finding was a real defect and the consequence was worse than the comment describes. Fixed in e31740b621, with the regression that should have existed in the first place.

Why it was worse than "some rows get dropped". The gate made a valid heartbeat never deliver, so delivered_at stays NULL; and it stays read = 0 until an explicit check. Both filters are in the query, so those rows never leave it — they accumulate at the head of ORDER BY sequence permanently. At the preamble's 5-minute cadence, six lanes produce 72 heartbeats an hour and fill a 50-row page in about 42 minutes. From then on the query returns 50 heartbeats, the JS filter drops all 50, unread.length === 0, and the push returns early — so a worker_done behind them never points at all.

That is self-reinforcing in the wrong direction: the queue only drains on an explicit check, which is exactly what a coordinator stops doing once it trusts push. The feature would have starved the thing it was meant to protect.

The fix moves the rule into the read. getUndeliveredUnreadMessages gains excludeSilentHeartbeats, so the exclusion narrows the query and the LIMIT applies to rows that can actually be delivered:

NOT (type = 'heartbeat' AND instr(COALESCE(payload, ''), '_orcaLifecycleRejection') = 0)

instr rather than LIKE because the marker key contains an underscore, which LIKE reads as a single-character wildcard; the key is written verbatim by addLifecycleRejectionMarker, so this cannot produce a false negative, and a false positive would only deliver a heartbeat — the behaviour that predates the gate. No JSON1 dependency, which nothing in this codebase uses yet.

The test I should have written. The existing case queued one heartbeat against a limit of 50 — right in intent, wrong in dose. The new one queues a full page plus five, then a worker_done. Against the previous implementation:

× still points a worker_done queued behind a full page of silent heartbeats
  AssertionError: expected 0 to be greater than 0

Zero pointers, which is the bug exactly.

Two smaller things that came out of the same pass:

  • The backlog surfaced an unhandled ERR_INVALID_STATE from mailbox-owner's deferred routing — the suite closed its database in the same tick the setImmediate callbacks were still queued in. afterEach now drains one immediate turn before closing. It was latent before; 55 messages made it reliable.
  • orchestration.dispatchShow and the tasks-dispatch fixture are fixed in the PR below this one (047afc436d), since both belong to it.

Verification on e31740b621, Node 24: 10 affected suites, 231 tests passed; typecheck:node, typecheck:cli, oxlint src config tests --deny-warnings, the bundled-skill-guide check and the max-lines ratchet all clean.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d4dde344-43b9-4bd5-89cf-f8c5e4e669e1

📥 Commits

Reviewing files that changed from the base of the PR and between e04b8b1f257997051632646b5b7687bff61fcfd7 and e31740b621c4a2358688e4c6079a64c6f9769666.

📒 Files selected for processing (8)
  • src/cli/handlers/orchestration-fleet-echo-cli.test.ts
  • src/cli/handlers/orchestration.test.ts
  • src/cli/handlers/orchestration.ts
  • src/main/runtime/orchestration-heartbeat-push-silence.test.ts
  • src/main/runtime/orchestration/db/messages/message-inbox.ts
  • src/main/runtime/orchestration/mailbox-pointer-eligibility.ts
  • src/main/runtime/rpc/methods/orchestration-send-fleet-echo.test.ts
  • src/main/runtime/rpc/methods/orchestration-tasks-dispatch.test.ts
🚧 Files skipped from review as they are similar to previous changes (4)
  • src/main/runtime/rpc/methods/orchestration-send-fleet-echo.test.ts
  • src/main/runtime/rpc/methods/orchestration-tasks-dispatch.test.ts
  • src/main/runtime/orchestration-heartbeat-push-silence.test.ts
  • src/cli/handlers/orchestration.ts

Included review availability: Your plan includes up to 10 reviews per rolling hour; 9 remain after this review.

Comment thread src/cli/handlers/orchestration-fleet-echo-cli.test.ts Outdated
Comment thread src/cli/handlers/orchestration-fleet-echo-cli.test.ts Outdated
@Tauri-EPO

Copy link
Copy Markdown
Contributor Author

Holding this branch rather than pushing the standalone re-cut, and recording why, because the reason is the interesting part.

The re-cut works, and it is cleanly separable

This PR turned out to be one commit on top of the fleet-echo stack, not an entangled diff. The heartbeat commit is e31740b621; the fleet surface lives entirely in the commits below it plus one test-only commit above. Re-cutting it onto today's main needed no manual reconciliation at all — the three runtime files are byte-identical between this PR's base and main, because none of the 185 commits in between touch them.

Verified on the standalone branch:

typecheck                          EXIT=0
targeted (10 suites)               195 passed | 1 skipped
broader orchestration sweep        181 passed | 3 skipped
new heartbeat suite                5 passed
lint                               clean up to generate-skill-bundle-manifest,
                                   which fails identically on a pristine tree (#14479)

Regression injection, two mutations with disjoint failures:

gate disabled  -> 2 failed | 3 passed   (survivors are the cases that must push either way)
filter moved after the LIMIT -> 1 failed | 4 passed
   "still points a worker_done queued behind a full page of silent heartbeats"

That second one pins the property that matters: the filter has to narrow the SQL read, not the result, or a page that is entirely heartbeats starves the worker_done behind it.

I also re-checked the interaction with #14961 on current main: its retyping covers escalation / decision_gate only, so a rejected heartbeat keeps type = 'heartbeat' and gains the _orcaLifecycleRejection marker, and the filter's instr(...) = 0 excludes only unmarked ones. Rejected heartbeats still deliver, which is the behaviour I want.

Why I am not opening it

Standalone, this removes the wake without adding anywhere to read. I checked what exists on main: last_heartbeat_at is written by recordHeartbeat and read by exactly one consumer, getStaleDispatches, for internal stale detection. No CLI or RPC surface exposes heartbeat freshness to a coordinator. So after this change a coordinator's only way to see a heartbeat is an explicit orchestration check, where the row is still unread.

That is the same objection this PR's own notes raise against #13235 — "removes the wake and leaves the coordinator with nowhere to see freshness" — and it would be inconsistent of me to ship it and not mind. It is an ordering problem, not a code problem: exposing heartbeat freshness on a read has to come first, and it is a small change of its own since the column already exists.

So the sequence is two isolated PRs, and this is the second one. The branch is ready and will keep.

One thing I left as the original wrote it

The comment claims instr "cannot produce a false negative", which is true. It does not mention the other direction: a worker-supplied payload containing the literal string _orcaLifecycleRejection anywhere would escape the silence filter and still push. That fails safe — it delivers rather than hides — but it is worth knowing, and I will say so in whatever eventually opens.

Tracked as REC-04 on #15190, where it now sits behind the read step rather than in front of it.

@Tauri-EPO
Tauri-EPO force-pushed the Tauri-EPO/heartbeat-state branch from 7544a32 to 558af98 Compare September 7, 2026 11:54
@Tauri-EPO Tauri-EPO changed the title feat(orchestration): make the heartbeat state a coordinator reads, not mail that wakes it feat(orchestration): make a recorded heartbeat state the coordinator reads, not mail that wakes it Sep 7, 2026
@Tauri-EPO

Copy link
Copy Markdown
Contributor Author

Rewritten from scratch. The branch was force-pushed to a single commit on current main; head is
558af98715d98746eced2c28d0f8b903e8c67684, on top of 9d29e6878e.

What changed about this PR

The previous version was 13 commits on a 17 Aug base and carried roughly 1,500 lines that no longer
had a reason to exist. It is now one commit, +145/-6 across 6 files, doing one thing: a recorded
heartbeat no longer enters the mailbox pointer batch, plus two guide sentences saying a heartbeat is
state the coordinator reads rather than mail that nudges it.

Three things were dropped:

  • Fleet echo. Those hunks depended on
    #14891 and
    #14892, which were closed on 22 Aug. With the base
    retired the code here was dead, so it is gone rather than rebased.
  • Heartbeat age per lane. That is the read half, and it lives in
    #15963. Keeping it here duplicated that PR.
  • The dispatch-show caller-handle commit, which belongs to
    #15965 and had no business on this branch.

Why it was reimplemented rather than rebased

#16904 rewrote mailbox-pointer-delivery.ts and, in
the process, extracted batch selection into selectOrchestrationPointerBatch in
mailbox-pointer-eligibility.ts — which is where the old branch had been moving it by hand. So the
seam this PR needs already exists upstream, and the change is now a flag on the existing selector
instead of a refactor. Both delivery lanes share that selector, so the structured-session lane is
covered by the same two lines rather than a second copy.

I re-checked the current behavior before rewriting rather than assuming the old diagnosis still held:
on 9d29e6878e, message-inbox.ts offers only excludeTypes, and mailbox-notification-coordinator.ts
routes any arrival with no waiter straight to the pointer push. Heartbeats are still pushed today.

Verification

pnpm tc:node, pnpm tc:cli and pnpm verify:bundled-skill-guides are clean; the changed-code
quality gate reports 0 findings against upstream/main. src/main/runtime/orchestration is
888 passed / 11 skipped, and the RPC orchestration methods plus the three mailbox delivery suites are
729 passed. The new test file fails 3 of its 5 cases with the flag removed, so it is a real guard and
not a tautology. Full output is in the Testing section of the description, including one pre-existing
guidance-test failure that reproduces unmodified on upstream/main on Windows and is a CRLF artifact
of that test's regex.

I have not exercised this in a live multi-agent run, so the manual-testing box stays unchecked.

…reads, not mail that wakes it

A heartbeat's whole content is "nothing changed", and the liveness it carries is
already written to dispatch_contexts.last_heartbeat_at on the send path, before
any pointer runs. Pushing it a second time as mail spends a coordinator turn to
convey state that worker-list and worker-show can read at any time.

Exclude a recorded heartbeat from the pointer-delivery batch, in SQL rather than
after the read: the rows are never delivered and never read, so they accumulate
at the head of `sequence` and a post-LIMIT filter would eventually return a page
that is entirely heartbeats while a worker_done waits behind them.

A rejected heartbeat still delivers. It is not liveness -- it means someone other
than the assignee claimed it, which would otherwise mask a hung assignee. Nothing
is hidden either way: both rows stay unread for check, inbox, and worker-show.

Both delivery lanes share selectOrchestrationPointerBatch, so the PTY lane and
the structured-session lane are covered by the same change.
…ession lane

A structured coordinator owns its `run:` mailbox through the structured pointer lane, which
selects its batch with the same selectOrchestrationPointerBatch. Drive that lane against a real
database so the shared filter is proven there too, not only on the PTY lane.
@Tauri-EPO
Tauri-EPO force-pushed the Tauri-EPO/heartbeat-state branch from 558af98 to f156e48 Compare October 2, 2026 19:55
@Bkankim

Bkankim commented Oct 10, 2026

Copy link
Copy Markdown

A data point from real runs, in case it helps prioritise this.

Setup: Orca 1.4.218, one orchestrator, one coordinator, several worktree workers per run, multi-hour unattended runs (issue -> worker -> review -> merge).

Run Coordinator spec Heartbeat deliveries that reached the orchestrator
A default preamble (heartbeat every 5 min) 95
B first line of the coordinator spec forbids heartbeats 0

Each of those 95 deliveries woke an idle session just to say nothing changed. Run B finished normally with zero, so in our workflow heartbeats carried no information the coordinator needed; worker completion and escalation messages were enough.

The workaround has costs that this PR would remove: it relies on a prompt instruction (not enforced), it also gives up the hung-worker signal, and a filtered check --wait still returns on heartbeat-only deliveries, so we additionally auto-ack heartbeat-only deliveries in a wrapper loop. Treating a heartbeat as recorded state instead of mail, as this PR does, would let us keep stall detection without the wakeups.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: heartbeats wake the coordinator to say nothing changed — make them state, not mail

2 participants