Repository navigation
agent_prompt_stalled: dispatch fails ~8s after launch for slow-booting agents (Hermes, agy) — worker completes task but worker_done rejected #16660
Description
Summary
orca orchestration worker-start marks a dispatch failed with agent_prompt_stalled ~8 seconds after launch, before slow-booting agents (Hermes Agent CLI, Google Antigravity CLI agy) reach their first prompt. The worker agent then does complete the assigned task in its worktree and attempts orca orchestration send --type worker_done — but the send is rejected (exit 1) because the dispatch capability was already revoked. The task stays failed in the UI despite the work being done.
Environment
- Orca v1.4.188, headless
orca serve --pairing-address <tailscale-ip>on Ubuntu 24.04 (systemd, userdev) - Agents tried as workers, both hit the same issue:
- Hermes Agent v0.20.5 (
hermes --tuidefault launch, andhermes --cliviaagentCmdOverrides) — boots in ~15–60s (Python venv, model init) - Antigravity CLI
agyv1.1.21 (first-run onboarding prints help) — boots >8s
- Hermes Agent v0.20.5 (
Steps to reproduce
orca serverunning (headless, Tailscale)- From a live agent terminal (coordinator):
orca orchestration run-create --objective "test" --from <coordinator-terminal> orca orchestration task-create --spec "append a line to README.md" --run <run_id> --from <coordinator-terminal> orca orchestration worker-start --task <task_id> --worktree new-child --name test-worker --agent hermes --repo name:<repo> --timeout-ms 120000 --from <coordinator-terminal> - Dispatch returns
"state": "failed"/"lastError": "agent_prompt_stalled"within ~8s (dispatched_at→completed_atdelta ≈ 8s),capability_revoked_atset,last_heartbeat_at: null. - Meanwhile the worker terminal shows the agent booting and eventually executing the task:
- Hermes: edits README.md, runs
orca orchestration send --type worker_done --outcome succeeded --files-modified "README.md"→[exit 1](rejected, capability revoked) - The work is present in the worktree (uncommitted diff), proving the task was executed
- Hermes: edits README.md, runs
orca orchestration worker-show --dispatch <id>→"status": "failed", "last_failure": "agent_prompt_stalled"
Expected behavior
The dispatch should wait for the agent's first prompt longer than 8s (or the stall window should be configurable), so slow-booting agents like Hermes/agy can complete the dispatch_input stage. --timeout-ms 120000 on worker-start did not change the stall window.
Notes
- Direct connectivity is fine:
getUpdates-style API calls and agent one-shot invocations complete in <2s; the delay is purely agent startup (venv/init/onboarding). - Workaround attempt that did NOT help:
settings.agentCmdOverrides = {"hermes": "hermes --cli"}(line-based CLI instead of TUI); dispatching to an already-running agent terminal viaworker-start --terminalfails withagent_unconfiguredunless the terminal was launched by Orca;dispatch --injectto a manually-started agent terminal was not accepted either.
Happy to provide logs/terminal captures or test a patch.
Activity
fredogemilang commentedon Aug 26, 2026
Follow-up after local patch test
We patched the installed bundle (v1.4.188):
AGENT_PROMPT_EFFECT_TIMEOUT_MSfrom5e3→120e3and repacked app.asar. Results:dispatched_at 18:35:57→completed_at 18:38:06≈ 129s). The timeout itself is not the (only) problem.agent_prompt_stalled— becauseverifyAgentPromptSubmissionnever observesworkingSequenceadvancing past baseline.worker_done(rejected, exit 1, capability revoked after the stall).Suspected root cause
Hermes has the Orca-managed status-hook plugin installed (
~/.hermes/plugins/orca-status, reportson_session_start/pre_llm_call/post_llm_call/... viaORCA_AGENT_HOOK_ENDPOINT), and the runtime'sreadActivity()for hermes mapspre_llm_call/on_session_startto working-sequence advancement. In the dispatched-worker launch (hermes --tui --yolo+ injected preamble), the LLM clearly runs (UI shows synthesizing), yet no activity advance is observed — suggesting the hook events either don't fire in that launch context or their reporting to the hook endpoint is failing (e.g., env not propagated to the dispatched PTY, or the plugin is skipped in this mode).Request
Would it be possible to make the prompt-verification either (a) fall back to terminal output activity (any new bytes since baseline) when agent status hooks are unavailable/silent, or (b) expose
AGENT_PROMPT_EFFECT_TIMEOUT_MSvia env/setting? Both would make slow-booting or hook-silent agents (Hermes, agy) usable as dispatched workers. Happy to test a patch or provide the hook logs.fredogemilang commentedon Aug 26, 2026
Root cause fully mapped — final update
After extensive local testing (all with the 120s
AGENT_PROMPT_EFFECT_TIMEOUT_MSbundle patch), the complete causal chain for Hermes as a dispatched worker:What works
hermes --tui --yolo): the worker receives the full 91-line preamble, performs the task in the worktree, and attemptsworker_done. README modified correctly in every run.dcap_capability token is minted at dispatch and embedded ONLY in the preamble text (dispatch_contextsstores its sha256). With the original launch the token reaches the agent via the startup query path.The single remaining blocker: Hermes TUI emits no OSC terminal titles
applyTrackedPtyTitle→detectAgentStatusFromTitle(agent-title-status.ts), which requires the title to containhermes+working|thinking|running(or a spinner glyph).workingSequencenever advances past baseline →agent_prompt_stalledalways fires at the (now 120s) window → capability revoked → the worker'sworker_doneis rejected withdispatch_capability_invalid(--dispatch-capabilitytoken never reaches it via the normal path since the TUI composer collapses the injected preamble to a placeholder — see below).Additional finding: wrapper/override breaks hermes-query injection
planHermesStartupQuery(hermes-startup-query.ts) requires the launch executable token to behermes(isHermesExecutableToken). Settingsettings.agentCmdOverrides = {hermes: '<anything-not-hermes>'}silently disables the-q/--querystartup injection and falls back to pasting the 91-line preamble into the Ink composer, which collapses to[[ You are working.. [91 lines] .. or: <title> ]]— the worker then receives only the placeholder (no task, no capability token). This is a footgun worth documenting or guarding.Suggested upstream fixes (any one unblocks Hermes workers)
hermes working/hermes idletransitions) — feature request for NousResearch/hermes-agent.workingSequencewhen agent status hooks/titles are silent — most robust for all slow/headless agents.planHermesStartupQueryfails (non-hermesexecutable), fail loudly instead of silently degrading to composer paste.Happy to test any of these if a patch is proposed.
JSanchezFDZ commentedon Aug 27, 2026
Same failure on the same version, but with a bundled agent —
opencode— so this is notlimited to custom
agentCmdOverridesCLIs. Repro rate 6/6 on Orca 1.4.188, headlessorca-serve.service, Ubuntu 24.04, opencode 1.18.23.The agent then works normally and its first action is the heartbeat Orca handed it, which comes
back as
Dispatch <id> capability is revoked.— exactly the sequence described above.Measurements, in case they narrow it down. I tried to rule out the obvious candidates before
reporting:
render-cursor-after-bracketed-pastedoes arrive:on a raw PTY,
ESC[?2004hat 1.5–1.7 s andESC[?25hafter it at 5.4–6.4 s — inside the8 s render budget. Measured in a warm checkout and in a freshly created worktree with MCP servers
configured.
orca-opencode-status.jsis materialized in~/.config/orca/opencode-hooks/shared/plugins/withOPENCODE_CONFIG_DIRpointing at it, andmarker plugins confirm opencode 1.18.23 loads both
<config>/plugin/and<config>/plugins/.worktree, and
--setup skip. The effects always showaction: "reused_agent_terminal"for theagent terminal created by the agent-first worktree flow.
--timeout-ms 45000onworker-startdoes not change the outcome, andorca orchestration dispatch --injectinto the idle TUI fails the same way.the worker execute the task correctly.
Two things that look load-bearing from the source:
agent-prompt-submission-verification.tsaccepts three proofs.workingSequencecomes from thesynthetic-title route, which the comment in that file says does not survive headless serve
([Bug]: agent_prompt_stalled for codex/kimi workers — fixed 5s confirmation window vs ~20s first hook-event latency (root-cause evidence) #16095);
observedDeliveryEvidencerequiresbaseline.status === 'working', false for an idleagent taking its first task; so only the hook edge is left — and
opencodeis not inHOOK_OBSERVED_TURN_START_AGENTS, so it gets 5 s instead of the 15 s that codex/kimi got from[Bug]: agent_prompt_stalled for codex/kimi workers — fixed 5s confirmation window vs ~20s first hook-event latency (root-cause evidence) #16095. A reasoning model behind a remote API easily exceeds that: in my runs the first tool call
landed at 12.9 s.
opencodeandmimo-code(same profile,synthesizeTerminalTitle: false)look like they belong in that set.
TUI_AGENT_CONFIG.opencode.promptInjectionModeisflag-prompt, andtui-agent-startup.tsbuilds
<cmd> --prompt <quotedPrompt>for it — but the supervised path starts the agent throughcreateManagedWorktree({ startupAgent })without a prompt (process argv on the host is abare
opencode) and then pastes into the live TUI. Honouring the declaredflag-promptlaunchwhen the terminal is created for the dispatch would skip the paste-and-verify step for these
agents entirely.
Happy to send a PR for (1) if maintainers agree with the direction; I held off because I cannot
rebuild and test the Electron app on this headless host and would rather not ship an untested
change to this path.
Workaround, for anyone else blocked: launch the agent with the task text and stop expecting
worker_done:orca terminal create --worktree path:/path/to/worktree \ --command "opencode --model <provider>/<model> --auto --prompt '<task text>'"then follow the worker with
orca terminal read/ git instead of the dispatch state.nakedsleepwalker commentedon Aug 28, 2026
Same shape with a bundled agent (Grok) on 1.4.190 / Windows: stall at 14s, capability revoked, task completed in the worktree,
worker_donerejected. Details: #16902feat(orchestration): make multi-agent workflows durable (#16904)
nwparker commentedon Sep 19, 2026
Fresh live verification on the current migration stack (head
4ef37c14709, draft #21655), macOS, installed agy 1.2.7, isolated non-git folder:terminal.isRunningAgentrecognized a newly launched agy pane.dispatch --injectreturnedinjected: truewithinput_accepted; the task marker reached the CLI.worker-start --agent antigravityreturnedready, stageinput_accepted. Its Dispatch was stilldispatchedwith zero failures 63 seconds later. It did not fail withagent_prompt_stalledafter eight seconds.send --to @antigravityenqueued two messages, one in each test Dispatch mailbox. The first group send shortly after direct injection had returned no recipients; the later send included both the direct and supervised dispatches. That transient first lookup is not yet explained.ELI5: current Orca can deliver the task and keep its assignment alive without pretending it observed the agent start working. This machine's agy receives the task but generation returns HTTP401
ACCESS_TOKEN_TYPE_UNSUPPORTED. The receipt correctly says turn observation is unsupported; it is not proof of successful execution, mailbox consumption, or acceptedworker_done.This was normal local startup, not a reproduced 15–60 second boot or a Windows/SSH test. Both test Dispatches were explicitly stopped and their test terminals closed afterward. Authentication and hooks were unchanged. No new production change was needed for the injection/receipt checks; #21655 changes group addressing separately.
Visual proof below is a rendered report derived from the actual CLI receipts, rather than a screenshot of the lifecycle preamble (which contains private capability material).
nwparker commentedon Sep 19, 2026
Follow-up verification on the migration draft stack (
b109d5083be): a deliberately delayed local agy launch did not reproduce the eight-second dispatch failure.In plain language: I made the launcher wait 20 seconds before starting the installed Antigravity 1.2.7, then asked Orca to start a supervised worker with a 90-second timeout. Orca waited and delivered the task instead of failing at eight seconds.
Evidence:
sleep 20, thenexec agy "$@". Restored the original override afterward.ready,input_accepted. At 12:58:38 UTC, the dispatch remaineddispatched, with zero failures and no last failure.ACCESS_TOKEN_TYPE_UNSUPPORTED. No successful generation or realworker_donereport is claimed.Current source also already preserves reporting capability when dispatch input fails specifically with
agent_prompt_stalled(worker-start-receipt.ts). The existing database recovery tests verify that a subsequent worker report can correct that failed record, while other startup failures still revoke capability. Eight focused recovery/classification tests passed.This is stronger than the earlier normal-startup check, but it does not cover Hermes, Ubuntu/headless/remote execution, first-run onboarding, or successful task completion. No production patch is justified by this local non-reproduction; keeping the issue open for those remaining conditions.
JSanchezFDZ commentedon Sep 21, 2026
Retest of my 6/6
opencoderepro on 1.4.206 — the Ubuntu/headless/remote gap is now covered, and it passes.Same setup as my August report: headless
orca-serve.serviceon Ubuntu 24.04, dispatched from a separate machine with--environment, bundledopencode(now 1.18.31), child worktrees off a real repo, trivial task ("create a file, don't commit, then reportworker_done").Dispatch input: 3/3 accepted (was 0/6 on 1.4.188).
No
agent_prompt_stalledin any of the three, andturnStart: "unsupported"is exactly the honest receipt described above: Orca says it cannot observe the turn instead of failing the dispatch over it.worker_doneaccepted end to end: 2/3 — the condition the previous comment explicitly did not cover.Both worktrees contained the requested file, uncommitted, as asked.
The third run is not an Orca failure. Its agent's first model turn hung at the provider (no output, no tool call) and it never acted. The Dispatch stayed healthy throughout:
On 1.4.188 that same run would have been killed at ~5s with
agent_prompt_stalledand its capability revoked before the agent produced anything. I stopped it withworker-stop --dispatch <id>and removed all three worktrees.So from my side: Ubuntu 24.04 + headless
orca-serve+ remote dispatch + bundledopencode+ successful completion is covered and working on 1.4.206. I can't speak to Hermes, Windows, or first-run onboarding.OrcaWin commentedon Oct 9, 2026
Fixed by #16904, which made orchestration dispatch durable. The headless Ubuntu retest on 1.4.206 passed 3/3 with no agent_prompt_stalled, and a deliberately delayed 20s launch no longer fails at 8s. Thanks for the thorough repro! If you still see this on a current build, please reopen with details and we'll take another look.
feat(orchestration): make multi-agent workflows durable (stablyai#16904)