Repository navigation
feat(cli): orca serve stats [--json] — live headless-serve runtime stats - #24
Open
LesleyMurfin wants to merge 24 commits into
Open
LesleyMurfin wants to merge 24 commits into
LesleyMurfin wants to merge 24 commits into
Conversation
LesleyMurfin
force-pushed
the
ai/serve-stats-json-10608
branch
2 times, most recently
from
July 25, 2026 20:47
25ecf5c to
348f368
Compare
This was referenced Jul 25, 2026
8 of 9 tasks
svc-orca
Bot
force-pushed
the
ai/serve-stats-json-10608
branch
from
September 7, 2026 19:55
716a8d1 to
32ee3ed
Compare
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Add a read-only CLI + serve.stats RPC that returns live factory occupancy from a running runtime (agents, tasks, terminals, worktrees) plus version, uptime, and the post-bind WebSocket port (or null). - Register under SERVE_COMMAND_SPECS next to bare `serve` - Honor --environment / pairing (only bare serve ignores remote selection) - Wire setServePort from RPC server; optional-chain for runtime proxies - Tests: aggregator, RPC registry dispatch, port-bind regression Closes stablyai#10608
Add handler unit tests for serve.stats RPC dispatch, factory count fields, null port rendered as "none", and JSON mode shape.
… ptys Adds runtimeId plus counts.browserPages, counts.browserPagesRetained and counts.terminalsUnverifiable to the (unshipped) serve.stats result. Retained pages hold one of the runtime's 256 page slots with no TTL and no reaper (stablyai#14552); the disconnected-pty count makes leaked ptys visible (stablyai#9819, stablyai#9479) without claiming they exited; runtimeId distinguishes a same-generation runtime from one that restarted underneath the caller (stablyai#9585).
serve.stats documents live current-state counts, but countTasks() counted every row in the tasks table. Settled tasks persist until an explicit orchestration reset, so the advertised task count grew monotonically. Exclude completed/failed; keep blocked, which is resumable work.
stop() empties activeTransports/transports before stopping them, then rethrew the first rejected stop before clearing the advertised port, so serve stats kept reporting a dead port after a failed shutdown. Clear the port unconditionally, then rethrow.
`serve stats` reported only scalars, so the histograms operators actually needed were hand-tabulated from worker-list and task-list output (stablyai#13047, stablyai#19388, stablyai#18737). Adds three fixed-key breakdowns to the same single walk: - counts.tasksByStatus: one grouped scan (countTasksByStatus), every TaskStatus always present. Includes completed/failed, which counts.tasks excludes, so it deliberately does not sum to counts.tasks. - counts.agentsByState: working/permission/idle/unknown. PTY-backed agents are bucketed from their own current-incarnation evidence (retained hook status, prompt lifecycle, live-observed title); structured sessions and ptys with no current evidence report `unknown`, because identity and liveness prove presence, not turn state (stablyai#19548). - counts.workersByTerminalState: countWorkerTerminalInventory keyed by all six WorkerTerminalListState values, so reclaimable/release_unknown pileups are visible without worker-list. TaskStatus and WorkerTerminalListState move to shared leaf modules so the wire contract keys on the same unions the DB derives (shared cannot import main); FleetTerminalState now aliases the latter instead of repeating it.
…tract change Upstream createTask now requires a runId, and the serve-stats agent-state read is a new hook-row consumer the census has to pin.
…lay p99 serve stats could show every Orca-side count while the host itself was saturated: stablyai#14552 and stablyai#19312 both read as 'Reconnecting to remote runtime' network faults for days (loadavg 105-112, systemd reporting the unit healthy), and stablyai#12588's 60 GiB terminal child degraded SSH with no visible signal. Adds host.{loadAverage1m,cpuCoreCount,memoryTotalBytes,memoryAvailableBytes,memoryAvailableSource,swapUsedBytes} under its own key (never inside counts, since these are host-wide and not Orca-attributed) and health.eventLoopDelayP99Ms. Unmeasurable fields are null and render n/a, never 0, so Windows' meaningless os.loadavg() zeros cannot read as an idle host. Linux prefers /proc/meminfo MemAvailable over os.freemem() and labels the source; every procfs read is guarded so a container without it yields null instead of failing the command. The delay histogram is one long-lived monitorEventLoopDelay enabled with the RPC listener and reset on read, so each read reports the window since the previous one rather than a multi-day percentile that goes stale-flat.
`counts.browserPages` counted only RuntimeBrowserPageRegistry, which `browserTabCreate` populates on the client-hosted branch alone. A headless serve has no renderer, so `tab create` takes the offscreen branch and the page was never published there — the field reported 0 for a live, listable, agent-opened tab, which is exactly the population stablyai#14552 asked for. Count the WebContents registration map the bridge reads as well, de-duplicated by page id, and read it via `getRegisteredTabs` rather than `tabList` so a stats read deregisters nothing. `browserPagesRetained` stays scoped to client-hosted pages and now says so: an offscreen page has no separate host to lose. `counts.terminalsUnverifiable` incremented on `connected === false`, but `onPtyExit` clears `connected` while recording an `exited` liveness verdict from a host-delivered exit code. Confirmed-dead ptys therefore landed in the one field documented as carrying no evidence of exit and as not authorizing cleanup. Split the bucket on that verdict: `terminalsExited` for a positive death certificate, `terminalsUnverifiable` for absence of contact only, with anything ambiguous staying on the conservative side. `terminalsUnverifiable` narrows in meaning; `serve.stats` has not shipped in a release, so no consumer contract breaks.
The liveness register documented an invariant it did not enforce: an earned death certificate is kept, and nothing weaker than a host-delivered exit frame may reach it. Both doubt writers ignored that. The controller-inventory sweep deleted the entry for any id an answering provider disowned, and recorded `unverifiable` for any id no provider could be asked about — so a host-delivered exit only held between `onPtyExit` and the next `terminal list` / `worktree ps` / `worktree rm`, after which `serve stats` reported the proven death as `terminalsUnverifiable`, the field whose whole meaning is that cleanup is NOT authorized. Enforce it in the register rather than at the sweep's call sites: three writers in that one sweep shared the bug, plus the stop-error path, and loss of contact is never a retraction of positive evidence anywhere (docs/reference/ssh-execution-boundary.md). Doubt stays exactly as clearable as before. Disposal is unaffected: retiring a pty or reusing its id deletes the entry directly (onPtySpawned, registerPty, dropDisconnectedPtyRecord), `getServeStats` counts from `ptysById` so a disposed pty contributes to neither counter, and the register's orphan ratchet still bounds retained verdicts at 256.
If current.stop() rejects in widenWebSocketBind(), the servePort clear on the next line never runs. When the loopback recovery in recoverWebSocketBindAfterFailedWiden then also fails, wsBoundHost and the transport are torn down but servePort stays stale, so serve stats can report a port with nothing listening on it. Also resyncs en-runtime-required.json: TerminalPane.minimumContrast.* text had drifted from en.json after an unrelated upstream change, failing the runtime-required-catalog test and static-analysis CI checks. Regenerated via sync:localization-runtime-catalog; unrelated to serve-stats itself.
…roup pids Three resource-exhaustion signals `serve stats` could not answer, each null (never 0) wherever the platform cannot measure it: - counts.browserPageMemoryTotalBytes / browserPageMemoryMaxBytes: resident set of the renderers behind counts.browserPages, de-duplicated by pid because Electron may back several pages with one renderer. stablyai#14552 was one 1.3 GB page among six, which a total alone hides, so the max ships beside it. One /proc/<pid>/status VmRSS read per pid — no subprocess, no `ps` sweep. Page counting moves into the same collector, since the bridge's page-id map is both the page population and the only handle a footprint read can start from. - health.longPolls: active vs cap for the total, ask, browser-host and specialized admission pools. stablyai#19342 hit `runtime_busy` at loadavg 4.8 with 44 GB free and could only find the cap by unpacking app.asar. The counters and caps live on the RPC server, so it registers a reader with the runtime where it already sets the serve port — a pull, so nothing pushes on the long-poll hot path and the snapshot cannot lag the counters. Null when no listener is serving. - host.pids: cgroup v2 pids.current / pids.max, resolved through /proc/self/cgroup and walked to the nearest enclosing cgroup with the pid controller enabled. A fixed /sys/fs/cgroup read only works inside a container: the kernel omits controller files from the root cgroup, and stablyai#18789's pids.max=4096 was a systemd unit limit. The literal `max` maps to null (unlimited), never to a number. Verified live on a built headless serve: three offscreen pages reported total=259588096 max=88240128, matching an independent /proc read of the three renderer pids exactly; two held terminal.wait calls moved total.active 0 -> 2 -> 0 against caps 16/8/8/12; host.pids.max matched the unit cgroup's 16384 exactly, with current tracking pids.current to within the measuring process's own tasks.
…d use POSIX cgroup paths
svc-orca
Bot
force-pushed
the
ai/serve-stats-json-10608
branch
from
September 11, 2026 06:29
52c0038 to
2776ecd
Compare
- Route webContents OS process ID lookup through RuntimeDesktopSurface to keep electron out of runtime modules. - Extract serve stats collection logic to serve-stats-aggregation.ts, removing the max-lines bypass. - Install fake AppEnvironment in serve-stats-lifecycle.test.ts.
Merge order: land after #21, and before #19This is a merge-ordering note, not a review blocker. The coupling below is file adjacency and rebase churn only — it is not a functional dependency. This PR is correct and self-contained on its own: merging out of the suggested order produces a merge conflict someone has to resolve, not a broken build or a missing symbol. Shared with #21 — 1 file:
Shared with #19 — 3 files:
|
This was referenced Oct 9, 2026
ACO structural triage (prompt_version=1.2-quick)Workspace: factory fork Scope (≤15 production targets)Feature slice
Blast radius (rg, non-AST)
Findings (sev-ordered)P1 — Lifecycle unit test fails (structured native-chat fixture)
P1 — Local tip ≠ PR tip
P1 — CLI help implies bare JSON; runtime emits RPC envelope
P2 — ELD monitor never disabled on RPC stop
P2 —
|
ACO triage follow-up — fixes applied (local workspace)Applied on local tip Checklist from triage
Files touched
VerificationResidual
|
…ests for upstream native chat
Disable the process-wide ELD histogram when RPC start fails or stop runs, document the CLI --json RPC envelope/.result contract, require workspace permission on serve.stats, drop the browser-pages unknown cast, and null-guard structured worker host scope when location is missing.
selectFreshExplicitAgentStatus accepts a nullable handle after the main merge, but the join API is Iterable<string>. Mirror the paneKey guard so a missing handle becomes an empty list instead of a type error.
Recovery + ACO close-outLocal worktree had diverged (merge wreckage + unpushed ACO fixes). Reset to PR tip
Verification
PR tip now includes the ACO fixes; draft cleared. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
orca serve stats [--json]— a first-class CLI command andserve.statsRPC that returns live factory occupancy from a running headlessorca serve(agents, tasks, terminals, worktrees, version, uptime, post-bind WebSocket port), plus host load/memory pressure and event-loop health. Scripts and agents can query the runtime without scraping the UI or openingorchestration.db.Closes stablyai#10608
Updates since this draft was last reviewed
Since the "Testing"/"AI Review Report" sections below were written (commit
348f368), three feature commits and two fix-ups landed — full detail instablyai/orca#10612(the upstream PR this fork branch feeds):50a836f9— addshost.*(load average, CPU cores, memory total/available + source, swap) andhealth.eventLoopDelayP99Ms. This supersedes the "Host CPU/RAM metrics" non-goal below — it's now in scope and shipped.82bfc0ff— splits proven pty exits (terminalsExited) from unverifiable ones, and separates headless browser pages from client-hosted ones.2817d68e— keeps a proven pty exit recorded across inventory refresh.a53803ba(current head) — clearsservePortwhen a WebSocket-widen recovery also fails (open CodeRabbit finding on feat(cli): orca serve stats [--json] — live headless-serve runtime stats stablyai/orca#10612), plus an unrelated i18n catalog resync that fixes CI (en-runtime-required.jsonhad drifted fromen.jsonon an unrelated key, unrelated to this feature).The JSON contract, non-goals, diff size, and commit reference further down are historical for
348f368; seestablyai/orca#10612for the current contract and CI status.Screenshots
No visual change.
User-visible change
Stable JSON contract:
{ "version": "1.4.x", "uptimeSeconds": 1234, "port": 6768, "counts": { "agents": 0, "tasks": 0, "terminals": 0, "worktrees": 0 } }portis the post-bind WebSocket port, ornullwhen no WS listener is active.counts.agentsis currently live agents (StatsCollector), not lifetime spawned totals.Related
orca serve stats [--json]— live headless-serve runtime stats stablyai/orca#10608 (this PR implements the factory-stats feature end-to-end once merged and deployed).Implementation notes
serve statsregistered underSERVE_COMMAND_SPECSnext to bareserve; handler mirrorsstatus(client.call+printResult).['serve']ignores remote env;serve statshonors--environment/ pairing.getServeStats()aggregator;setServePortset from the RPC server after WS bind (optional-chained for remote runtime proxies, same pattern asactivateRecentPtyPathCandidateTracking?.()).serve.statsregistered in the method registry with dispatch coverage.Explicit non-goals
Testing
oxlinton all touched files (0 warnings / 0 errors); pre-commit oxlint + oxfmt on committsc --noEmitforconfig/tsconfig.tc.cli.jsonandconfig/tsconfig.node.jsonsrc/main/runtime/serve-stats.test.tssrc/main/runtime/rpc/methods/serve-stats.test.tssrc/main/runtime/runtime-rpc.test.ts(includes port-bind regression + runtime-proxy optional-chain path)pnpm lint/pnpm typecheck/pnpm test/pnpm buildnot run as a single gate in this session (scope limited to the serve-stats surface + registry suite above)Live headless serve (PC factory — correct surface)
Target runtime (already running; not a desktop client):
orca-serve@factory.servicesvc_orca/data/opt/revive/orca_serve/data/opt/revive/orca_serve/share/orca-linux.AppImage→ versioned store/data/opt/revive/orca_serve/state/factory/config/orcaWhat we verified against that live factory:
orca-serve@factory, AppImageserve --port 6768 …)status→ reachable)method_not_found: serve.stats(expected until the serve process runs a build that includes this PR)Unknown command: serve stats(feature not in deployed binary yet)orca-serve@factory(or canary on 6771), then assvc_orcarunorca serve stats --jsonagainst the live headless serve and confirm the JSON contract aboveNot part of the plan: starting a second Electron “client” process, or asking upstream maintainers to run a manual smoke for us.
Commands run
AI Review Report
Review focused on the serve-stats surface (CLI, RPC, runtime aggregator, remote-selection guard, port lifecycle).
nullcovers unix-only and WS bind failure.serve statshonors--environment/ pairing; bareservestill ignores remote selection.setServePort?.()safe for partial remote runtime proxies.Changes from review: optional-chain
setServePortfor runtime proxies (matches existingactivateRecentPtyPathCandidateTracking?.()); register CLI underSERVE_COMMAND_SPECS; keep port sourced from post-bindresolvedPort.Security Audit
--jsononly changes output encoding.Notes
svc_orcaunder/data/opt/revive/orca_serve(factory :6768, canary :6771). CLI extract noise from AppImage is expected when invokingorcaunder that prefix. Live green e2e requires swapping the serve AppImage/serve-root to a build that includes this PR, then restarting the unit — not replacing a user desktop client.X handle
@LesleyMurfin
Provenance
git worktreefromexternal/orca/fork(not via/orca launch)mtl-02-dev-001orca-serve@factoryassvc_orca/data/projects/external/orca-wt/ai-serve-stats-json-10608ai/serve-stats-json-10608348f368297455490141c1dc96ede28e5234a0d34