feat(runners): long-running agent sessions in their own container - #7487
Draft
joeyorlando wants to merge 12 commits into
Draft
feat(runners): long-running agent sessions in their own container#7487joeyorlando wants to merge 12 commits into
joeyorlando wants to merge 12 commits into
Conversation
A Runner is one long-running agentic session in its own pod, attachable and steerable while it runs. These are the generic building blocks a system like crab-env can be assembled from, rather than that system productized. Data model. The runner row is the source of truth and the pod is cattle, which is the inverse of the skills sandbox (an ordered replay log with no live process). State transitions are compare-and-set, so a reconciler pass racing a user's stop loses instead of resurrecting a torn-down session. Runtime. Each runner is a Job, not a Deployment: a session runs to completion, and a Deployment would restart a finished one forever and a crashed one behind the user's back, re-executing side effects the first attempt already had. tmux is PID 1 in every pod, which is what makes a session attachable and steerable at all. Steering has two modes: a FIFO the Archestra runner-agent reads at a turn boundary, so a message can never interleave with a tool call in flight, and tmux send-keys for bring-your-own CLIs that own their own input loop. Cluster mutations serialize on the existing cluster-lease table under a new scope rather than a leader election. Credentials. An agent declares what its runner needs and at which scope. Shared values come from the agent's secret bag; per-user values come from the invoking person's own store, for credentials that carry an individual's identity upstream. A user who has not supplied one gets a 409 with a machine-readable code, and reads the field list from the preflight endpoint. Those stay separate from shared credentials only an administrator can fix: asking someone for a value they cannot provide is worse than failing. Wiring. A runner needs no proxy or gateway configuration of its own. The LLM proxy URL, a personal-scope virtual key (so spend attributes to the human who started it) and the user's gateway bearer are all derived from the invoking identity. Egress to the platform is a second NetworkPolicy selecting only runner pods, so it composes with the environment's own policy without widening anything for MCP servers. Steer, stop and delete are restricted to the runner's creator or a runner admin, deliberately narrower than the runner:update permission that reaches the route: a session runs under its creator's credentials. ARCHESTRA_RUNNERS_ENABLED is an independent switch rather than one falling back to ARCHESTRA_BETA, and the API reports the feature only when the Kubernetes runtime is configured too.
Five Archestra MCP tools — start_runner, list_runners, get_runner, send_to_runner, stop_runner — so an agent can spin up a long-running session, watch it, talk to it and shut it down without anyone touching the API directly. This is what makes the crab-env shape reproducible from inside the product: a chat agent asked to take on a task starts a runner for it, then relays progress. The identity a runner acts as is always the gateway-authenticated caller, never a tool argument — a model must not be able to start a session as someone else, and the session carries that person's credentials. Steering and stopping additionally require having started the session (or runner:admin), the same rule the REST routes apply. When a start is blocked only because the caller has not supplied a credential the agent declares, the tool says which ones and where to add them, so the agent can tell the human something actionable rather than relaying an opaque failure. Route and tool now share one start path, so the identity rules and the failure handling cannot drift apart.
Fleet-level signals for a feature whose whole cost is holding pods: how many sessions are alive in each state, how they end, how long provisioning takes, and how much steering happens. Labelled only by closed sets — never by runner id, agent, image or user, which would explode series cardinality; per-runner detail already lives in the database. The active gauge is rewritten from each reconcile pass and reports zeros explicitly, so a state that empties clears instead of holding its last non-zero sample forever. Termination outcomes distinguish a completed session from a failure, a user's stop, and each expiry clock. A runner can have both clocks set, so the idle one is only reported when it has actually elapsed.
…bjects A runner image runs arbitrary code holding the caller's credentials, so it is now held to the same trusted-registry rule an MCP server image already is. An environment with no configured registries places no restriction; one with a list admits only those, and anything else is refused before a workload exists. Runtime-created objects also carry owner references now, so a release uninstall collects them instead of leaving orphaned Jobs, Secrets and network policies behind. Resolution is best-effort: an unavailable role lookup logs and proceeds rather than blocking a start.
…aults Review findings, in severity order. A privileged runner pod holds host devices and full capabilities, so configuring one is node-level access rather than an agent setting. Two comments claimed an admin gate that did not exist: any member able to edit an agent could set the flag and start a runner from it. The flag is now refused unless the writer holds runner:admin. runnerSecretId was accepted in the agent request body while nothing server-side ever wrote it. A member could point an agent at any secret row in the deployment — another user's personal credential, an MCP server's env bag — declare a shared credential naming one of its fields, and have the value injected into a pod whose image they also chose. The column is now omitted from both agent schemas, so only server code can set it. The per-runner virtual key was never revoked despite the schema saying it was: a stopped session's key kept working and kept charging its creator. Teardown now revokes it. A launch that failed after the Secret was written left it behind holding the caller's personal credentials, because `failed` is terminal and the reconciler never revisits those rows. The failure path tears down first. stop() marked a runner stopped even when the lease was held and nothing had been removed, hiding a still-running pod from every sweep that would have caught it. It now stays in `stopping` for the next pass. Runner pods got no DNS egress. Once any egress policy selects a pod its egress is clamped to the union of selecting policies, and no other policy selects these — so every session would have failed to resolve the platform's own hostname, surfacing as a generic agent failure. Both resolver shapes the other runtimes handle are now allowed. Also: the audit invariant test was failing because two tables were marked audited with no registered route — runner lifecycle already has its own append-only surface in runner_events, and personal credentials have no administrative CRUD, so both are now explicitly not audited with that reasoning. /api/config reported a different predicate than the routes gate on, so the UI could advertise a feature whose endpoints answer 404. Agent-supplied environment entries could override the proxy URL the launch path refuses to run without. The MCP start tool skipped the feature gate. A steer message beginning with a dash was parsed by tmux as its own options, and an embedded newline submitted half a message on both delivery paths. Every comment that described behaviour the code did not implement is either now true or gone.
…down Three faults in the reconciler's state machine, all reachable without artificial simultaneity because two web replicas sweep on independent timers. The claiming transitions were compare-and-set but the terminal ones were not, and only the explicit-stop path took the lease. So one replica could conclude a session had failed while another was still tearing the same session down, and that second replica's unguarded write would overwrite the verdict — a crashed session reported as cleanly stopped, with the termination metric counted twice and a timeline that contradicts itself. Both terminal transitions are now guarded, and the natural-completion paths take the lease the same way the explicit stop already did. A runner in `stopping` was wedged permanently: no branch in the reconciler advanced one, so a process that died between claiming the state and tearing down left a Job running with nothing ever collecting it. `stopping` is now resumable — the next pass retries the teardown and finishes the transition. Teardown swallowed every non-404 error without retrying, while callers went on to write a terminal state regardless. A terminal runner leaves every sweep, so one transient API error stranded the Job, Secret and network policy for good behind a row claiming success. Deletions now retry like the create path does.
The harness a Runner starts from when its agent names no image of its own. It is deliberately thin: the model, the tool set, the policies and the budget are all resolved by the platform behind the LLM proxy and MCP gateway this process talks to, so the loop's own job is to keep a conversation going, print it legibly for anyone attached to the tmux session, and take direction from a human without losing its place. Steering is consumed at a turn boundary, never mid-turn. The control plane writes one line per message into a FIFO; the loop reads it continuously but only folds messages into the conversation between turns, so a steer cannot be spliced into a tool call in flight. With nothing left to answer the loop parks on that channel rather than spinning, which is what makes a session that idles for days almost free — the long-running-monitor case this feature exists for. The gateway is dialled as an ordinary external MCP client with the invoking user's bearer. The pod has no privileged path back into the platform, so the tool set is exactly that person's. Configuration is read once and nothing is guessed: a missing value names itself and exits, because a pod started wrong is far easier to diagnose than an agent quietly talking to the wrong place. The image installs with pnpm rather than npm so the AI SDK version stays governed by the workspace catalog instead of a copy that drifts, and it runs as the base image's existing unprivileged user — an image that only works as root would push every deployment towards a privileged runner.
The interactive shell already built for MCP servers now serves both: it keeps xterm lifecycle, fitting, resize and the status chrome, and takes the connection itself as a prop. The two features speak different WebSocket messages, but that difference was never part of running a terminal, so the MCP component is now a thin adapter over the shared one and the runner component is another. The terminal reconnects on a session key rather than the transport's object identity, so a caller that forgets to memoize cannot cause a reconnect loop. Attaching runs `tmux attach` rather than opening a shell: the point is to land in the pane the agent is already working in, seeing its scrollback and typing into the same session — which is what makes interjecting possible. Detaching leaves it running, so closing a tab never ends a session mid-task. Terminal dimensions ride the SPDY resize channel; without it tmux would keep redrawing to a size nobody is looking at. Authorization matches the REST and MCP surfaces: attaching reaches a shell holding the creator's own credentials, so only they or a runner admin may do it, and every attach is recorded on the session timeline. Output is a separate read-only view, because the moment you most want to read a session is after it failed and there is no live pane left to attach to. The detail page mounts the attach only while its tab is showing — a background tab holding an exec open against a pod is a cost with no reader.
…nnect Two things a closer read of the existing MCP viewer turned up. The log readout was already extracted once, as LogConsole, precisely so every log surface in the product reads the same — pod logs, connector sync runs. The runner panel had grown its own near-copy instead, which would have drifted from the others on the first styling change. It now uses the shared one and keeps only what is genuinely its own: the subscription and the follow-the-tail behaviour. Neither the MCP terminal nor its logs viewer re-subscribed after a WebSocket reconnect, though the client exposes a connection-change hook for exactly that reason. The server forgets every subscription along with the socket it was made on, so a dropped connection left a terminal that looked live and received nothing until the reader thought to refresh. Both terminals — the MCP one included, since it now shares the component — and the runner log panel re-open on the connection edge.
The agent columns arrived a step after the tables only because that is the order the work happened in; nothing depends on them landing separately. One migration is what a reader would expect for one feature, and it is one fewer step to reason about on the way to production.
joeyorlando
force-pushed
the
jorlando/agent-runners
branch
from
August 26, 2026 22:07
57fa85f to
7a939ea
Compare
The availability gate asked whether the two orchestrator environment variables were set. The client loader also falls back to the ambient ~/.kube/config, which is how a developer machine runs MCP server pods — so on exactly the setups where the Kubernetes runtime demonstrably works, Runners reported themselves unavailable and every endpoint answered 404. It now asks whether a client can be built at all, cached because the answer involves the filesystem. Found by running the page locally: the API reported the MCP runtime enabled and Runners disabled on the same deployment, which is not a state that should be reachable.
The runtime worked and the page was empty, because nothing in the product could make an agent runnable or start a session. Three pieces close that. An agent's Advanced settings now carry runner configuration: the image, command, steering mode, lifetime, and the credentials the session needs with the scope of each. The privileged toggle only appears to someone who could actually grant it — the backend refuses it either way, but a control you cannot use should not be offered. Runners can be started from the page, and a start that needs a personal credential asks for it inline rather than failing. The agent declares what it needs, the person supplies it once, and the runner starts in the same action — being sent elsewhere to paste a token and come back is the friction the whole declaration mechanism exists to remove. Account gains a Credentials section for the same values outside that flow. They stay write-only: replaceable, removable, never readable, and invisible to administrators.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Generic building blocks for long-running agentic sessions — a Runner is one agent session in its own Kubernetes pod, attachable and steerable while it runs. The goal is that a system like our crab task-environments could be assembled from these pieces inside the product, rather than crab itself being productized.
Behind
ARCHESTRA_RUNNERS_ENABLED, an independent switch — a runner carries the credentials of whoever started it, so theARCHESTRA_BETAmaster switch must not enable it by implication. The API reports the feature only when the Kubernetes runtime is configured too.Why Kubernetes and not Dagger
Worth stating, since Dagger was the obvious candidate:
Long-running interactive sessions are the negation of the replay model, so Runners ride the pod path — where bidirectional attach already works — and the skills sandbox keeps Dagger untouched, where replay determinism is the feature.
How it works
One Job per session, not a Deployment. A session runs to completion; a Deployment would restart a finished one forever and a crashed one behind the user's back, re-executing side effects the first attempt already had.
tmux is PID 1 in every pod, which is what makes a session attachable and steerable at all. Steering has two modes: a FIFO the Archestra runner-agent reads at a turn boundary, so a message can never interleave with a tool call in flight, and
tmux send-keysfor bring-your-own CLIs that own their own input loop.Credentials are declared per agent, at a scope.
sharedvalues come from the agent's secret bag;per_uservalues come from the invoking person's own store, for credentials that carry an individual's identity upstream — a personal Claude token, a personal GitHub PAT. Someone who hasn't supplied one gets a 409 with a machine-readable code and reads the field list from a preflight endpoint, so the UI can prompt them. Those stay separate from shared credentials only an administrator can fix: asking someone for a value they cannot provide is worse than failing.No proxy or gateway configuration. The LLM proxy URL, a personal-scope virtual key (so spend attributes to the human who started it) and the user's gateway bearer are all derived from the invoking identity. Egress to the platform is a second NetworkPolicy selecting only runner pods, so it composes with the environment's own policy without widening anything for MCP servers.
Five MCP tools —
start_runner,list_runners,get_runner,send_to_runner,stop_runner— so an agent orchestrates sessions by tool call. The identity is always the gateway-authenticated caller, never a tool argument.Review findings fixed in this PR
Two independent adversarial reviews ran against the branch. Both found real bugs; the last three commits are the fixes.
Privilege escalation, two paths.
privilegedwas a plain field on the agent's runner config with no gate, despite comments claiming one — any member who could edit an agent could grant themselves host root. AndrunnerSecretIdwas accepted in the agent request body while no server code ever wrote it, so a member could point an agent at any secret row in the deployment, declare a shared credential naming one of its fields, and have the value injected into a pod whose image they also chose.No DNS egress. Once any egress policy selects a pod its egress is clamped to the union of selecting policies, and no other policy selects these — so every session would have failed to resolve the platform's own hostname, surfacing as a generic agent failure.
A standing credential per session. The per-runner virtual key was never revoked despite the schema saying it was: a stopped session's key kept working and kept charging its creator.
Terminal-state race. The claiming transitions were compare-and-set but the terminal ones weren't, and only the explicit-stop path took the lease — so one replica could report a session failed while another overwrote that verdict with "stopped".
Also: a failed launch stranded a Secret holding personal credentials; a runner in
stoppingwedged permanently with a live Job; teardown swallowed transient errors without retrying behind a row claiming success; agent-supplied environment could override the proxy URL the launch path refuses to run without; the MCP start tool skipped the feature gate; a steer message starting with a dash was parsed by tmux as options, and an embedded newline submitted half a message.Every comment that described behaviour the code did not implement is now either true or gone.
Tests
86 passing, on real Postgres and a real secrets manager, with no mocking of the system under test. They pin the behaviour that matters: compare-and-set transitions that make a reconciler lose to a user's stop, expiry arithmetic, concurrent event-sequence allocation, credential rotation discarding the superseded secret, per-user isolation, the privileged-config gate, and assertions that secret values never appear in a serialized row or a pod spec.
Not in this PR
runner-agentimage — the default harness. Bring-your-own-image works today; the default image reference has nothing behind it yet.pending-forever case where a request dies before the workload is created.Open question for review
start_runneris a trusted Archestra tool, which by convention bypasses tool-invocation policy. That calibration was made for lower-blast-radius tools; this one starts an autonomous container running under the caller's real credentials, with the initial instruction as an argument. If a chat agent is prompt-injected through untrusted tool output, it could start a background session that spends the user's quota out of sight. Worth deciding whether starting a runner should require explicit user confirmation, unlike list and get.