jb-ai-orchestrator

A deployable agent harness. Point it at a source holding an AI playbook — .cursor/, .claude/, or a plain AGENTS.md — and it provisions an isolated workspace, discovers the instructions, rules, skills and subagents inside, and runs a full tool loop against any model LiteLLM can reach. It is an HTTP microservice, so one deployment serves many callers, and a paused run can be resumed by a different replica.

Python 3.11 · FastAPI · Postgres · Apache-2.0

Designed and built by Joseph Benraz

Contents

• What makes it different
• Quick start
  • What the shipped defaults allow
• Security model
• How it works
• Features
• How it compares
• API reference
• Configuration
• Workspaces, input and durable output
• Running many instances
• Development
• Limitations and open work
• Documentation
• License

What makes it different

Agent harnesses usually ship as a CLI or an IDE feature that runs on one developer's machine. This one is a service, which changes what you can build on it:

• The playbook is an input, not a build artifact. Callers hand over a source location in the folder field: a git URL, an HTTP(S) URL to a .zip/.tar.gz archive — including a blob or bucket object, provided it points straight at the file — or a path the service can reach. The service detects which layout it is looking at, discovers the primitives inside — rules, skills, commands and subagents — and composes the system prompt from what it finds; initiate returns that catalog. Nothing is inlined by the caller and nothing is baked into the image.
• Pauses survive the process. When a run needs a human — a UI tool, the built-in ask_user, or a hook asking permission — it persists its state to Postgres and returns. Any replica can resume it. There are no sticky sessions and no in-process waiters.
• Human interaction is a protocol, not a prompt. The service speaks AG-UI over SSE, streams tokens and tool calls, and accepts batch interrupt resolution, so a UI can resolve or cancel several open interrupts in one request.
• The UI can push its own tools. Frontend tool catalogs are supplied per run, so many applications can share one deployment without clobbering each other's tool lists.
• Many MCP servers, unambiguous names. Every tool is suffixed with the name of the server that provides it (search_google, weather_general), so collisions across servers are impossible and every call is traceable.
• Provider-neutral. Model choice is a per-request string resolved by LiteLLM. Nothing in the harness is tied to one vendor.

Quick start

Prerequisites: Docker Desktop, or — if you bring your own infrastructure — Postgres 16+ and a LiteLLM endpoint reachable from the service. At least one MCP server is needed either way.

Configuration is per environment: ENV selects the directory under src/.env/, and that directory's .env is the only one loaded. Docker and local runs therefore read different files, so follow one path below, not both.

1a. Docker, with the shared stack

The shared stack deploys what the agent depends on — Postgres, LiteLLM and Ollama — as its own Compose project, so the agent joins infrastructure instead of carrying it. Copy the compose-layer example, then start both:

./start-shared-mac.sh     # macOS: Ollama stays on the host for the Metal GPU
./start-agent-mac.sh      # or ./start-agent-mac.sh --ensure-shared to do both

On Windows use start-shared-windows.bat and start-agent-windows.bat; there Ollama runs in a container and the default model is pulled for you.

There is no file to copy first. On their first run the launchers create both git-ignored environment files from the tracked examples, and generate this machine's LITELLM_MASTER_KEY and POSTGRES_PASSWORD into .env. Nothing is overwritten afterwards, so the credentials keep matching the data volumes already created from them, and no two clones share a password. Running docker compose directly instead of the launchers fails with the name of the variable to set rather than falling back to a well-known default.

The launcher never deploys infrastructure. It joins a running jb-ai-shared stack, and if that is not up it stops and prints the command to run. Since the agent then reaches services by name, docker-compose.stack.yml overrides DATABASE_URL, LITELLM_BASE_URL and LITELLM_API_KEY; every other setting still comes from src/.env/docker/.env.

Two environment files, two jobs: .env at the root configures the stack itself and is git-ignored, while src/.env/docker/.env configures the agent process.

1b. Docker, against your own Postgres and LiteLLM

The container sets ENV=docker, and Compose loads src/.env/docker/.env. Edit that file — it already points at host.docker.internal to reach Postgres, LiteLLM and MCP servers on your host:

DATABASE_URL=postgresql+asyncpg://postgres:<password>@host.docker.internal:5432/pyagent
LITELLM_BASE_URL=http://host.docker.internal:4000
LITELLM_API_KEY=<your-litellm-key>
MCP_SERVER_URLS=[{"name": "local", "url": "http://host.docker.internal:8001/mcp"}]

docker compose up --build -d

./start-agent-mac.sh --no-shared (or start-agent-windows.bat --no-shared) runs exactly this, skipping shared-stack detection — the mode to use when Postgres and LiteLLM are managed elsewhere.

Only src/.env/docker/.env.example is tracked. The launchers copy it to src/.env/docker/.env, which is git-ignored and is where your own values belong. Standalone mode reaches Postgres and LiteLLM directly, so you must replace the __POSTGRES_PASSWORD__ and __LITELLM_API_KEY__ placeholders — the service refuses to start while any placeholder remains, naming the variables it found. The shared-stack path supplies both for you.

1c. Local, without Docker

Create the localhost environment file from the tracked example and fill in the values marked CHANGE_ME:

cp src/.env/localhost/.env.example src/.env/localhost/.env

DATABASE_URL=postgresql+asyncpg://postgres:<password>@localhost:5432/pyagent
LITELLM_BASE_URL=http://localhost:4000
LITELLM_API_KEY=<your-litellm-key>
MCP_SERVER_URLS=[{"name": "local", "url": "http://localhost:8001/mcp"}]

pip install -r src/requirements.txt
ENV=localhost alembic upgrade head
cd src && ENV=localhost python -m app.main

Run alembic from the repository root, where alembic.ini lives; ENV tells it which src/.env/<env>/.env to read DATABASE_URL from. On Windows PowerShell, set $env:ENV='localhost' once, then run alembic upgrade head and cd src; python -m app.main.

Either way, DATABASE_URL must use the postgresql+asyncpg:// scheme, not plain postgresql://, and MCP_SERVER_URLS must be an array of {name, url} objects. A bare list of URL strings is ignored rather than rejected, so unless you also set the numbered MCP_SERVER_URL_{n} variables, startup fails with no servers configured. MCP servers below covers each malformed shape.

2. Verify

curl http://localhost:8000/isalive

Interactive API docs are at http://localhost:8000/swagger, unless you set DOCS_ENABLED=false, which removes both /swagger and /openapi.json. List the tools your MCP servers actually exposed, then run a request:

curl http://localhost:8000/v1/agent/getTools

curl -X POST "http://localhost:8000/v1/agent/executeRequest" \
  -H "Content-Type: application/json" \
  -d '{
    "task": "Search for technology updates and check the weather",
    "tools": ["search_google", "weather_general"],
    "model": "gpt-4o"
  }'

Tool names are whatever your servers expose, suffixed with the server name — replace the tools above with names from /getTools. Omit tools entirely to let the agent use everything available.

What the shipped defaults allow

The service authenticates no one, so what you just started is deliberately the least capable version of it. Each item below is off until you turn it on, and Security model explains what changes when you do.

Out of the box	To change it
Every port binds to 127.0.0.1, so nothing is reachable from the network	BIND_ADDR=0.0.0.0, once something in front of it authenticates callers
No browser origin can call the API; requests from a page are refused	CORS_ALLOWED_ORIGINS=https://your.app
/swagger and /openapi.json are served	DOCS_ENABLED=false
execute_local is absent — the agent cannot run shell commands	LOCAL_SHELL_ENABLED=true, ideally with SHELL_COMMAND_ALLOWLIST naming what may run
Workspace hooks do not run	HOOKS_ENABLED=true, remembering that hooks are shell commands the bound workspace supplies
LITELLM_MASTER_KEY and POSTGRES_PASSWORD are generated on this machine's first launcher run	supply your own in .env before that first run

Two of those the Docker quickstart relaxes for you, both confined to this machine. It sets ALLOW_INPLACE_WORKSPACE=true so a bind-mounted folder can be edited in place, with WORKSPACE_ALLOWED_ROOTS restricting that to the folder you mounted — startup refuses in-place access with an empty allowlist. And it lists http://localhost:5173 and http://127.0.0.1:5173 in CORS_ALLOWED_ORIGINS, the origins the bundled ag-ui-demo runs on, so the demo works without editing configuration first. Both are relaxations of src/.env/docker/.env.example only; the code defaults stay closed, so anything not built from that file starts with no origin allowed. Remove both from your copy for any deployment that is not the local demo.

Skipping the launchers and running docker compose directly is supported, but then nothing has generated the secrets: Compose stops and names the variable to set rather than falling back to a default that would be identical everywhere.

Database schema

The container image runs alembic upgrade head before serving, which is why the Docker path above has no migration step. Running outside Docker you apply them yourself, as in the local sequence above:

ENV=localhost alembic upgrade head

Startup also calls init_db(), which creates the database if it is absent and issues create_all, so an empty database works without a manual step. The migrations additionally adopt databases that were first created by create_all. See docs/ORCHESTRATOR.md §12 for migration and backfill detail, and §13 for the schema itself.

Security model

This service authenticates no one. Every endpoint is open to anyone who can reach the port. That is deliberate — it is a component designed to sit behind a gateway that authenticates callers, terminates TLS and enforces tenancy — but it means the deployment topology is the security boundary, and the defaults assume you have not built that gateway yet.

So the shipped defaults are closed. Published Compose ports bind to 127.0.0.1, no cross-origin browser call is accepted until you name an origin, and the two capabilities that execute commands — the local shell and workspace-supplied hooks — are off until you turn them on:

Setting	Default
BIND_ADDR	127.0.0.1
CORS_ALLOWED_ORIGINS	empty — no cross-origin call accepted
LOCAL_SHELL_ENABLED	false
HOOKS_ENABLED	false
OUTPUT_BINDINGS_ENABLED	false
DOCS_ENABLED	true — set false off your own machine

Startup refuses three configurations outright rather than running them: a wildcard CORS origin combined with credentials, in-place workspaces with no WORKSPACE_ALLOWED_ROOTS to confine them, and any value still holding an unreplaced __PLACEHOLDER__.

The one thing to internalise before pointing this at anything real: a playbook is executable input. The rules, skills and hooks found in a bound workspace shape the system prompt, and hooks run as shell commands. Binding a workspace you do not trust is equivalent to running its code, and no harness — this one included — solves prompt injection. Constrain what a run can reach instead of relying on the model to refuse.

SECURITY.md has the hardening checklist for a real deployment, the full threat model, and how to report a vulnerability privately.

How it works

flowchart LR
    A[Caller] -->|folder + prompt| B[Bind workspace]
    B --> C[Scan playbook<br/>AGENTS.md · rules · skill catalog]
    C --> D[Compose system prompt]
    D --> E[Tool loop]
    E -->|local · MCP · frontend tools| E
    E -->|needs a human| F[(Persist state<br/>Postgres)]
    F -->|resume, any replica| E
    E -->|SSE tokens + tool calls| A

Loading

A run binds a workspace, scans it once per segment, composes an authoritative system prompt from the discovered instructions plus a lazily-loaded catalog of skills, commands and subagents, then loops: the model picks tools, the service executes them, oversized results are offloaded to .agent/offload/, and older history is summarized and compacted as the context grows. If a tool needs user input the loop persists and returns rather than blocking. docs/ORCHESTRATOR.md is the source of truth for every step.

Vocabulary

A few words carry precise meanings throughout the API, and the distinctions matter when reading the lifecycle rules:

Term	Meaning
Playbook	The instructions a caller points the service at: a .cursor/ or .claude/ directory, or a plain AGENTS.md. It is input, never baked into the image.
Session	A bound folder you can execute against repeatedly. Created by initiate, identified by orchestratorGuid. A thread is the AG-UI equivalent.
Run	One request executing against a session. Recorded as an ExecutionRun so a failed attempt is visible without disabling the session it belonged to.
Segment	The bounded slice of a run between two pauses — one tool loop plus its model calls, governed by RUN_SEGMENT_DEADLINE_SEC. A run that pauses for input resumes as a new segment, possibly on a different replica.
Interrupt	A pause needing a human: a frontend tool, the built-in ask_user, or a hook asking permission. Surfaces as awaitsResponse with a stateGuid.
Binding	Where a run reads input and writes durable output. Input can be bound in place or copied into a sandbox; durable output survives the session.

The harness-side concepts — adapters, primitives, loading policy, compaction — are defined in docs/ORCHESTRATOR.md §1.

Features

• Harness discovery — Cursor, Claude Code and generic adapters; eager loading of AGENTS.md (including .cursor/AGENTS.md), CLAUDE.md and always-on rules, with progressive disclosure for skills, commands and subagents. Project hooks run when enabled, and a permission: ask hook surfaces as an interrupt. An adapter is two methods — score a detection, return a manifest — over shared primitive discovery, so covering another tool's layout is around a hundred lines rather than a new subsystem.
• Built-in coding tools, namespaced *_local on workspace-backed runs: list_files, read_file (text plus images and PDFs), write_file, edit_file, create_file, create_folder, glob, grep, execute, write_todos, task for subagents, git helpers and ask_user. Binding durable output adds write_output_local, edit_output_local, read_output_local and list_output_local.
• Context management — token streaming, oversized tool results offloaded to disk and referenced by handle, LLM summarization of older turns, and tool message compaction on a character or token budget.
• Human interaction — AG-UI SSE runs, per-run frontend tool catalogs, canonical batch resume[] with resolve and cancel, a fan-out event stream, and Postgres-persisted holds resumable from any replica.
• Multiple MCP servers — named servers aggregated into one tool list with universal attribution, per-server fault isolation, and optional filtering of remote tools that collide with a built-in.
• Operational safety — path and shell allow/deny policy, workspace root restriction, size caps and clone timeouts, close endpoints with a 200/202 contract, run heartbeats and stale-run reconciliation.

How it compares

The harness follows patterns established by Claude Code and LangChain Deep Agents: root instructions and always-on rules for stable context, progressive disclosure so skills cost a catalog entry until they are needed, subagents to keep detailed work out of the parent context, and filesystem tools plus summarization and large-result offload for long-running work. None of that is unique, and neither is connecting several MCP servers at once.

What differs is the delivery model:

	Claude Code	Cursor agent¹	Deep Agents	This service
Form factor	CLI / IDE session	IDE / local agent	Python library you embed	HTTP microservice
Instruction source	CLAUDE.md, skills	.cursor/rules, AGENTS.md	Code you write	Any bound folder, scanned per run
Who supplies the playbook	The developer at the keyboard	The developer at the keyboard	The application author	The caller, per request
Multiple MCP servers	Yes	Yes	Via adapters	Yes, with universal name attribution
Human-in-the-loop	Interactive prompts	In-editor approvals	Framework interrupts	AG-UI protocol over SSE, batch resolve/cancel
Resume after restart	Local session resume	Local session resume	Application-selected checkpointer or store	Built in: Postgres-backed, any replica, no sticky sessions
UI-supplied tools	—	—	Application-defined	Per-run frontend tool catalog
Model routing	Anthropic models	Multiple providers	Your choice	Any LiteLLM model, per request

Durable resume is not the distinguishing part — Deep Agents runs on LangGraph, whose checkpointers and stores can be external and durable, and its hosted server persists state too. The difference is that here it is not something you assemble: the HTTP resume contract, the ownership rules that let a different replica claim a paused run, and the persistence behind them ship as the service's public behaviour.

¹ Cursor is an IDE product and this is a deployable service, so this compares delivery models rather than feature parity — it consumes the same playbook formats (.cursor/, AGENTS.md) that Cursor users already write. All four projects move quickly; check their own documentation for current behaviour.

API reference

Interactive documentation is served at /swagger while DOCS_ENABLED is true, which is the default; setting it false withdraws both /swagger and /openapi.json.

Health

Method	Path	Purpose
GET	/isalive	Health check

Agent

Method	Path	Purpose
POST	/v1/agent/executeRequest	Execute a request with optional MCP and AG-UI tools
GET	/v1/agent/getTools	List available MCP tools (plus legacy globally-bound frontend tools)
POST	/v1/agent/resumeRun	Resume a paused execution with an AG-UI tool result and optional model / max_tool_calls overrides
GET	/v1/agent/execution/{executionGuid}	Latest execution status and result metadata

Orchestrator

Method	Path	Purpose
POST	/v1/orchestrator/initiate	Bind a folder to a new session, provision the workspace, detect the orchestration type, return orchestratorGuid plus discovered agents and skills. No LLM call.
POST	/v1/orchestrator/execute	Run a prompt against an initiated session. Returns the result, or awaitsResponse plus stateGuid when input is needed. Send the complete current frontendTools list: their names are named in the system prompt so the model knows an interaction tool exists.
POST	/v1/orchestrator/resume	Continue an awaiting run with { orchestratorGuid, stateGuid, toolCallId, result } and optional model / maxToolCalls overrides. Pod-agnostic.
GET	/v1/orchestrator/{orchestratorGuid}	Session and run status. While awaiting input, also replays canonical interrupts and pendingToolCallIds from persisted state so authorized callers can rebuild interaction UI.
POST	/v1/orchestrator/{orchestratorGuid}/close	Close the session. 200 closed or 202 closing; retry after active runs finish or go stale. Removes service-owned runtime and workspace; never touches in-place caller input or durable output.

AG-UI runtime

Method	Path	Purpose
POST	/api/ag-ui/run	SSE run: bind frontendTools, stream output, receive tool-call callbacks
GET	/api/ag-ui/events	Fan-out SSE stream of all tool-call events
POST	/api/ag-ui/threads/{threadId}/close	Close thread lifecycle. 200 closed or 202 closing. Cancels holds and removes thread runtime and AG-UI-owned sandboxes only.
POST	/api/ag-ui/threads/{threadId}/abandon	Idempotently discard active holds. Non-destructive; leaves runtime and workspaces in place.

All three resume paths can select the model and tool budget for the new segment. AG-UI and orchestrator requests use model / maxToolCalls; the legacy direct-agent endpoint uses model / max_tool_calls. When omitted, a persisted await snapshot remains authoritative, preserving existing behavior. An explicit override is persisted if the segment awaits again, and switching models or providers replays the existing history without translation.

/api/ag-ui/events emits the same TOOL_CALL_START, TOOL_CALL_ARGS and TOOL_CALL_END payloads the AG-UI runtime produces, for every run rather than one caller's. Multiple subscribers are supported, each holding its own SSE connection and filtering client-side on toolCallName or toolCallId. This is how a service that drives /v1/agent/executeRequest over plain HTTP can still observe UI callbacks in real time.

POST /api/ag-ui/run accepts an optional workspacePath (plus inPlace) to bind the built-in *_local tools to an existing directory. That requires ALLOW_INPLACE_WORKSPACE=true, and WORKSPACE_ALLOWED_ROOTS can restrict which paths are bindable. When a workspacePath is bound on a fresh run the folder is scanned exactly as the orchestrator scans it, and its orchestration context plus a lazy-load skill/command/agent catalog become the system prompt — callers hand over the folder instead of inlining AGENTS.md. Project hooks and path/shell policy apply when enabled. read_file_local on an image or PDF attaches multimodal parts to the next turn if the model is vision-capable. Resume patches the persisted run-binding block without re-scanning.

Recommended client sequence

1. Register frontend tools (optional). POST /api/ag-ui/run with a threadId, your frontendTools and empty messages returns a short bind-only stream (RunStarted + RunFinished). Interactive runs use the frontendTools sent with each run, so this only refreshes the legacy global cache behind /v1/agent/getTools.
2. Subscribe to global events (optional). GET /api/ag-ui/events for a fan-out view of tool activity, useful for dashboards.
3. Start an interactive run. POST /api/ag-ui/run with messages, threadId, an optional runId, and the current frontendTools. Keep the SSE stream open for Run/Tool/Text events.
4. Handle tool calls. On TOOL_CALL_START + TOOL_CALL_ARGS, render the component described by the tool parameters schema.
5. Resolve interrupts. If a run ends with RUN_FINISHED.outcome.type="interrupt", collect every open interrupt and resume with POST /api/ag-ui/run carrying resume: [{ "interruptId": "<interrupt.id>", "status": "resolved", "payload": ... }]. Use status: "cancelled" without a payload to cancel. Several entries may be resolved in one request.

Always copy interruptId from each Interrupt.id. Frontend-tool interrupts normally reuse the originating tool_call_id; hook-permission interrupts carry their own permission ID and reference the deferred tool ID in metadata. Legacy state.toolResponse is still accepted but is not the preferred contract.

Include the complete current frontendTools list on every interactive run. If they are absent, the model cannot see them. If tools explicitly names an AG-UI tool that is not present, the request fails with a "Requested tools not found" error.

Execution lifecycle (awaitsResponse)

stateDiagram-v2
    [*] --> Running: execute
    Running --> AwaitingResponse: a tool needs a human
    AwaitingResponse --> Running: resume, on any replica
    Running --> Completed: run finishes
    Running --> Failed: segment fails
    Failed --> Running: execute again
    AwaitingResponse --> AwaitingResponse: claim goes stale, restored for retry
    Completed --> [*]

Loading

A failed segment is recorded against that attempt and leaves the session runnable — only an explicit close makes it terminal. That is why Failed above returns to Running rather than ending the diagram.

When a tool pauses a run, /v1/agent/executeRequest responds with:

{
  "awaitsResponse": true,
  "executionGuid": "2f8e7c1b-...",
  "stateGuid": "0ad6c9f0-...",
  "agui_tool_calls": [
    {
      "tool_call_id": "call_123",
      "tool_name": "Weather-Gate",
      "arguments": {"city": "New York"}
    }
  ],
  "result": null
}

agui_tool_calls[0].tool_call_id is the same ID emitted on the SSE stream. Collect the user's answer, then either post it through AG-UI with resume[], or call POST /v1/agent/resumeRun with {executionGuid, stateGuid, toolCallId, result/error} for a pure API flow. GET /v1/agent/execution/{executionGuid} reports the latest status and the currently-waiting stateGuid. Every execution ends completed or failed, and final responses are persisted, so dashboards can poll without replaying the run.

Error codes

A failed initiate, execute or resume may carry a stable errorCode alongside the human-readable error. Provider failures use QUOTA, RATE_LIMIT, AUTH or UNAVAILABLE; a segment that exhausts its tool budget uses MAX_TOOL_CALLS. Raw provider response bodies are never returned.

Two codes describe a workspace whose root instructions (AGENTS.md / CLAUDE.md) cannot be loaded. They are separate because they need different fixes:

Code	Meaning	Fix
ROOT_INSTRUCTIONS_TOO_LARGE	The file loaded but exceeds the eager budget	Split the file, or raise HARNESS_EAGER_BUDGET_CHARS
ROOT_INSTRUCTIONS_UNREADABLE	The file could not be read at all — permissions, I/O, or a path discovery refuses such as a symlink or a non-regular file	Check the file's permissions and type

Both fail the call rather than silently running against partial or stale context, and initiate and execute report the same code for the same condition.

Request and response shapes

{
  "role": "string (optional)",
  "task": "string (required)",
  "context": "string (optional)",
  "outputInstruction": "string (optional)",
  "tools": ["tool1", "tool2"],
  "model": "gpt-4o",
  "max_tool_calls": 10
}

{
  "success": true,
  "response": "AI response content",
  "error": null,
  "tool_calls_made": 2,
  "total_tokens": 150,
  "partial_response": null,
  "tool_calls_info": [
    {
      "tool_index": 1,
      "tool_name": "google_search_general",
      "llm_tool_interaction_index": 1,
      "mcp_server_id": "general",
      "mcp_server_url": "https://server1.com/mcp"
    }
  ]
}

GET /v1/agent/getTools returns each tool with its attributed name, bracketed server description, input_schema, server_url, server_id, and original_name.

Tool selection

• tools filters MCP and AG-UI tools only. Workspace-backed *_local built-ins are added independently.
• Omitted or null selects all available MCP tools and the run's AG-UI tools.
• An empty list selects no MCP or AG-UI tools.
• A non-empty list selects exactly those names; an unknown name is a bad-request error.
• Use the full attributed name (search_google, not search).
• One model turn may select several tools, including tools from different servers.

Configuration

MCP servers

Two forms are supported, both taking {name, url} objects. The name becomes the suffix on every tool from that server.

# Recommended: JSON array
MCP_SERVER_URLS=[{"name": "general", "url": "https://server1.com/mcp"}, {"name": "google", "url": "https://mcp.google.com/mcp"}]

# Numbered objects, discovered from _1 upwards until the first gap
MCP_SERVER_URL_1={"name": "general", "url": "https://server1.com/mcp"}
MCP_SERVER_URL_2={"name": "google", "url": "https://mcp.google.com/mcp"}

MCP_SERVER_URLS is preferred when both are set, but only if its first element is an object — otherwise parsing falls through to the numbered variables. A blank or missing name becomes default.

The two forms handle bad input differently, so it is worth knowing which one you are using:

Input	Result
Numbered entry that is not an object with both keys	Skipped with a warning; discovery continues at the next index
Numbered index missing	Discovery stops there, so _1 and _3 yields only _1
Array entry missing url	Kept, then rejected at startup: MCP server at index N must be an object with 'name' and 'url' properties
Array mixing objects and bare strings	Fails with AttributeError rather than a configuration error
Array of only bare strings	Ignored — falls through to the numbered variables, or leaves the list empty
Empty array ([])	Ignored the same way, so the numbered variables decide
MCP_SERVER_URL (no suffix)	Never read

Not supported: an array of bare URL strings (MCP_SERVER_URLS=["https://a/mcp"]) and the single-server MCP_SERVER_URL=... variable, both of which earlier documentation described as backward-compatible. With no numbered variables set, either leaves the server list empty and startup fails with At least one MCP server URL must be configured.

Every tool is renamed {original_name}_{server_name} and its description is prefixed [server_name], whether or not the base name collides. Original names are preserved for the call to the server, and execution logs carry the server name and URL. See docs/MULTI_SERVER_IMPLEMENTATION.md.

Core

Variable	Default	Purpose
DATABASE_URL	—	Required. Async SQLAlchemy URL, e.g. postgresql+asyncpg://pyagent:pyagent@localhost:5432/pyagent
LITELLM_BASE_URL	http://localhost:4000	LiteLLM endpoint
LITELLM_API_KEY	—	Required. LiteLLM key. No default: it must match what your gateway expects, and a shipped one would be the same known credential everywhere. The launchers generate one for the bundled stack
LITELLM_REQUEST_TIMEOUT_IN_SEC	300	HTTP idle/network timeout for each LiteLLM request
LITELLM_MODEL_DEADLINE_SEC	240	Absolute wall-clock deadline for one complete LiteLLM call, including streamed response consumption
LITELLM_MAX_COMPLETION_TOKENS	0 (off)	Optional operator resource guard; when positive, sends a completion-token cap and accepts possible truncation
LITELLM_DROP_PARAMS	True	Drop parameters a model does not accept
MAX_TOOL_CALLS	10	Tool calls per request
ServiceName	jb-ai-orchestrator-service	Service name
Environment	DEV	Environment name
Region	USC1	Region
SwaggerBasePath	empty	Base path when served behind a gateway
DOCS_ENABLED	true	Serves /swagger and /openapi.json. false withdraws both, leaving no route that describes the API
CORS_ALLOWED_ORIGINS	empty	Comma-separated origins allowed to call the API from a browser. Empty sends no CORS headers, so cross-origin calls are refused. * is accepted, but startup fails if combined with credentials
CORS_ALLOW_CREDENTIALS	false	Whether cross-origin requests may carry cookies and authorization headers
General_LogFolder	./Logs	Log directory
Logging_LogLevel_Default	Information	Log level

BIND_ADDR is not read by the service: it belongs to the Compose layer in ./.env and decides which host interface each published port binds to. It defaults to 127.0.0.1 — see Security model.

Harness

Variable	Default	Purpose
HARNESS_EAGER_BUDGET_CHARS	24000	Total characters of eagerly-injected context. Root AGENTS.md/CLAUDE.md are one mandatory allocation of this budget and fail explicitly when they exceed it, rather than being silently truncated; raise this instead of splitting a large playbook. Optional rules take what remains, whole or not at all. A non-numeric or non-positive value is logged and ignored.

Lifecycle

Variable	Default	Purpose
RESUME_CLAIM_TIMEOUT_SEC	300	Age at which a stale in-progress resume claim is restored for retry
RUN_HEARTBEAT_INTERVAL_SEC	5	Heartbeat interval for active runs; must be ≥ 1
RUN_HEARTBEAT_STALE_SEC	300	Runs without a newer heartbeat are stale and reconciled before close or a replacement claim; must exceed the interval
RUN_SEGMENT_DEADLINE_SEC	270	Absolute wall-clock deadline for workspace/binding preparation plus the complete model/tool loop
RUN_CANCELLATION_WARN_SEC	5	Emit a structured diagnostic if a cancelled worker has not stopped after this many seconds
CLOSE_WAIT_TIMEOUT_SEC	10	Wait before close returns 202 closing instead of 200 closed

The default ordering is model call 240s, complete segment 270s, caller 300s, and reverse proxy 310s or more. A deadline cancels and awaits the worker before its execution_runs claim becomes terminal. A new claim may reconcile a crash-orphaned row only after its heartbeat is stale; a fresh row still returns RUN_CONFLICT.

The service is a generic execution engine: it enforces containment, isolation, bounded execution, safe claim release, and retryable session state. It does not set workflow policy for model output or behavior. Failed attempt records remain diagnostic history, while the execution, pending interaction, or conversation returns to its real runnable state unless the caller explicitly closed it.

Workspaces and bindings

Variable	Default	Purpose
WORKSPACES_ROOT	<tmp>/jb-agent-workspaces	Root for per-execution sandboxes
ALLOW_INPLACE_WORKSPACE	false	Allow inPlace=true to run on a local path without copying
WORKSPACE_ALLOWED_ROOTS	empty	Absolute roots bindable in place or via AG-UI workspacePath
GIT_CLONE_TIMEOUT_SEC	120	Timeout for git clone and archive downloads
MAX_WORKSPACE_MB	0	Soft cap on provisioned workspace size; 0 disables
OUTPUT_BINDINGS_ENABLED	false	Allow non-null output bindings and durable output tools
OUTPUT_READ_MAX_BYTES	5242880	Durable output read cap
OUTPUT_WRITE_MAX_BYTES	5242880	Durable output write cap
OUTPUT_LIST_MAX_ENTRIES	1000	Durable output listing page cap
AZURE_BLOB_TIMEOUT_SEC	60	Per-operation Azure Blob timeout
PROMPTS_DIR	empty	Override directory for YAML prompt templates

Tools

Variable	Default	Purpose
LOCAL_TOOLS_ENABLED	true	Expose the built-in local coding tools
LOCAL_TOOLS_NAMESPACE	local	Suffix marking local tools, e.g. read_file_local
FILTER_MCP_TOOLS_CONFLICTING_WITH_LOCAL	true	Drop remote MCP tools whose base name collides with a built-in
LOCAL_SHELL_ENABLED	false	Expose execute_local. Off by default: the service authenticates no one, so this grants a shell to anyone who can reach the API
LOCAL_SHELL_TIMEOUT_SEC	60	Default shell timeout
LOCAL_SHELL_MAX_OUTPUT_BYTES	100000	Captured bytes per output stream
SUBAGENT_ENABLED	true	Expose the task_local subagent tool
SUBAGENT_MAX_TOOL_CALLS	8	Tool calls inside one subagent
SUBAGENT_MAX_DEPTH	2	Maximum task_local nesting depth

Context management

Variable	Default	Purpose
CONTEXT_COMPACTION_ENABLED	true	Offload oversized tool results and compact older tool messages
TOOL_RESULT_OFFLOAD_CHARS	8000	Offload a tool result above this JSON size
CONTEXT_COMPACTION_CHARS	120000	Compact when total message characters exceed this
CONTEXT_COMPACTION_TOKENS	0	When above 0, use a token budget instead of characters
CONTEXT_COMPACTION_KEEP_RECENT_TOOL_MSGS	8	Recent tool messages kept intact
CONTEXT_COMPACTION_KEEP_RECENT_TURNS	4	Recent turns kept during summarization
CONTEXT_SUMMARIZATION_ENABLED	true	Summarize older turns before tool compaction
CONTEXT_SUMMARIZATION_MODEL	run model	Model used for summarization
LLM_STREAMING_ENABLED	true	Stream tokens to AG-UI when a UI channel exists

Hooks, policy and multimodal

Variable	Default	Purpose
HOOKS_ENABLED	false	Run project Cursor/Claude hooks. Off by default: hooks are shell commands supplied by the bound workspace
HOOKS_FAIL_CLOSED	false	Block the action when a hook errors or times out
HOOKS_TIMEOUT_SEC	30	Per-hook subprocess timeout
HOOKS_MAX_OUTPUT_BYTES	100000	Cap on hook output
PATH_POLICY_ENABLED	true	Enforce path and shell allow/deny lists
PATH_DENYLIST / PATH_ALLOWLIST	empty	Comma-separated globs under the workspace
SHELL_COMMAND_DENYLIST / SHELL_COMMAND_ALLOWLIST	empty	Regex filters for execute_local
MULTIMODAL_ENABLED	true	Attach images and PDFs read by read_file_local
MULTIMODAL_MAX_BYTES	5242880	Maximum attachment size
MULTIMODAL_MODEL_ALLOWLIST	empty	When set, only matching model ids receive attachments
MULTIMODAL_MODEL_DENYLIST	gpt-3.5-turbo, embeddings, …	Model ids that never receive attachments
MULTIMODAL_VISION_MARKERS	gpt-4o, Claude, Gemini, …	Vision-capable markers used when no allowlist is set

Hook configuration keeps Cursor-first precedence, followed by Claude hooks.json and settings.json. Files are opened within the workspace without following symlinks and are limited to 64,000 characters. Unsafe, malformed, excessively nested, or oversized hook configuration is ignored and no hooks from it execute; discovery reports the reason in manifest notes. Primitive discovery also refuses symlinks and reports one manifest note with the number of unique symlinked workspace paths skipped during that collection.

Full orchestrator and harness knobs are documented in docs/ORCHESTRATOR.md §11, and every key above appears in src/.env/localhost/.env.example.

Workspaces, input and durable output

A workspace is the input playbook or working copy. It is distinct from optional durable output and from service-owned .agent runtime files. Two entry points create one:

Endpoint	Field	What you pass	How the workspace is created
POST /api/ag-ui/run	workspacePath (+ inPlace)	an existing container path only, no git or URL	In place — binds that exact directory, no copy
POST /v1/orchestrator/initiate	folder (+ inPlace)	a git URL, an archive URL (.zip/.tar.gz/.tgz/.tar over HTTP(S)), a file:// or UNC folder, or a local path	Provisioned per the source kind below

initiate source kinds (workspace_manager.classify_source):

Source (folder)	Handling	Workspace location	Output persisted to host?
Local path + inPlace: true	Runs in place, no copy. Needs ALLOW_INPLACE_WORKSPACE=true and a path under WORKSPACE_ALLOWED_ROOTS	your folder itself	Yes — written straight back
Local path + inPlace: false	Copied into a sandbox	WORKSPACES_ROOT/{executionId}/workspace	No host target — kept in service storage until operator cleanup
Git URL (*.git, git@…, github/gitlab/…)	git clone --depth 1; an optional input.branch adds --branch --single-branch	WORKSPACES_ROOT/{executionId}/workspace	No host target
Archive / shared folder (.zip, .tar.gz, .tgz, .tar, file://, UNC \\host\share)	Downloaded or copied, then extracted	WORKSPACES_ROOT/{executionId}/workspace	No host target

There is no object-store input provider: nothing lists a bucket or container. A remote source must be a single archive fetched over HTTP(S), so a blob or S3 URL only works when it addresses the .zip/.tar.gz object itself — either pre-signed, or with credentials.inputAccessToken, which is sent as an Authorization header (Bearer is added unless the value already carries a scheme). Object storage appears on the output side only, where the providers are shared_folder and azure_blob.

With OUTPUT_BINDINGS_ENABLED=true, callers may bind shared_folder or azure_blob output, and the model gains write_output_local, edit_output_local, read_output_local and list_output_local. Targeted edits use exact string replacement and fail on missing or ambiguous text rather than silently rewriting the file. For Azure, credentials.outputAccessToken is a SAS supplied fresh with each execute or resume; it must stay valid for that segment and is never persisted.

New-contract sessions default to workflow, which is observation-only for input even when output is null. Use working_copy for writable input, or bind durable output for workflow state. Output paths are logical relative paths, and .agent/** is service runtime that cannot be written as output. Input, runtime and output roots stay separate; output is never removed by session lifecycle. Use the /close endpoints for teardown — /abandon only discards holds. AG-UI does not enable git or URL input.

Sharing a folder between host and container

To let in-place runs read and write a folder on your machine, bind-mount it at /workspaces. docker-compose.yml is already wired for this through WORKSPACES_HOST_DIR:

volumes:
  - ${WORKSPACES_HOST_DIR:-./workspaces}:/workspaces

Allow in-place binding and restrict it to that mount in src/.env/docker/.env:

ALLOW_INPLACE_WORKSPACE=true
WORKSPACE_ALLOWED_ROOTS=/workspaces
OUTPUT_BINDINGS_ENABLED=false

Then pass the host path when bringing the stack up:

# macOS / Linux
WORKSPACES_HOST_DIR=/Users/you/workflows docker compose up --build -d
# Windows PowerShell
$env:WORKSPACES_HOST_DIR="c:/git/workflows"; docker compose up --build -d

Callers now reference a workflow by absolute container path under the mount — workspacePath: "/workspaces/<folder>" on /api/ag-ui/run, or folder: "/workspaces/<folder>" with inPlace: true on /v1/orchestrator/initiate. The path is literal, so every caller sharing the mount must map the same host folder to the same container path. Durable output uses a separate shared_folder binding under a root listed in WORKSPACE_ALLOWED_ROOTS; close never deletes caller-owned input or output, so the caller owns output retention.

Deployments that send non-null output bindings must opt into OUTPUT_BINDINGS_ENABLED=true in their own compose or env, and then WORKSPACE_ALLOWED_ROOTS must cover every workflow and output mount. Local shared_folder output needs no credentials.

Private git input (planned). Fetching a workflow from a git URL is the service's job: as a deploy-once orchestrator serving many teams it would hold its own service-level git credential rather than accepting tokens per request. Today initiate clones public git URLs; private-repo auth and a durable output target for cloned input are planned.

Caller examples

ag-ui-demo/ is a protocol sandbox, not production web code. It demonstrates env-driven bindings on fresh AG-UI runs and thread close with bounded 202 retry; see ag-ui-demo/README.md. Production integration belongs in the calling application; this service stays a generic orchestrator.

An end-to-end web application built on this orchestrator will be released soon: jb-web-code.

Running many instances

Interactive state is durable, so the service scales horizontally without sticky sessions:

• Frontend tools are per run. The agent loop uses the frontend_tools passed with each request, so concurrent sessions and separate applications sharing one deployment never overwrite each other. The process-global cache exists only for the getTools listing and legacy bind-only runs. Tool definitions needed for resume are persisted in LLMState.state_payload.
• Nothing waits in memory. When a tool awaits a user response — an AG-UI tool or the built-in ask_user — the loop persists to LLMState and returns. Any instance serves the follow-up through /api/ag-ui/run (resume[]), /v1/orchestrator/resume, or /v1/agent/resumeRun. AG-UI locates the hold by interrupt or tool ID; orchestrator and direct-agent resumes load the supplied stateGuid and claim it atomically.
• The only per-pod resource is the SSE socket, which needs no stickiness because resume arrives as a fresh request.

GET /api/ag-ui/events subscribers remain per-process queues. Correctness does not depend on them; cross-pod dashboard fan-out would need shared pub/sub.

Development

Install the runtime requirements plus pytest and pytest-asyncio, then run the suite from the repository root:

pytest

pytest.ini sets testpaths = tests and asyncio_mode = auto, so async tests need no marker. Tests marked live require a separately running service and are excluded by default. The suite needs no database and no network.

scripts/ holds checks that need a running service, so they are not part of the suite:

python scripts/smoke_service.py        # agent endpoints and tool attribution
python scripts/smoke_agui.py           # AG-UI bind, stream, interrupt and resume
python scripts/orchestrator_continuity_exercise.py --help

Set PY_MAIN_AGENT_BASE_URL to target something other than http://localhost:8000, and PY_MAIN_AGENT_SMOKE_MODEL to change the model (default gpt-4o). Both smoke checks exit non-zero on the first failure. Because tool names depend on which MCP servers you configured, smoke_service.py takes them from /getTools unless you set PY_MAIN_AGENT_SMOKE_TOOLS, and smoke_agui.py exercises only its own UI tools unless you name an MCP tool in PY_MAIN_AGENT_SMOKE_MCP_TOOL.

orchestrator_continuity_exercise.py is dry-run by default; use --run --simulate-execute when no LLM is configured, and pass --host-shared-root (or set WORKSPACES_HOST_DIR) for live Docker runs.

scripts/smoke_agui.py and ag-ui-demo/ demonstrate compatibility paths. Do not treat them as references for canonical resume[], cancellation, multi-interrupt, thread-abandon or typed-history behaviour — those contracts are covered by tests/test_agui_controller_resume.py and tests/test_agui_batch_resume.py.

The service depends on FastAPI and uvicorn, the mcp client, SQLAlchemy with asyncpg, Alembic, ag-ui-protocol, and azure-storage-blob for the optional blob output provider. It talks to LiteLLM over HTTP with httpx rather than importing the LiteLLM SDK. src/requirements.txt is the authoritative list.

Limitations and open work

• Serial tool loops. Independent reads are not always batched, so a workflow can need many model round trips. Batch or programmatic tool execution would cut latency and context churn.
• Context thresholds. Token-aware compaction exists, but character thresholds remain the default. A model-window fraction trigger would be safer across providers.
• Offload lifecycle. .agent/offload/ preserves oversized results; fresh AG-UI runs prune unreferenced files when discarding stale awaits, and thread close removes runtime. Durable output is never session-cleaned.
• Prompt caching. Static harness sections are not explicitly segmented for provider prompt-cache reuse.
• Diagnostics. There is no /context-style endpoint exposing the loaded root instructions, catalogs, tools and context budget.
• Harness coverage. Only Cursor and Claude Code have dedicated adapters. Anything else falls back to the generic one, which reads AGENTS.md (or a README as a last resort), so a repository organised for another tool — GitHub Copilot's .github/copilot-instructions.md, for instance — contributes nothing beyond that. The adapter contract is deliberately small, so these are additions rather than redesigns.
• MCP scopes. Named multi-server configuration has no local/project/user precedence or managed policy layer.
• Event fan-out. /api/ag-ui/events subscribers are process-local.
• Pending-state visibility. There are no metrics for rows stuck in awaiting_response, which would surface abandoned interactions.

Documentation

• docs/ORCHESTRATOR.md — source of truth for the harness: workspace and instruction loading, prompt composition, the tool loop, compaction, resume, lifecycle and close, persistence, and roadmap
• docs/MULTI_SERVER_IMPLEMENTATION.md — multiple MCP servers and universal tool attribution
• docs/DOCUMENTATION_INDEX.md — navigation
• ag-ui-demo/README.md — protocol sandbox caller
• SECURITY.md — threat model, hardening checklist, and how to report a vulnerability privately
• CONTRIBUTING.md — what belongs in the engine, the branch and review flow, and the local test recipe
• CODE_OF_CONDUCT.md — community expectations
• changes/ — what shipped, per change

License

First-party code in this repository is licensed under the Apache License, Version 2.0. See LICENSE and NOTICE. Third-party material remains subject to its own license and notices.