Skip to content

test-deployment.md

Latest commit

 

History

History
170 lines (136 loc) · 8.04 KB

test-deployment.md

File metadata and controls

170 lines (136 loc) · 8.04 KB
Raw

Test deployment — Docker/Postgres stack

Stand up the full containerized stack (Postgres + server + ad-hoc Docker workers) in isolation and validate the end-to-end run → worker container → git checkout → PR → live stream loop that the unit tests can't cover. Nothing here touches the prod systemd services or the prod SQLite DB.

Everything runs as a separate compose project (task-orch-test) with its own network and throwaway volumes on a non-conflicting port (3005).

Validated 2026-07-03 against nodetool-ai/nodetool: a worker container spawned, cloned from the repo-cache mirror, branched off main, wrote + committed DEPLOY_TEST.md, pushed, and opened a PR — with events/messages streaming into Postgres and the container exiting --rm. The test PR was closed and its branch deleted immediately after. The worker runs as the non-root node user (uid 1000): the Claude Code CLI refuses --dangerously-skip-permissions as root, and uid 1000 matches the host user that owns the mounted ~/.claude.

Prerequisites

  • Docker + docker compose v2 on the host.
  • A throwaway GitHub repo to open test PRs against (e.g. you/orch-sandbox) with at least one commit on its default branch.
  • A GitHub token with repo scope on that repo (gh auth token, or a PAT).
  • ~/.claude authenticated on the host (Claude Code login), so workers inherit the session for the agent turn.

1. Configure

cp .env.test.example .env.test
# edit .env.test:
#   AUTH_SECRET   = $(openssl rand -base64 32)
#   GH_TOKEN      = <scoped token>
#   TARGET_REPOS  = you/orch-sandbox          # space-separated owner/repo
#   CLAUDE_HOME   = /home/claude/.claude
#   SERVER_PORT   = 3005                        # keep non-conflicting

2. Bring up the stack

scripts/test-deploy.sh up

This builds both images, starts postgres + server, mirrors TARGET_REPOS into the repo-cache volume, and waits for health. On success it prints http://localhost:3005. The server runs initDb() on boot — migrations + seed (R-default, personas, the run_stream trigger) apply to the fresh Postgres.

Sanity checks:

scripts/test-deploy.sh status            # services + any live worker containers
scripts/test-deploy.sh psql "\dt"        # tables exist
scripts/test-deploy.sh psql "SELECT id FROM repositories;"   # R-default seeded

3. Point the default repo at your sandbox + seed a task

The CLI talks to the DB directly and is run inside the server container (which has DATABASE_URL, the source, and docker.sock). Set R-default's remote to your sandbox so runs clone it, then create a plan + task:

DC="docker compose -p task-orch-test --env-file .env.test"

# point the default repo at the sandbox (matches TARGET_REPOS mirror name)
scripts/test-deploy.sh psql \
  "UPDATE repositories SET remote='git@github.com:you/orch-sandbox.git' WHERE id='R-default';"

# seed a plan + a trivial task
$DC exec server npx tsx cli.ts new plan --title="Test deploy $(date -u +%H%M)"
$DC exec server npx tsx cli.ts new task --plan=<PLAN_ID> \
  --title="Add a HELLO.md" \
  --body="Create a file HELLO.md containing 'hello from a worker container', commit it, and open a PR."

4. Acceptance test — dispatch a run through the containerized server

Create the run's row, then await dispatchRun so the docker run of the worker container completes before the process exits. This exercises the real path: runs.create → dispatchRun → docker run a worker container via the mounted socket → in-container checkout → push/PR.

Do not use cli.ts agent <TASK_ID> here. In the detached/containerized model, runs.create fires the dispatch in a background void (async …)() IIFE and the CLI's tailSession waits on the in-process event bus — which a separate worker container never feeds (it writes to Postgres). So the CLI tail hangs and never reports completion. Await dispatchRun (below) or drive it through the HTTP API and watch via the DB / the run-view SSE.

First create the run row (deferred — no in-process kickoff), then dispatch it:

# Create an <implement> run for the task WITHOUT the in-process kickoff, capturing
# its id, then await dispatchRun (the worker container spawns on the mounted socket).
$DC exec -T server npx tsx -e '
import "./lib/runs";
import * as runs from "./lib/runs";
import { dispatchRun } from "./lib/run-dispatch";
const task = process.argv[2];
const run = await runs.create({
  goal: "<implement>", cwdStrategy: "worktree",
  toolsProfile: "orchestrator,repo_write,gh_pr,gh_ci",
  taskId: task, baseBranch: "main", defer: true,
});
const r = await dispatchRun(run.id);
console.log(`run #${run.id} -> ${r}`);
process.exit(0);
' <TASK_ID>

Then watch it work:

# a run-<id>-<nonce> worker container should appear:
watch -n1 'docker ps --filter name=run- --format "{{.Names}}\t{{.Status}}"'

# events/messages streaming into Postgres (proves the worker->PG path):
scripts/test-deploy.sh psql \
  "SELECT id, status, branch, pr_url FROM agent_runs ORDER BY id DESC LIMIT 1;"
scripts/test-deploy.sh psql \
  "SELECT count(*) FROM agent_messages; SELECT count(*) FROM agent_events;"

# worker container's own logs (if it fails, the error is here — note the agent's
# turn output goes to Postgres, not stdout; stdout only shows the tsx bootstrap):
docker logs <run-container-name>

5. Pass criteria

  • A run-<id>-* worker container starts (and exits --rm when done).
  • The run goes preparing → running and agent_messages/agent_events grow.
  • A branch + PR appear on the sandbox repo (gh pr list -R you/orch-sandbox).
  • The run view streams live: open http://localhost:3005/runs/<id> (needs a signed-in user — create one with $DC exec server npx tsx cli.ts user add you@x.com --password=...) and confirm messages appear without polling.
  • docker compose restart server mid-run → the worker container keeps running and the run completes (restart-survival).
  • Cancel from the UI → the worker aborts within ~one heartbeat (20s) and/or the container is stopped.

6. Teardown

scripts/test-deploy.sh down      # stops everything + removes the pg + repo-cache volumes

Troubleshooting

Symptom Check
Run wedged in preparing, no worker container Server can reach docker.sock: $DC exec server ls -l /var/run/docker.sock; worker image exists: docker images task-orchestrator-worker.
Worker container exits immediately docker logs <run-container> — usually a missing env (DATABASE_URL/GH_TOKEN) or the repo-cache mirror absent.
Clone/PR fails with auth error GH_TOKEN lacks repo scope on the sandbox, or the credential helper env didn't reach the worker.
containerCheckout "no GitHub remote" The run's repository row has no remote (step 3).
No live stream in the run view run_stream trigger present (scripts/test-deploy.sh psql "SELECT tgname FROM pg_trigger WHERE tgname='agent_events_notify';") and the listener connected.
Agent turn fails with no API key ~/.claude not authenticated on the host, or CLAUDE_CODE_OAUTH_TOKEN/ANTHROPIC_API_KEY unset.
Worker exits 1, log shows --dangerously-skip-permissions cannot be used with root/sudo privileges The worker image must run as the non-root node user (USER node in Dockerfile.worker); the SDK's bypassPermissions maps to that flag, which Claude Code refuses as root.
Worker can't read the repo-cache mirror The mirror's git objects must be world-readable (repo-cache-init's alpine/git writes them 444/755); the non-root node user (uid 1000) reads them directly.

Notes / known gaps (see PR #57)

  • followUp / CI-autofix still runs the host worktree path — not exercised here.
  • docker.sock is mounted raw; put a least-privilege socket proxy in front before any non-local exposure.
  • This is a test deploy. A prod cutover additionally needs deploy.sh rewritten for docker compose up, the ETL run into the prod Postgres, and repository remotes configured.