From 26b2b839830f7c07f2ac3820da36dc1b76a78947 Mon Sep 17 00:00:00 2001 From: ArkNill <48707894+ArkNill@users.noreply.github.com> Date: Wed, 20 May 2026 18:02:49 +0900 Subject: [PATCH] docs(onboarding): add AGENT_SETUP.md playbook (Path B final piece) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third and final slice of Path B (LLM-driven onboarding). With env-fingerprint (PR #19) describing state and verify (PR #20) asserting expectations, this document sequences them into an end-to-end install flow an agent can follow without scraping human-friendly output. Audience: AI coding agents (Claude Code / Codex / Gemini) running an llm-relay setup on a user's behalf. The document is explicit that it is NOT a human tutorial -- humans should use README.md and `llm-relay init` directly. Structure: Phase 0 — Probe (env-fingerprint) Phase 1 — Install or upgrade the package (with extras decision table) Phase 2 — Initialize local state (llm-relay init) Phase 3 — Per-CLI integration (claude-code / openai-codex / gemini-cli) Phase 4 — Optional: start the server (Linux/macOS vs Windows service) Phase 5 — Final acceptance (verify all) Each phase ends with a `verify` call, and the playbook spells out how to respond to pass / warn / fail outcomes. Permission protocol: The playbook never assumes consent for system-modifying actions. Five explicit markers gate any operation that touches outside of read-only probes: [PERMISSION: install-package] pip install [PERMISSION: write-config] llm-relay init (writes ~/.llm-relay/, edits ~/.claude/settings.json) [PERMISSION: edit-claude-settings] direct settings.json edit (last resort) [PERMISSION: fix-permissions] chmod / chown on home dir files [PERMISSION: install-service] Windows background service registration "When to stop and ask" section enumerates ambiguous states (no CLI installed, verify fail with no remediation, multiple Python interpreters, etc.) where the agent must defer to the user instead of improvising. "What not to do" section pins down the agent's scope: do not modify session transcripts, do not install LLM CLIs on the user's behalf, do not auto-edit shell rc files, do not retry a failing remediation more than once. README updated with an "Agent-driven setup" section linking to the playbook and showing the four entry-point commands. CHANGELOG documents the playbook under Unreleased ### Added. Schema contract: both env-fingerprint and verify use schema_version "1"; the playbook calls out that an agent should fall back to the in-tree playbook for whatever release it has installed if the schema version differs. --- CHANGELOG.md | 1 + README.md | 18 +++ docs/AGENT_SETUP.md | 326 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 345 insertions(+) create mode 100644 docs/AGENT_SETUP.md diff --git a/CHANGELOG.md b/CHANGELOG.md index de4b781..9f25fe5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ All notable changes to llm-relay are documented here. ## [Unreleased] ### Added +- **Agent-driven setup playbook** (`docs/AGENT_SETUP.md`): structured onboarding document targeted at AI coding agents (Claude Code / Codex / Gemini) automating an llm-relay install on a user's behalf. Sequences `env-fingerprint` + `verify` into a five-phase flow (probe → install → init → per-CLI integration → final acceptance) with explicit `[PERMISSION: …]` markers wherever the agent must pause for user consent, and explicit "stop and ask" rules for ambiguous states. README section "Agent-driven setup" links to it. - **`llm-relay verify` command group** (`verify/`): four idempotent verification subcommands designed for agent-driven onboarding. `verify install` confirms the package itself (Python version, importability, entry points, optional extras, version consistency). `verify config` confirms local state (db dir, schema, writability, config file, knowledge dir, port availability, no deprecated env vars). `verify integration --cli {claude-code,openai-codex,gemini-cli,all}` confirms each CLI is wired through the relay (binary on PATH, settings file present, `ANTHROPIC_BASE_URL` routing to localhost, MCP server registered). `verify all` aggregates install + config + integration. Shared output schema (`schema_version: "1"`) emits per-check status (`pass`/`fail`/`warn`/`skipped`) with optional remediation strings; overall priority is `fail > warn > pass`. Skipped checks (e.g. CLI not installed) never escalate to fail. Optional `--live` probes `/_health` on the proxy. Common options: `--format {text,json}`, `--quiet`, `--no-remediation`. Exit code is 0 on pass/warn, 1 on fail. - **`llm-relay env-fingerprint` command** (`env_fingerprint.py`): single-shot, idempotent snapshot of the local LLM CLI environment for agent-driven onboarding. Emits a structured JSON (or YAML) document with sections for `llm_relay` package state, per-CLI install/auth/version, port availability, filesystem layout, redacted environment variables (API keys reported as `set`/`empty`/`null` only), and an optional `doctor` summary. Designed so an agent (Claude Code / Codex / Gemini) automating an llm-relay install can parse the environment instead of scraping `init` output. Schema versioned (`schema_version: "1"`); sub-probe failures surface as `_error` markers without crashing the snapshot. Options: `--format {json,yaml}`, `--no-doctor`, `--ports`. diff --git a/README.md b/README.md index 812fa4d..80559ee 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,24 @@ llm-relay-mcp # stdio transport, 8 tools llm-relay connect # Auto-configures Claude Code proxy ``` +### Agent-driven setup + +If you would rather have your existing coding agent (Claude Code, Codex, +Gemini) run the install for you, point it at +[`docs/AGENT_SETUP.md`](docs/AGENT_SETUP.md). It is a structured playbook +the agent follows step by step, using `llm-relay env-fingerprint` and +`llm-relay verify` to probe and check each step without scraping output. + +```bash +llm-relay env-fingerprint --format json # state snapshot +llm-relay verify install --format json # is the package usable? +llm-relay verify config --format json # is local state set up? +llm-relay verify integration --cli claude-code # is the CLI wired? +llm-relay verify all # everything at once +``` + +Exit code is 0 on pass/warn, 1 on fail. + ## CLI Status | CLI | Status | diff --git a/docs/AGENT_SETUP.md b/docs/AGENT_SETUP.md new file mode 100644 index 0000000..c2dc07b --- /dev/null +++ b/docs/AGENT_SETUP.md @@ -0,0 +1,326 @@ +# Agent Setup Playbook + +> **Audience**: an AI coding agent (Claude Code, OpenAI Codex, Gemini CLI) +> automating an llm-relay install on behalf of a human user. +> **This document is not a tutorial for humans** — it is a sequence of +> structured probes and actions an agent can follow without asking the +> user for technical details. + +If you are a human, you probably want [`README.md`](../README.md) and the +`llm-relay init` command instead. + +--- + +## What this playbook is + +llm-relay ships with two agent-consumable primitives: + +- **`llm-relay env-fingerprint --format json`** — describes the *state* of + the user's environment (which CLIs are installed, what ports are free, + what env vars are set, etc.). Read-only. +- **`llm-relay verify {install|config|integration|all} --format json`** — + asserts *expectations* about that state, returning pass/fail/warn/skipped + per check with optional remediation strings. Read-only. + +This playbook sequences those two primitives — plus existing commands like +`llm-relay init` — into an end-to-end install flow. + +You do not need to scrape human-friendly output. Every step in this +playbook either: + +1. Calls a command that emits structured JSON. +2. Performs a single, idempotent filesystem / shell action. +3. Pauses for explicit user permission via a clearly labelled marker. + +## What it is not + +- It is not a substitute for the user's permission. Markers like + **`[PERMISSION: install-package]`** below indicate steps where the user + must approve a destructive or system-modifying action before you proceed. +- It is not a place to make undocumented decisions. If `env-fingerprint` + reveals state this playbook does not branch on, **stop and ask the + user** rather than guessing. + +--- + +## Schema contract + +Both primitives use `"schema_version": "1"`. If you see a different +`schema_version`, this playbook may be out of date for your llm-relay +release — fall back to the in-tree +[`docs/AGENT_SETUP.md`](AGENT_SETUP.md) from that release. + +`verify` exit codes: +- `0` — overall is `pass` or `warn`. Safe to proceed. +- `1` — overall is `fail`. Do not proceed past the failing step. + +--- + +## Phase 0 — Probe + +Goal: understand what the user already has so later phases skip the work +that is already done. + +```bash +llm-relay env-fingerprint --format json --no-doctor +``` + +Parse the result. Treat `_error` markers on any section as "section +unavailable" — do not abort the whole flow; just proceed with the +remaining sections. + +Key fields you will need later: + +| field | used in | +|---|---| +| `llm_relay.version` | Phase 1 — decide whether to install | +| `llm_relay.db_dir` | Phase 3 — where init writes | +| `clis[*].id`, `clis[*].installed`, `clis[*].auth.preferred` | Phase 5 — per-CLI wiring | +| `ports["8083"]` | Phase 3 — choose `--port` for init | +| `env.ANTHROPIC_BASE_URL` | Phase 5 — already-wired Claude Code | + +If `clis[*].installed` is `false` for **every** CLI: stop and ask the user +to install at least one LLM CLI first — llm-relay has no purpose without +one. Do **not** install CLIs on the user's behalf in this playbook. + +--- + +## Phase 1 — Install or upgrade the package + +### Decision + +- If `llm_relay.version` is `null` → llm-relay is not installed yet. +- If `llm_relay.version` is older than the version you are setting up + against → upgrade. +- Otherwise → skip to Phase 2. + +### Which extras to install + +Decide from the fingerprint: + +| Condition | Install | +|---|---| +| Always | `llm-relay` (base) | +| Any CLI installed | `llm-relay[proxy]` (the dashboard / proxy needs httpx + uvicorn) | +| `clis[*].id == "claude-code"` and `installed` | `llm-relay[mcp]` (Claude Code can register the MCP server) | +| User says they want PostgreSQL | `llm-relay[pg]` | +| Otherwise | `llm-relay[all]` is a safe default | + +### Action + +**`[PERMISSION: install-package]`** — installing into the user's Python +environment modifies their site-packages. Ask before running: + +```bash +pip install llm-relay[] +``` + +If the user uses a virtual environment that you can detect (e.g. they ran +the agent from inside an activated venv), install there; otherwise default +to `pip install --user` and tell them which Python interpreter received the +install. + +### Verify + +```bash +llm-relay verify install --format json +``` + +- `overall == "pass"`: proceed. +- `overall == "warn"`: read the `warn` entries — typically + `entry_point_mcp` or `proxy_extras` is missing because the user opted + for a slim install. Acceptable unless the user explicitly asked for + those features. +- `overall == "fail"`: read the failing check's `remediation`, attempt the + remediation **once**, then re-verify. If it still fails, stop and report + to the user. + +--- + +## Phase 2 — Initialize local state + +### Decision + +- If `verify config` already returns `overall == "pass"` (or `warn` for + only the optional `knowledge_dir` / `config_file` items) → skip to + Phase 4. +- Otherwise → init. + +### Action + +**`[PERMISSION: write-config]`** — `llm-relay init` writes to +`~/.llm-relay/` and edits `~/.claude/settings.json` (if Claude Code is +installed). Ask before running. + +Pick `--port`: +- If `ports["8083"]` is `"free"` → default `--port 8083`. +- If it is `"in_use"` → either the relay is already running on it (check + by hitting `/_health` — Phase 5 covers this) or another process owns it. + Ask the user before overriding; do not silently move to a new port. + +```bash +llm-relay init --port +``` + +### Verify + +```bash +llm-relay verify config --port --format json +``` + +`fail` here usually means the install partially completed. Read each +failing check's `remediation`. The most common cause is a permissions +issue on `~/.llm-relay/` — do not chmod files on the user's behalf +without **`[PERMISSION: fix-permissions]`**. + +--- + +## Phase 3 — Per-CLI integration + +For every CLI where `env-fingerprint` reported `installed: true`, run: + +```bash +llm-relay verify integration --cli --format json +``` + +Then act based on the failing checks below. **Do not run integration +steps for CLIs the user does not have installed.** + +### Claude Code (`claude-code`) + +| Failing check | Action | +|---|---| +| `binary` | Cannot happen if `env-fingerprint` said `installed: true`; if it does, treat as a transient PATH issue and re-probe. | +| `settings_present` | Have the user open Claude Code once; the binary creates `~/.claude/settings.json` on first run. | +| `proxy_route` | `llm-relay init` should have set this. If `init` already ran and this is still `fail`, inspect `~/.claude/settings.json` `env.ANTHROPIC_BASE_URL` — fix to `http://localhost:` only with **`[PERMISSION: edit-claude-settings]`**. | +| `mcp_server` | Same as above — `init` writes `mcpServers["llm-relay"]`. If missing, re-run `init`; do not edit the JSON directly unless `init` itself fails. | + +### OpenAI Codex (`openai-codex`) + +The `proxy_route` check is `skipped` by design — Codex does not currently +expose a stable routing knob. Do **not** attempt to monkey-patch Codex's +config to route through llm-relay; that is upstream work, not yours. + +`binary` and `config_present` should both pass after the user has run +`codex` at least once. + +### Gemini CLI (`gemini-cli`) + +`oauth_known_issue` is always `warn` and surfaces upstream +[google-gemini/gemini-cli#25425](https://github.com/google-gemini/gemini-cli/issues/25425). +If the user reports a 403 from Gemini, tell them to set +`GEMINI_API_KEY` instead of relying on oauth-personal. + +--- + +## Phase 4 — Optional: start the server + +Only if the user explicitly asked for the dashboard or a background +service. Otherwise leave it for them to start with `llm-relay serve`. + +**Linux / macOS (manual):** + +```bash +llm-relay serve --port +``` + +**Windows (background service):** + +**`[PERMISSION: install-service]`** — this registers an auto-start entry +in the user's Startup Folder. Ask before running. + +```bash +llm-relay service install --port +llm-relay service start --port +``` + +### Verify + +After the server is running: + +```bash +llm-relay verify integration --cli all --live --port --format json +``` + +The `--live` flag adds a `proxy_reachable_live` check that hits +`/_health` on the running server. A `pass` here is the strongest signal +that the install actually works end-to-end. + +--- + +## Phase 5 — Final acceptance + +Run the full suite: + +```bash +llm-relay verify all --port --format json +``` + +- `overall == "pass"` or `"warn"` → tell the user the install is + complete, summarising any `warn` items so they know what is + intentionally optional. +- `overall == "fail"` → do **not** declare success. Report which check + failed, what its `remediation` was, and what you attempted. Hand back + to the user. + +A reasonable summary message to the user (text, not JSON): + +> Set up llm-relay 0.9.X. Detected ``. Claude Code is now +> routed through `http://localhost:` and the llm-relay MCP server +> is registered. Open `http://localhost:/dashboard/` for the +> dashboard. `N warn(s)` — optional pieces you can install later if you +> need them: ``. + +--- + +## Permission markers (collected) + +This playbook never assumes user consent for any of the following. Each +must be preceded by a `[PERMISSION: ...]` prompt that names exactly what +you are about to do: + +- **`install-package`** — running `pip install`. +- **`write-config`** — running `llm-relay init`, which writes to + `~/.llm-relay/` and may edit `~/.claude/settings.json`. +- **`edit-claude-settings`** — directly editing `~/.claude/settings.json` + outside of `init` (rare; only as a last resort if `init` cannot + resolve a `verify integration` failure). +- **`fix-permissions`** — `chmod` / `chown` on files in the user's home + directory. +- **`install-service`** — registering the Windows background service. + +--- + +## What not to do + +- Do **not** read or modify `~/.llm-relay/usage.db` or + `~/.claude/*.jsonl` session transcripts as part of setup. Those are + runtime data; setup should not touch them. +- Do **not** install LLM CLIs on the user's behalf. Each CLI vendor has + its own install path (`npm install -g @anthropic-ai/claude-code`, + Codex installer, Gemini CLI install, etc.) and the user should run + those themselves. +- Do **not** change `~/.bashrc` / `~/.zshrc` to "fix PATH" automatically. + If the entry point is missing from PATH, report it and ask the user + how they want to handle it. +- Do **not** retry a failing `verify` more than once with the same + remediation. If the first attempt did not fix it, the situation needs + human attention. + +--- + +## When to stop and ask + +The agent should fall back to the human user — not improvise — in any of +these situations: + +- `env-fingerprint` reports no CLI installed. +- `verify` returns `overall == "fail"` and the failing check has no + `remediation` string. +- A `remediation` proposes editing a file outside the standard paths + (`~/.llm-relay/`, `~/.claude/settings.json`). +- Anything in this playbook is ambiguous for the user's specific setup + (e.g. multiple Python interpreters, multiple Claude Code installs). + +A short, specific question to the user is always preferable to a wrong +auto-fix.