diff --git a/CHANGELOG.md b/CHANGELOG.md index de4b781..9f25fe5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ All notable changes to llm-relay are documented here. ## [Unreleased] ### Added +- **Agent-driven setup playbook** (`docs/AGENT_SETUP.md`): structured onboarding document targeted at AI coding agents (Claude Code / Codex / Gemini) automating an llm-relay install on a user's behalf. Sequences `env-fingerprint` + `verify` into a five-phase flow (probe → install → init → per-CLI integration → final acceptance) with explicit `[PERMISSION: …]` markers wherever the agent must pause for user consent, and explicit "stop and ask" rules for ambiguous states. README section "Agent-driven setup" links to it. - **`llm-relay verify` command group** (`verify/`): four idempotent verification subcommands designed for agent-driven onboarding. `verify install` confirms the package itself (Python version, importability, entry points, optional extras, version consistency). `verify config` confirms local state (db dir, schema, writability, config file, knowledge dir, port availability, no deprecated env vars). `verify integration --cli {claude-code,openai-codex,gemini-cli,all}` confirms each CLI is wired through the relay (binary on PATH, settings file present, `ANTHROPIC_BASE_URL` routing to localhost, MCP server registered). `verify all` aggregates install + config + integration. Shared output schema (`schema_version: "1"`) emits per-check status (`pass`/`fail`/`warn`/`skipped`) with optional remediation strings; overall priority is `fail > warn > pass`. Skipped checks (e.g. CLI not installed) never escalate to fail. Optional `--live` probes `/_health` on the proxy. Common options: `--format {text,json}`, `--quiet`, `--no-remediation`. Exit code is 0 on pass/warn, 1 on fail. - **`llm-relay env-fingerprint` command** (`env_fingerprint.py`): single-shot, idempotent snapshot of the local LLM CLI environment for agent-driven onboarding. Emits a structured JSON (or YAML) document with sections for `llm_relay` package state, per-CLI install/auth/version, port availability, filesystem layout, redacted environment variables (API keys reported as `set`/`empty`/`null` only), and an optional `doctor` summary. Designed so an agent (Claude Code / Codex / Gemini) automating an llm-relay install can parse the environment instead of scraping `init` output. Schema versioned (`schema_version: "1"`); sub-probe failures surface as `_error` markers without crashing the snapshot. Options: `--format {json,yaml}`, `--no-doctor`, `--ports`. diff --git a/README.md b/README.md index 812fa4d..80559ee 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,24 @@ llm-relay-mcp # stdio transport, 8 tools llm-relay connect # Auto-configures Claude Code proxy ``` +### Agent-driven setup + +If you would rather have your existing coding agent (Claude Code, Codex, +Gemini) run the install for you, point it at +[`docs/AGENT_SETUP.md`](docs/AGENT_SETUP.md). It is a structured playbook +the agent follows step by step, using `llm-relay env-fingerprint` and +`llm-relay verify` to probe and check each step without scraping output. + +```bash +llm-relay env-fingerprint --format json # state snapshot +llm-relay verify install --format json # is the package usable? +llm-relay verify config --format json # is local state set up? +llm-relay verify integration --cli claude-code # is the CLI wired? +llm-relay verify all # everything at once +``` + +Exit code is 0 on pass/warn, 1 on fail. + ## CLI Status | CLI | Status | diff --git a/docs/AGENT_SETUP.md b/docs/AGENT_SETUP.md new file mode 100644 index 0000000..c2dc07b --- /dev/null +++ b/docs/AGENT_SETUP.md @@ -0,0 +1,326 @@ +# Agent Setup Playbook + +> **Audience**: an AI coding agent (Claude Code, OpenAI Codex, Gemini CLI) +> automating an llm-relay install on behalf of a human user. +> **This document is not a tutorial for humans** — it is a sequence of +> structured probes and actions an agent can follow without asking the +> user for technical details. + +If you are a human, you probably want [`README.md`](../README.md) and the +`llm-relay init` command instead. + +--- + +## What this playbook is + +llm-relay ships with two agent-consumable primitives: + +- **`llm-relay env-fingerprint --format json`** — describes the *state* of + the user's environment (which CLIs are installed, what ports are free, + what env vars are set, etc.). Read-only. +- **`llm-relay verify {install|config|integration|all} --format json`** — + asserts *expectations* about that state, returning pass/fail/warn/skipped + per check with optional remediation strings. Read-only. + +This playbook sequences those two primitives — plus existing commands like +`llm-relay init` — into an end-to-end install flow. + +You do not need to scrape human-friendly output. Every step in this +playbook either: + +1. Calls a command that emits structured JSON. +2. Performs a single, idempotent filesystem / shell action. +3. Pauses for explicit user permission via a clearly labelled marker. + +## What it is not + +- It is not a substitute for the user's permission. Markers like + **`[PERMISSION: install-package]`** below indicate steps where the user + must approve a destructive or system-modifying action before you proceed. +- It is not a place to make undocumented decisions. If `env-fingerprint` + reveals state this playbook does not branch on, **stop and ask the + user** rather than guessing. + +--- + +## Schema contract + +Both primitives use `"schema_version": "1"`. If you see a different +`schema_version`, this playbook may be out of date for your llm-relay +release — fall back to the in-tree +[`docs/AGENT_SETUP.md`](AGENT_SETUP.md) from that release. + +`verify` exit codes: +- `0` — overall is `pass` or `warn`. Safe to proceed. +- `1` — overall is `fail`. Do not proceed past the failing step. + +--- + +## Phase 0 — Probe + +Goal: understand what the user already has so later phases skip the work +that is already done. + +```bash +llm-relay env-fingerprint --format json --no-doctor +``` + +Parse the result. Treat `_error` markers on any section as "section +unavailable" — do not abort the whole flow; just proceed with the +remaining sections. + +Key fields you will need later: + +| field | used in | +|---|---| +| `llm_relay.version` | Phase 1 — decide whether to install | +| `llm_relay.db_dir` | Phase 3 — where init writes | +| `clis[*].id`, `clis[*].installed`, `clis[*].auth.preferred` | Phase 5 — per-CLI wiring | +| `ports["8083"]` | Phase 3 — choose `--port` for init | +| `env.ANTHROPIC_BASE_URL` | Phase 5 — already-wired Claude Code | + +If `clis[*].installed` is `false` for **every** CLI: stop and ask the user +to install at least one LLM CLI first — llm-relay has no purpose without +one. Do **not** install CLIs on the user's behalf in this playbook. + +--- + +## Phase 1 — Install or upgrade the package + +### Decision + +- If `llm_relay.version` is `null` → llm-relay is not installed yet. +- If `llm_relay.version` is older than the version you are setting up + against → upgrade. +- Otherwise → skip to Phase 2. + +### Which extras to install + +Decide from the fingerprint: + +| Condition | Install | +|---|---| +| Always | `llm-relay` (base) | +| Any CLI installed | `llm-relay[proxy]` (the dashboard / proxy needs httpx + uvicorn) | +| `clis[*].id == "claude-code"` and `installed` | `llm-relay[mcp]` (Claude Code can register the MCP server) | +| User says they want PostgreSQL | `llm-relay[pg]` | +| Otherwise | `llm-relay[all]` is a safe default | + +### Action + +**`[PERMISSION: install-package]`** — installing into the user's Python +environment modifies their site-packages. Ask before running: + +```bash +pip install llm-relay[] +``` + +If the user uses a virtual environment that you can detect (e.g. they ran +the agent from inside an activated venv), install there; otherwise default +to `pip install --user` and tell them which Python interpreter received the +install. + +### Verify + +```bash +llm-relay verify install --format json +``` + +- `overall == "pass"`: proceed. +- `overall == "warn"`: read the `warn` entries — typically + `entry_point_mcp` or `proxy_extras` is missing because the user opted + for a slim install. Acceptable unless the user explicitly asked for + those features. +- `overall == "fail"`: read the failing check's `remediation`, attempt the + remediation **once**, then re-verify. If it still fails, stop and report + to the user. + +--- + +## Phase 2 — Initialize local state + +### Decision + +- If `verify config` already returns `overall == "pass"` (or `warn` for + only the optional `knowledge_dir` / `config_file` items) → skip to + Phase 4. +- Otherwise → init. + +### Action + +**`[PERMISSION: write-config]`** — `llm-relay init` writes to +`~/.llm-relay/` and edits `~/.claude/settings.json` (if Claude Code is +installed). Ask before running. + +Pick `--port`: +- If `ports["8083"]` is `"free"` → default `--port 8083`. +- If it is `"in_use"` → either the relay is already running on it (check + by hitting `/_health` — Phase 5 covers this) or another process owns it. + Ask the user before overriding; do not silently move to a new port. + +```bash +llm-relay init --port +``` + +### Verify + +```bash +llm-relay verify config --port --format json +``` + +`fail` here usually means the install partially completed. Read each +failing check's `remediation`. The most common cause is a permissions +issue on `~/.llm-relay/` — do not chmod files on the user's behalf +without **`[PERMISSION: fix-permissions]`**. + +--- + +## Phase 3 — Per-CLI integration + +For every CLI where `env-fingerprint` reported `installed: true`, run: + +```bash +llm-relay verify integration --cli --format json +``` + +Then act based on the failing checks below. **Do not run integration +steps for CLIs the user does not have installed.** + +### Claude Code (`claude-code`) + +| Failing check | Action | +|---|---| +| `binary` | Cannot happen if `env-fingerprint` said `installed: true`; if it does, treat as a transient PATH issue and re-probe. | +| `settings_present` | Have the user open Claude Code once; the binary creates `~/.claude/settings.json` on first run. | +| `proxy_route` | `llm-relay init` should have set this. If `init` already ran and this is still `fail`, inspect `~/.claude/settings.json` `env.ANTHROPIC_BASE_URL` — fix to `http://localhost:` only with **`[PERMISSION: edit-claude-settings]`**. | +| `mcp_server` | Same as above — `init` writes `mcpServers["llm-relay"]`. If missing, re-run `init`; do not edit the JSON directly unless `init` itself fails. | + +### OpenAI Codex (`openai-codex`) + +The `proxy_route` check is `skipped` by design — Codex does not currently +expose a stable routing knob. Do **not** attempt to monkey-patch Codex's +config to route through llm-relay; that is upstream work, not yours. + +`binary` and `config_present` should both pass after the user has run +`codex` at least once. + +### Gemini CLI (`gemini-cli`) + +`oauth_known_issue` is always `warn` and surfaces upstream +[google-gemini/gemini-cli#25425](https://github.com/google-gemini/gemini-cli/issues/25425). +If the user reports a 403 from Gemini, tell them to set +`GEMINI_API_KEY` instead of relying on oauth-personal. + +--- + +## Phase 4 — Optional: start the server + +Only if the user explicitly asked for the dashboard or a background +service. Otherwise leave it for them to start with `llm-relay serve`. + +**Linux / macOS (manual):** + +```bash +llm-relay serve --port +``` + +**Windows (background service):** + +**`[PERMISSION: install-service]`** — this registers an auto-start entry +in the user's Startup Folder. Ask before running. + +```bash +llm-relay service install --port +llm-relay service start --port +``` + +### Verify + +After the server is running: + +```bash +llm-relay verify integration --cli all --live --port --format json +``` + +The `--live` flag adds a `proxy_reachable_live` check that hits +`/_health` on the running server. A `pass` here is the strongest signal +that the install actually works end-to-end. + +--- + +## Phase 5 — Final acceptance + +Run the full suite: + +```bash +llm-relay verify all --port --format json +``` + +- `overall == "pass"` or `"warn"` → tell the user the install is + complete, summarising any `warn` items so they know what is + intentionally optional. +- `overall == "fail"` → do **not** declare success. Report which check + failed, what its `remediation` was, and what you attempted. Hand back + to the user. + +A reasonable summary message to the user (text, not JSON): + +> Set up llm-relay 0.9.X. Detected ``. Claude Code is now +> routed through `http://localhost:` and the llm-relay MCP server +> is registered. Open `http://localhost:/dashboard/` for the +> dashboard. `N warn(s)` — optional pieces you can install later if you +> need them: ``. + +--- + +## Permission markers (collected) + +This playbook never assumes user consent for any of the following. Each +must be preceded by a `[PERMISSION: ...]` prompt that names exactly what +you are about to do: + +- **`install-package`** — running `pip install`. +- **`write-config`** — running `llm-relay init`, which writes to + `~/.llm-relay/` and may edit `~/.claude/settings.json`. +- **`edit-claude-settings`** — directly editing `~/.claude/settings.json` + outside of `init` (rare; only as a last resort if `init` cannot + resolve a `verify integration` failure). +- **`fix-permissions`** — `chmod` / `chown` on files in the user's home + directory. +- **`install-service`** — registering the Windows background service. + +--- + +## What not to do + +- Do **not** read or modify `~/.llm-relay/usage.db` or + `~/.claude/*.jsonl` session transcripts as part of setup. Those are + runtime data; setup should not touch them. +- Do **not** install LLM CLIs on the user's behalf. Each CLI vendor has + its own install path (`npm install -g @anthropic-ai/claude-code`, + Codex installer, Gemini CLI install, etc.) and the user should run + those themselves. +- Do **not** change `~/.bashrc` / `~/.zshrc` to "fix PATH" automatically. + If the entry point is missing from PATH, report it and ask the user + how they want to handle it. +- Do **not** retry a failing `verify` more than once with the same + remediation. If the first attempt did not fix it, the situation needs + human attention. + +--- + +## When to stop and ask + +The agent should fall back to the human user — not improvise — in any of +these situations: + +- `env-fingerprint` reports no CLI installed. +- `verify` returns `overall == "fail"` and the failing check has no + `remediation` string. +- A `remediation` proposes editing a file outside the standard paths + (`~/.llm-relay/`, `~/.claude/settings.json`). +- Anything in this playbook is ambiguous for the user's specific setup + (e.g. multiple Python interpreters, multiple Claude Code installs). + +A short, specific question to the user is always preferable to a wrong +auto-fix.