feat(onboarding): add verify primitives (install/config/integration) - #20
Merged
Conversation
Second slice of Path B (LLM-driven onboarding). Where env-fingerprint
describes the *state* of the user's environment, `verify` asserts
expectations about that state -- producing pass/fail/warn/skipped per
check so an agent (Claude Code / Codex / Gemini) can decide what to do
next without scraping human-friendly output.
New CLI surface:
llm-relay verify install
llm-relay verify config [--port]
llm-relay verify integration --cli {claude-code,openai-codex,gemini-cli,all} [--live]
llm-relay verify all [--live]
All subcommands share:
--format {text,json} text for humans, json for agents
--quiet suppress pass-only entries
--no-remediation omit remediation strings
Exit code: 0 on pass/warn, 1 on fail.
Output schema (`schema_version: "1"`):
{
"schema_version": "1",
"target": "install|config|integration|<cli-id>|all",
"captured_at": "...",
"overall": "pass|fail|warn",
"summary": {"pass": N, "fail": N, "warn": N, "skipped": N},
"checks": [
{"id", "label", "status", "detail", "remediation"?, "data"?}
]
}
Status semantics:
pass expectation met
warn met with caveat the operator/agent should know
fail not met; remediation should fix it
skipped not applicable (e.g. CLI not installed)
Overall priority: fail > warn > (pass | skipped). Skipped checks never
escalate to fail.
Checks (24 total, distributed across the modules):
install (6): python_version, package_importable, entry_point_relay,
entry_point_mcp (warn-on-missing), proxy_extras (warn-on-missing),
version_consistency
config (7): db_dir_exists, db_initialized, db_writable, config_file
(warn), knowledge_dir (warn), port_available (warn-on-busy),
no_deprecated_env (warn-on-CCPULSE_*/CC_RELAY_*)
integration claude-code (4): binary, settings_present, proxy_route
(ANTHROPIC_BASE_URL → localhost), mcp_server (llm-relay in mcpServers)
integration openai-codex (3): binary, config_present, proxy_route
(always skipped pending upstream support)
integration gemini-cli (3): binary, config_dir_present,
oauth_known_issue (warn referencing upstream #25425)
Implementation notes:
- Pure-Python; reuses setup_init / orch.discovery / recover.doctor /
detect.scanner helpers so detection logic is not duplicated.
- `run_check()` wrapper captures exceptions as fail-with-detail so a
single broken probe doesn't crash a whole report.
- `aggregate()` namespaces sub-report check ids as `<sub-target>.<id>`
to avoid collisions in `verify all` / `verify integration --cli all`.
- `--live` is opt-in because it makes a network call; pure verify
remains offline-friendly.
- Tests (47 new) cover schema contracts, overall-priority rules,
exception capture, per-check pass/fail/warn/skipped wiring across
all three CLIs, and `verify all` aggregation.
Full suite: 593 pass (546 baseline + 47 new). Ruff clean. CHANGELOG
documents the schema and exit-code contract.
4 tasks
cnighswonger
pushed a commit
to cnighswonger/llm-relay
that referenced
this pull request
Jun 1, 2026
Third and final slice of Path B (LLM-driven onboarding). With env-fingerprint (PR ArkNill#19) describing state and verify (PR ArkNill#20) asserting expectations, this document sequences them into an end-to-end install flow an agent can follow without scraping human-friendly output. Audience: AI coding agents (Claude Code / Codex / Gemini) running an llm-relay setup on a user's behalf. The document is explicit that it is NOT a human tutorial -- humans should use README.md and `llm-relay init` directly. Structure: Phase 0 — Probe (env-fingerprint) Phase 1 — Install or upgrade the package (with extras decision table) Phase 2 — Initialize local state (llm-relay init) Phase 3 — Per-CLI integration (claude-code / openai-codex / gemini-cli) Phase 4 — Optional: start the server (Linux/macOS vs Windows service) Phase 5 — Final acceptance (verify all) Each phase ends with a `verify` call, and the playbook spells out how to respond to pass / warn / fail outcomes. Permission protocol: The playbook never assumes consent for system-modifying actions. Five explicit markers gate any operation that touches outside of read-only probes: [PERMISSION: install-package] pip install [PERMISSION: write-config] llm-relay init (writes ~/.llm-relay/, edits ~/.claude/settings.json) [PERMISSION: edit-claude-settings] direct settings.json edit (last resort) [PERMISSION: fix-permissions] chmod / chown on home dir files [PERMISSION: install-service] Windows background service registration "When to stop and ask" section enumerates ambiguous states (no CLI installed, verify fail with no remediation, multiple Python interpreters, etc.) where the agent must defer to the user instead of improvising. "What not to do" section pins down the agent's scope: do not modify session transcripts, do not install LLM CLIs on the user's behalf, do not auto-edit shell rc files, do not retry a failing remediation more than once. README updated with an "Agent-driven setup" section linking to the playbook and showing the four entry-point commands. CHANGELOG documents the playbook under Unreleased ### Added. Schema contract: both env-fingerprint and verify use schema_version "1"; the playbook calls out that an agent should fall back to the in-tree playbook for whatever release it has installed if the schema version differs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second slice of Path B (LLM-driven onboarding). Where
env-fingerprint(PR #19)describes the state of the user's environment,
verifyasserts expectationsabout that state — producing pass/fail/warn/skipped per check so an agent
(Claude Code / Codex / Gemini) can decide what to do next without scraping
human-friendly output.
New CLI surface
Common options:
--format {text,json},--quiet,--no-remediation.Exit code: 0 on pass/warn, 1 on fail.
Output schema (schema_version "1")
{ \"schema_version\": \"1\", \"target\": \"install|config|integration|<cli-id>|all\", \"captured_at\": \"...\", \"overall\": \"pass|fail|warn\", \"summary\": {\"pass\": N, \"fail\": N, \"warn\": N, \"skipped\": N}, \"checks\": [ {\"id\", \"label\", \"status\", \"detail\", \"remediation\"?, \"data\"?} ] }Status semantics:
pass— expectation metwarn— met with caveatfail— not met; remediation should fix itskipped— not applicable (e.g. CLI not installed)Overall priority: fail > warn > (pass | skipped). Skipped never escalates to fail.
Checks (24 total)
Design notes
setup_init/orch.discovery/recover.doctor/detect.scannerhelpers — no duplicated detection logic.run_check()wrapper: captures exceptions as fail-with-detail so a single broken probe doesn't crash a whole report.aggregate(): namespaces sub-report check ids as<sub-target>.<id>to avoid collisions inverify all/verify integration --cli all.--liveis opt-in: requires a network call (/_health); pure verify remains offline-friendly.Test coverage
47 new tests under
tests/test_verify/:run_checkverify allaggregation with id namespacingFull suite: 593 pass (546 baseline + 47 new). Ruff clean.
Smoke output (text format)
What's next (separate PR)
Final Path B deliverable:
docs/AGENT_SETUP.md— the onboarding playbook that sequences env-fingerprint + verify into an end-to-end setup flow an agent can follow.