Skip to content

feat(onboarding): add verify primitives (install/config/integration) - #20

Merged
ArkNill merged 1 commit into
mainfrom
feat/verify-primitives
May 20, 2026
Merged

feat(onboarding): add verify primitives (install/config/integration)#20
ArkNill merged 1 commit into
mainfrom
feat/verify-primitives

Conversation

@ArkNill

@ArkNill ArkNill commented May 20, 2026

Copy link
Copy Markdown
Owner

Summary

Second slice of Path B (LLM-driven onboarding). Where env-fingerprint (PR #19)
describes the state of the user's environment, verify asserts expectations
about that state — producing pass/fail/warn/skipped per check so an agent
(Claude Code / Codex / Gemini) can decide what to do next without scraping
human-friendly output.

New CLI surface

llm-relay verify install
llm-relay verify config [--port]
llm-relay verify integration --cli {claude-code,openai-codex,gemini-cli,all} [--live]
llm-relay verify all [--live]

Common options: --format {text,json}, --quiet, --no-remediation.
Exit code: 0 on pass/warn, 1 on fail.

Output schema (schema_version "1")

{
  \"schema_version\": \"1\",
  \"target\": \"install|config|integration|<cli-id>|all\",
  \"captured_at\": \"...\",
  \"overall\": \"pass|fail|warn\",
  \"summary\": {\"pass\": N, \"fail\": N, \"warn\": N, \"skipped\": N},
  \"checks\": [
    {\"id\", \"label\", \"status\", \"detail\", \"remediation\"?, \"data\"?}
  ]
}

Status semantics:

  • pass — expectation met
  • warn — met with caveat
  • fail — not met; remediation should fix it
  • skipped — not applicable (e.g. CLI not installed)

Overall priority: fail > warn > (pass | skipped). Skipped never escalates to fail.

Checks (24 total)

target check ids
install (6) python_version, package_importable, entry_point_relay, entry_point_mcp (warn-on-missing), proxy_extras (warn-on-missing), version_consistency
config (7) db_dir_exists, db_initialized, db_writable, config_file (warn), knowledge_dir (warn), port_available (warn-on-busy), no_deprecated_env (warn-on-CCPULSE_/CC_RELAY_)
claude-code (4) binary, settings_present, proxy_route, mcp_server
openai-codex (3) binary, config_present, proxy_route (always skipped pending upstream support)
gemini-cli (3) binary, config_dir_present, oauth_known_issue (warn referencing upstream #25425)

Design notes

  • Pure-Python: reuses setup_init / orch.discovery / recover.doctor / detect.scanner helpers — no duplicated detection logic.
  • run_check() wrapper: captures exceptions as fail-with-detail so a single broken probe doesn't crash a whole report.
  • aggregate(): namespaces sub-report check ids as <sub-target>.<id> to avoid collisions in verify all / verify integration --cli all.
  • --live is opt-in: requires a network call (/_health); pure verify remains offline-friendly.

Test coverage

47 new tests under tests/test_verify/:

  • Schema contracts + status validation
  • Overall-priority rules
  • Exception capture via run_check
  • Per-check pass/fail/warn/skipped wiring for all three CLIs
  • verify all aggregation with id namespacing

Full suite: 593 pass (546 baseline + 47 new). Ruff clean.

Smoke output (text format)

$ llm-relay verify install --quiet
target:  install
overall: warn
summary: pass=5 fail=0 warn=1 skipped=0

[warn   ] entry_point_mcp: llm-relay-mcp binary not found (optional MCP extra)
          → pip install llm-relay[mcp] (only needed if exposing the MCP server)

What's next (separate PR)

Final Path B deliverable: docs/AGENT_SETUP.md — the onboarding playbook that sequences env-fingerprint + verify into an end-to-end setup flow an agent can follow.

Second slice of Path B (LLM-driven onboarding). Where env-fingerprint
describes the *state* of the user's environment, `verify` asserts
expectations about that state -- producing pass/fail/warn/skipped per
check so an agent (Claude Code / Codex / Gemini) can decide what to do
next without scraping human-friendly output.

New CLI surface:

  llm-relay verify install
  llm-relay verify config [--port]
  llm-relay verify integration --cli {claude-code,openai-codex,gemini-cli,all} [--live]
  llm-relay verify all [--live]

All subcommands share:
  --format {text,json}       text for humans, json for agents
  --quiet                    suppress pass-only entries
  --no-remediation           omit remediation strings

Exit code: 0 on pass/warn, 1 on fail.

Output schema (`schema_version: "1"`):

  {
    "schema_version": "1",
    "target": "install|config|integration|<cli-id>|all",
    "captured_at": "...",
    "overall": "pass|fail|warn",
    "summary": {"pass": N, "fail": N, "warn": N, "skipped": N},
    "checks": [
      {"id", "label", "status", "detail", "remediation"?, "data"?}
    ]
  }

Status semantics:
  pass     expectation met
  warn     met with caveat the operator/agent should know
  fail     not met; remediation should fix it
  skipped  not applicable (e.g. CLI not installed)

Overall priority: fail > warn > (pass | skipped). Skipped checks never
escalate to fail.

Checks (24 total, distributed across the modules):

  install (6): python_version, package_importable, entry_point_relay,
    entry_point_mcp (warn-on-missing), proxy_extras (warn-on-missing),
    version_consistency

  config (7): db_dir_exists, db_initialized, db_writable, config_file
    (warn), knowledge_dir (warn), port_available (warn-on-busy),
    no_deprecated_env (warn-on-CCPULSE_*/CC_RELAY_*)

  integration claude-code (4): binary, settings_present, proxy_route
    (ANTHROPIC_BASE_URL → localhost), mcp_server (llm-relay in mcpServers)

  integration openai-codex (3): binary, config_present, proxy_route
    (always skipped pending upstream support)

  integration gemini-cli (3): binary, config_dir_present,
    oauth_known_issue (warn referencing upstream #25425)

Implementation notes:

- Pure-Python; reuses setup_init / orch.discovery / recover.doctor /
  detect.scanner helpers so detection logic is not duplicated.
- `run_check()` wrapper captures exceptions as fail-with-detail so a
  single broken probe doesn't crash a whole report.
- `aggregate()` namespaces sub-report check ids as `<sub-target>.<id>`
  to avoid collisions in `verify all` / `verify integration --cli all`.
- `--live` is opt-in because it makes a network call; pure verify
  remains offline-friendly.
- Tests (47 new) cover schema contracts, overall-priority rules,
  exception capture, per-check pass/fail/warn/skipped wiring across
  all three CLIs, and `verify all` aggregation.

Full suite: 593 pass (546 baseline + 47 new). Ruff clean. CHANGELOG
documents the schema and exit-code contract.
@ArkNill
ArkNill merged commit 26a1139 into main May 20, 2026
6 checks passed
@ArkNill
ArkNill deleted the feat/verify-primitives branch May 20, 2026 09:00
cnighswonger pushed a commit to cnighswonger/llm-relay that referenced this pull request Jun 1, 2026
Third and final slice of Path B (LLM-driven onboarding). With
env-fingerprint (PR ArkNill#19) describing state and verify (PR ArkNill#20)
asserting expectations, this document sequences them into an
end-to-end install flow an agent can follow without scraping
human-friendly output.

Audience: AI coding agents (Claude Code / Codex / Gemini) running
an llm-relay setup on a user's behalf. The document is explicit
that it is NOT a human tutorial -- humans should use README.md and
`llm-relay init` directly.

Structure:

  Phase 0 — Probe (env-fingerprint)
  Phase 1 — Install or upgrade the package (with extras decision table)
  Phase 2 — Initialize local state (llm-relay init)
  Phase 3 — Per-CLI integration (claude-code / openai-codex / gemini-cli)
  Phase 4 — Optional: start the server (Linux/macOS vs Windows service)
  Phase 5 — Final acceptance (verify all)

Each phase ends with a `verify` call, and the playbook spells out how
to respond to pass / warn / fail outcomes.

Permission protocol:

The playbook never assumes consent for system-modifying actions. Five
explicit markers gate any operation that touches outside of read-only
probes:

  [PERMISSION: install-package]      pip install
  [PERMISSION: write-config]          llm-relay init (writes ~/.llm-relay/, edits ~/.claude/settings.json)
  [PERMISSION: edit-claude-settings] direct settings.json edit (last resort)
  [PERMISSION: fix-permissions]       chmod / chown on home dir files
  [PERMISSION: install-service]       Windows background service registration

"When to stop and ask" section enumerates ambiguous states (no CLI
installed, verify fail with no remediation, multiple Python
interpreters, etc.) where the agent must defer to the user instead
of improvising.

"What not to do" section pins down the agent's scope: do not modify
session transcripts, do not install LLM CLIs on the user's behalf,
do not auto-edit shell rc files, do not retry a failing remediation
more than once.

README updated with an "Agent-driven setup" section linking to the
playbook and showing the four entry-point commands.

CHANGELOG documents the playbook under Unreleased ### Added.

Schema contract: both env-fingerprint and verify use
schema_version "1"; the playbook calls out that an agent should
fall back to the in-tree playbook for whatever release it has
installed if the schema version differs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant