Skip to content

feat: make CLI execution provider-pinned#83

Merged
clawd800 merged 5 commits into
mainfrom
feat/provider-pinned-cli
Jul 23, 2026
Merged

feat: make CLI execution provider-pinned#83
clawd800 merged 5 commits into
mainfrom
feat/provider-pinned-cli

Conversation

@h1-hunt

@h1-hunt h1-hunt commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Make quote/call execution provider-pinned. Explicit --provider calls the concrete path directly; omission performs one wallet-free detail lookup, resolves defaultProvider, and executes one pinned request.
  • Add h402 show <category/action> [--provider <name>] for full provider-native contracts while keeping search compact.
  • Preserve structured 404/410/provider recovery details and never silently retry or select another provider.
  • Keep provider-native output under data and add CLI provenance under h402.cliProviderSelection.
  • Emit a reproducible pinned command that preserves semantic request, backend, wallet, and safety flags while excluding passphrases and old idempotency keys.

Dependency

Depends on Steemhunt/h402-web#632, and transitively #631. Part of Steemhunt/h402-web#628.

Behavior

  • h402 search "web search" returns compact task/provider summaries.
  • h402 show web/search returns every enabled provider contract.
  • h402 show web/search --provider stableenrich-exa returns only that provider contract plus task identity/default metadata.
  • Explicit provider execution performs no catalog lookup.
  • Omitted-provider execution rejects a missing, disabled, malformed, or non-concrete catalog default before payment.
  • Unknown routes remain structured and include h402 search "<route>" recovery without adding a suggestion engine.
  • Route, provider, method, and query preflight runs before network/payment work.
  • Existing x402 authorization, idempotency, max-USD, and pending-settlement boundaries remain unchanged.

Cross-repo smoke

Exercised against PR #632:

  • compact search returned only browse fields
  • full show web/search returned provider-native contracts
  • selected show did not mix another provider's schema/default fields
  • omitted-provider quote resolved one provider and sent no payment
  • omitted-provider free status call returned the provider-native terminal result and sent no payment

Verification

  • npm run typecheck
  • npm run lint
  • npm test — CLI 211 tests, core 14 tests
  • npm run build
  • npm run verify:pack
  • npm run smoke:pack from a clean tarball install
  • git diff --check
  • focused default-provider validation, pinned-command safety, payment retry, and no-retry 410 regressions
  • GitHub CI and CodeQL

Large diff warning

This PR changes 17 files: +1,011 / -138 (net +873).

  • Runtime: +351 / -45
  • Tests: +595 / -45
  • README and skill documentation: +65 / -48

Most of the increase is the provider-selection regression suite (including the 408-line focused test file), followed by the catalog/show and pinned-execution implementation.

@sebayaki

Copy link
Copy Markdown
Member

Claude Review — REJECT (1 finding)

Verified head b6f8201 against epic Steemhunt/h402-web#628; local test suite green (14/14).

Verified: wire construction is unconditionally provider-pinned (assertConcreteProvider rejects auto and non-slugs on every path; zero /routes/auto/ anywhere); 410 / gone-candidate is a hard stop with the machine-readable alternatives surfaced and no candidate-fallback loop exists (the only retry is the settlement re-poll on the same path + same idempotency key, covered by tests for both unpaid and signed flows); search prints the compact projection verbatim as JSON; show implements the two-tier detail with per-candidate schema/price/sample/stats and default badge; accepts[].extra round-trips whole so the server's quote/idempotency binding is echoed in the signed payload; follow-up calls stay pinned via the documented --provider-from-followUp.path flow; SKILL.md/README rewritten to the pinned model with docs-guard tests; no lockfile machinery; error-envelope conventions unchanged; tests strengthened (default resolution, auto rejection, 410 stop, path traversal).

Finding

  1. MINOR — the resolved default is invisible on failure paths. The epic requires the omitted---provider convenience to announce the resolved default loudly, including the fully-pinned reproducible command. Implementation emits h402.cliProviderSelection (with pinnedCommand, source: "catalog-default") only in the stdout success envelope (withProviderSelection, commands.ts:420/585/621). When an omitted-provider call fails (410, 5xx, network), the stderr error envelope contains no selection metadata — the silently-chosen provider is only inferable from detail.url. An agent debugging a failed default call loses exactly the reproduction hint the feature exists to provide.
    • Fix: for omitted-provider invocations, inject the resolved selection (provider, source, pinnedCommand) into CliError.detail / the stderr errorEnvelope as well, with a small regression (omitted-provider call → 410 → stderr envelope carries the selection).

Notes (non-blocking): show --provider's route-field allowlist (commands.ts:365-377) will silently drop future route-level display fields — fine today, worth a comment; encodeRouteId is computed up to 3× per call (harmless).

@sebayaki sebayaki added the rejected-claude Claude Reviewer found issues on this PR head label Jul 23, 2026
@sebayaki

Copy link
Copy Markdown
Member

GPT cross-stack review — REJECT

  1. P1 — The printed “reproducible” command does not reproduce the invocation. packages/cli/src/catalog.ts:35-40 emits only h402 call|quote <route> --provider <provider>. A POST made with --json is printed as a bodyless GET; --query, --method, and a custom --api-url are also lost. Build the command from shell-escaped semantic input arguments so method, payload/query, and backend are preserved, while deliberately excluding passphrases/secrets and unsafe idempotency-key reuse. Add GET-query and POST-body copy/paste tests.

  2. P1 — CLI-generated recovery is incompatible with the web server's settled shape. packages/cli/src/catalog.ts:43-48,128-140 emits top-level routeId/defaultProvider/alternatives[].pinnedPath; web #631 emits {error:{routeId, requestedProvider, defaultProvider, candidates:[{provider,candidateKey,status,path}]}}. The 410 fixtures in provider-selection.test.ts:176-190 and call-settlement-retry.test.ts:165-183 also use the invented top-level/pinnedPath shape. Align local show --provider errors and all fixtures to the exact server envelope so agents need one recovery parser.

  3. P1 — The unknown-route recovery test asserts data production does not return. provider-selection.test.ts:195-207 fabricates routeId and suggestions, while web #632's route-detail 404 currently returns only {error:{code:"route_not_found",message}}. Thus the promised one-round-trip self-service recovery is not available in production. Add the settled machine-readable route suggestions server-side and test the exact response, or explicitly revise the epic contract before merge.

  4. P2 — Default selection is “loud” only after success. In packages/cli/src/commands.ts:557-626, common post-resolution failures (price-cap refusal, missing/signing wallet, 410, and upstream errors) do not carry providerSelection or the pinned command in stderr details; quote has the same gap outside a valid challenge. Attach the already-resolved selection to every error occurring after provider resolution, without retrying or substituting providers.

  5. P2 — Provider-specific show drops route-owned lifecycle metadata. packages/cli/src/commands.ts:365-377 manually allowlists task fields but omits flow and flowFollowUps, even though #628 keeps flow route-owned. Inspecting an async status candidate therefore loses its parent/follow-up relationship. Preserve both fields and add one parent/status fixture.

CI and CodeQL are green, and the no-auto/no-failover execution path is directionally correct. These cross-repository contract mismatches must be resolved before release.

@h1-hunt

h1-hunt commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

GPT Reviewer

Verdict: REJECT

Reviewed head b6f8201b7152463987c5621549586eed70cba4a9 against review snapshot base ce71596970710e33a0f410ec784788ea37b9eb90 using a bounded, tool-free review bundle.

Provider pinning and payment-retry safety are largely sound, and all shared validations pass. However, the advertised reproducible command drops invocation arguments, selected show output loses lifecycle metadata, payable quotes use the wrong provenance location, and the documentation incorrectly removes the optional meta envelope field.

Findings

  1. MAJOR — The generated pinned command cannot reproduce the invocation (packages/cli/src/catalog.ts:39)
    • Evidence: selection() always emits only h402 <command> <route> --provider <provider> and has no access to the original arguments. For calls or quotes using --json, --query, --method, or --api-url, copying this command changes the request method, input, or backend and may execute a different paid operation. This contradicts the required fully pinned, reproducible command contract.
    • Suggested fix: Build the command from the validated invocation, inserting the resolved provider while retaining request-shaping, backend, idempotency, and applicable payment-safety flags. Shell-quote values safely and never echo passphrase values; add regressions for POST bodies, GET queries, and custom API URLs.
  2. MINOR — Selected show output drops the route lifecycle (packages/cli/src/commands.ts:365)
    • Evidence: The routeSummary whitelist includes method, tags, and default metadata but omits route.flow. The authoritative catalog model makes flow route-owned, so h402 show <route> --provider ... loses the lifecycle metadata for async routes even though the candidate cannot supply it.
    • Suggested fix: Include flow: route.flow in the selected route summary and add an async-route regression proving it survives provider filtering.
  3. MINOR — Payable quote provenance is emitted at an inconsistent location (packages/cli/src/commands.ts:415)
    • Evidence: A valid 402 quote exits successfully with top-level providerSelection, while free quote and call successes place the same provenance at h402.cliProviderSelection. The primary quote outcome therefore lacks the provenance location required by the CLI contract and forces machine consumers to branch on response type.
    • Suggested fix: Emit the quote selection at h402.cliProviderSelection while leaving paymentRequired unchanged, and update the quote-output regression accordingly.
  4. MINOR — Documentation incorrectly removes the optional meta envelope slot (README.md:56)
    • Evidence: The README now presents successful calls as { data, h402 }; the package README and SKILL repeat this, and docs tests explicitly reject the former meta notation. The authoritative server contract remains { data, meta?, h402 }, and withProviderSelection itself preserves a top-level meta field.
    • Suggested fix: Document { "data": ..., "meta"?: <reserved envelope metadata>, "h402": ... } in all three artifacts, clarify that meta is optional envelope metadata rather than normalized provider output, and update the docs tests.

Shared exact-snapshot validation

  • diff-check: PASS (0.0s)
  • npm-ci: PASS (1.577s)
  • typecheck: PASS (2.688s)
  • lint: PASS (2.34s)
  • test: PASS (17.674s)
  • build: PASS (1.961s)
  • No paid h402/provider calls were made.

@h1-hunt h1-hunt added the rejected-gpt GPT Reviewer found issues on this PR head label Jul 23, 2026
@h1-hunt

h1-hunt commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Claude Reviewer

Verdict: PASS

Reviewed head b6f8201b7152463987c5621549586eed70cba4a9 against review snapshot base ce71596970710e33a0f410ec784788ea37b9eb90 using a bounded, tool-free review bundle.

PR #83 converts every quote/call to provider-pinned execution and lands cleanly. Explicit --provider performs zero catalog lookups and sends exactly one pinned /routes/{provider}/{category}/{action} request; omitted provider does one wallet-free /api/catalog/routes/ detail fetch, strictly validates defaultProvider and all candidates as concrete lowercase slugs (rejecting an 'auto' default before execution), then sends one pinned request. Route/provider/method/query-value and dot/auto-sentinel validation all precede any catalog, wallet, signing, or execution side effect (fetch-not-called tests). Payment machinery is preserved: one authorization per invocation, byte-identical pending retries on the pinned path, replacement-challenge refusal, bounded reconciliation, strict terminal nonpayment proof, and signed/unsigned 410 stopping with no provider substitution or second signature, all covered by new tests. withProviderSelection adds h402.cliProviderSelection via spread without clobbering server envelope/payment fields, leaving data provider-native; show --provider provably excludes the default candidate's schema/example/sample from the selected contract. Docs/help/SKILL are rewritten to the capability-store, pinned-path framing with concrete providers and provider-native async params, enforced by updated doc tests. Deterministic validation is fully green (diff-check, npm ci, typecheck, lint, 209 CLI + 14 core tests, build). Minor asymmetries examined (quote-402 top-level providerSelection, pinned command omitting body flags, 'alternatives' key naming in CLI-local errors) are deliberate, test-locked, and consistent with the settled #628 direction and this PR's review criteria, so no actionable findings remain.

Findings

None.

Shared exact-snapshot validation

  • diff-check: PASS (0.0s)
  • npm-ci: PASS (1.577s)
  • typecheck: PASS (2.688s)
  • lint: PASS (2.34s)
  • test: PASS (17.674s)
  • build: PASS (1.961s)
  • No paid h402/provider calls were made.

@h1-hunt h1-hunt added passed-claude Claude Reviewer passed this PR head needs-rework and removed rejected-claude Claude Reviewer found issues on this PR head labels Jul 23, 2026
@h1-hunt

h1-hunt commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

H-2 parent verification — GPT findings confirmed

Exact reviewed head: b6f8201b7152463987c5621549586eed70cba4a9

The GPT REJECT and Claude PASS artifacts disagree on four asymmetries. Parent verification against h402-web issue #628, this PR's declared contract, and the exact source/tests confirms the GPT findings:

  1. packages/cli/src/catalog.ts:35-40 builds pinnedCommand from only command/route/provider. A call carrying --json, --query, --method, --api-url, --idempotency-key, or payment-safety flags therefore emits a command that cannot reproduce that invocation. The test at provider-selection.test.ts:84-97 locks this omission. Issue #628 requires a fully-pinned command whose copy/paste reproduces deterministically.
  2. packages/cli/src/commands.ts:365-377 allowlists selected-show route metadata but omits route-owned flow; issue #628 explicitly assigns flow to the route.
  3. packages/cli/src/commands.ts:413-420 emits payable quote provenance at top-level providerSelection, while free quote/call results use h402.cliProviderSelection; provider-selection.test.ts:160-173 locks that schema split despite this PR declaring CLI provenance at h402.cliProviderSelection.
  4. packages/cli/tests/docs.test.ts:24-31 explicitly requires all docs to reject the optional meta slot, while issue #628 explicitly retains the {data, meta?, h402} envelope.

The payment/retry and pinned-path safety work remains green; these are contract/output defects, led by the non-reproducible paid command. Original reviewer artifacts remain unchanged for audit. Final status remains needs rework because one completed reviewer rejected.

@sebayaki

Copy link
Copy Markdown
Member

Addressed all actionable rejected-review findings at 4a5dd41.

  • Omitted --provider now performs one wallet-free detail lookup, validates a concrete enabled default, and executes exactly that pinned provider without retry/failover.
  • Explicit-provider execution still performs no catalog lookup.
  • Reproducible pinnedCommand output preserves semantic request, backend, wallet, and safety flags while excluding passphrases and stale idempotency keys.
  • Structured 404/410 recovery is preserved, including h402 search "<route>", without adding a suggestion engine.
  • Updated the catalog retry fixture to match the intentional single-detail-lookup behavior.

Validation: CLI 211 tests plus core 14 tests, typecheck, lint, build, package verification, clean-tarball smoke, git diff --check, GitHub CI, and CodeQL all pass.

Ready for re-review.

@sebayaki sebayaki removed needs-rework passed-claude Claude Reviewer passed this PR head rejected-gpt GPT Reviewer found issues on this PR head labels Jul 23, 2026
@h1-hunt

h1-hunt commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

GPT Reviewer

Verdict: REJECT

Reviewed head 4a5dd4128132afc684e791965447fe17c1efa77c against review snapshot base ce71596970710e33a0f410ec784788ea37b9eb90 using a bounded, tool-free review bundle.

The flow metadata, quote provenance, optional meta documentation, provider validation, and retry behavior are corrected. However, the advertised pinned command still omits effective backend and budget settings inherited from environment/configuration, so it is neither deterministic nor payment-safety-equivalent.

Findings

  1. MAJOR — Pinned commands omit effective backend and configured payment cap (packages/cli/src/catalog.ts:44)
    • Evidence: pinnedCommand() serializes only raw args.flags. Although callCommand/quoteCommand resolve the actual apiUrl and callCommand derives paymentCap from config.maxUsd, neither effective value reaches command construction. Consequently, an invocation using H402_API_URL, config.backendUrl, or config.maxUsd emits no corresponding --api-url or --max-usd. Replaying it in another environment can target a different backend (including production) or sign a challenge above the original cap, contradicting the fully reproducible and safety-preserving command contract.
    • Suggested fix: Construct selection metadata from the validated effective invocation, including the resolved backend URL and effective max-USD cap when applicable. Continue excluding passphrases, session tokens, payment signatures, and stale idempotency keys. Add copy/paste regressions for H402_API_URL/config.backendUrl and config.maxUsd with no equivalent command-line flags.

Shared exact-snapshot validation

  • diff-check: PASS (0.0s)
  • npm-ci: PASS (1.566s)
  • typecheck: PASS (2.78s)
  • lint: PASS (2.25s)
  • test: PASS (17.694s)
  • build: PASS (2.045s)
  • No paid h402/provider calls were made.

@h1-hunt h1-hunt added the rejected-gpt GPT Reviewer found issues on this PR head label Jul 23, 2026
@h1-hunt

h1-hunt commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Claude Reviewer

Verdict: PASS

Reviewed head 4a5dd4128132afc684e791965447fe17c1efa77c against review snapshot base ce71596970710e33a0f410ec784788ea37b9eb90 using a bounded, tool-free review bundle.

Re-reviewed exact head 4a5dd41 against base ce71596. All four mandated findings are fixed and regression-locked. (1) pinnedCommand is now built from the validated invocation: it preserves --api-url, --json, --query, --method, --name, --wallet, --no-passphrase, --no-credit, and --max-usd with correct POSIX single-quote escaping (verified against an embedded apostrophe in a POST body), while deliberately excluding passphrases and the consumed idempotency key — exclusions the docs state explicitly and tests assert (no "never-print-this", no "idem-1"). (2) Selected show output now includes route-owned flow and flowFollowUps, asserted to survive provider filtering; the allowlist matches issue #628 decision 7's route-owned field set exactly while provably excluding contract-shaped candidate fields. (3) Provenance is uniform at h402.cliProviderSelection across payable quote ({paymentRequired, h402:{...}}), free quote, and free/paid call successes; post-resolution failures on call and quote carry the same metadata at error.detail.h402.cliProviderSelection (signed/unsigned 410, max-usd refusal, missing wallet, upstream 500 all tested), which also resolves the earlier "selection invisible on failure" finding. (4) All three docs restore the authoritative { data, meta?, h402 } envelope with meta as reserved envelope metadata; docs tests now require that notation, and withProviderSelection preserves meta and server h402 fields via spread. Money-safety invariants re-verified: one signature per invocation, byte-identical bounded settlement retries on the same pinned path and idempotency key (including the catalog-resolved provider via the realigned retry fixture), replacement-challenge refusal, hard stop on 410 with error.detail.error.candidates and no provider substitution or re-sign, and route/provider/method/query/cap validation before any catalog, network, wallet, or signing work (fetch-not-called assertions). Missing/auto/non-slug/disabled/unlisted catalog defaults are rejected fail-closed as invalid_catalog_response after exactly one wallet-free detail lookup with no retry or failover. CLI-local provider_unavailable errors and all fixtures use the server's nested {error:{routeId,requestedProvider,defaultProvider,candidates:[{provider,candidateKey,status,path}]}} shape; unknown routes pass through error.detail.error.recovery.command verbatim with no suggestion engine. Zero auto references remain outside the tombstone rejection. Shared validation is green (diff-check, npm ci, typecheck, lint, 211 CLI + 14 core tests, build). Asymmetries examined and judged non-actionable: show --provider prints top-level providerSelection on a non-execution catalog response (outside the quote/call provenance invariant, test-locked); the disabled-default error carries candidates additionally under detail.detail as bonus info while every documented recovery location holds; env/config-derived settings are not echoed into the flag-scoped recipe, matching the documented contract and reproducing correctly in-environment.

Findings

None.

Shared exact-snapshot validation

  • diff-check: PASS (0.0s)
  • npm-ci: PASS (1.566s)
  • typecheck: PASS (2.78s)
  • lint: PASS (2.25s)
  • test: PASS (17.694s)
  • build: PASS (2.045s)
  • No paid h402/provider calls were made.

@h1-hunt h1-hunt added passed-claude Claude Reviewer passed this PR head needs-rework labels Jul 23, 2026
@h1-hunt

h1-hunt commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

H-2 parent verification — GPT finding confirmed

Exact reviewed head: 4a5dd4128132afc684e791965447fe17c1efa77c

The four findings from the prior review are fixed, but the new GPT major finding is reproducible on this head:

  • quoteCommand() and callCommand() compute the effective backend with backendUrl(config, flagString(args.flags, "api-url")); callCommand() separately computes the effective payment cap with maxUsd(args, config).
  • Both then call resolveProvider(..., args.flags). pinnedCommand() serializes only those raw CLI flags, so values supplied by H402_API_URL, config.backendUrl, or config.maxUsd never reach the copied command.
  • The current provider-selection regression demonstrates the backend mismatch directly: its mocked effective backend sends both requests to https://test.example, while the asserted copied command embeds the raw --api-url https://custom.example/v1 flag. There is no coverage for an environment/config-only backend or configured maxUsd.

Therefore a command copied from one effective invocation can target a different backend or lose its payment ceiling in another environment. Claude's PASS artifact and its label remain preserved as the independent reviewer verdict; the mixed verdict is resolved operationally by rejected-gpt + passed-claude + needs-rework.

@sebayaki sebayaki removed needs-rework passed-claude Claude Reviewer passed this PR head rejected-gpt GPT Reviewer found issues on this PR head labels Jul 23, 2026
@clawd800
clawd800 merged commit e426cef into main Jul 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants