Repository navigation
Probe Bedrock and Vertex provider keys (ADR-0003 follow-up) #15
Description
Why
The current probe layer (src/main/probes/) only knows Anthropic. The catalog has 18 entries but probeProvider() returns error: No probe implemented for "<id>" for any provider whose probeKind is aws, gcp, or azure.
Bedrock and Vertex don't expose a cheap "validate this credential" HTTP endpoint the way OpenAI/Anthropic do — there's no /v1/models you can hit. The standard check is to call a tiny STS-shaped API and parse the response. Three axes:
- Bedrock — call
bedrock:ListFoundationModels(orbedrock:ListCustomModels) with the user's AWS credentials (default chain). 403/expired token ≠ working. - Vertex — call
aiplatform.googleapis.com/v1/projects/<project>/locations/global/publishers/google/modelswith the configured ADC + project + region. TheenvKeys[]for Vertex is empty today so we need to surface a "Project + Region required" step in the wizard before this can be wired. - Azure —
GET <endpoint>/openai/deployments?api-version=2024-02-01with the user'sAZURE_OPENAI_API_KEY. Or just probe the deployment the user names.
Scope
- Wire
awsprobe kind: uses SDK's default credential provider chain; minimal IAM sigv4 againstbedrock-runtimeorbedrockcontrol plane; returnsvalid | errorwith the AWS error code surfaced indetail. - Wire
gcpprobe kind: takes aprojectandregionfrom wizard config; uses@google-cloud/aiplatformor just an authenticated RESTGET; requiresgcloud auth application-default loginupstream. - Wire
azureprobe kind (andazure-openai): simpleGETagainst the deployment list with the configured key. - Vault entry for Vertex needs to carry
{project, region}alongside the credential. Either widenSecretWriteOptionsor surface project/region as separate UI fields in the Keys step. - Update
PROVIDER_CATALOGnotes for cloud_creds rows to indicate "wizard will ask for project/region first". - Renderer Keys step: when user picks Vertex, ask for project id + region before "Save"; persist into vault alongside the credential (or as a sibling secret).
- App + CLI both pick up the same probes via the shared
runProbe()dispatcher.
Acceptance criteria
-
runProbe({ providerId: 'bedrock' })returns{ valid: true, status: 'ok' }against a working AWS account with Bedrock access. - Same returns
{ valid: false, status: 'invalid', detail: 'AWS error: ...' }for an unconfigured credential chain. - Vertex probe requires a saved project + region; errors sensibly without one.
- Azure probe works against an actual deployment.
- All three appear in the
Validatebutton in the renderer without code changes beyond the probe files. - Tests (or fixture-mocked HTTP) for at least the error paths.
-
npm run typecheck && npm run lint && npm run buildclean.
References
- ADR-0003 (
docs/adr/0003-key-validity-and-expiry-probing.md) - AWS SigV4 reference: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
- Vertex
aiplatform.googleapis.com: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/rest - Azure OpenAI REST: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference
Activity
johnnyhuy commentedon Nov 8, 2026
Split for review size: the probe layer (
src/main/probes/aws/gcp/azure kinds, ProbeContext project/region, catalog notes, mocked-HTTP tests) is being worked as #47 on branchagent/issue-47-cloud-probes. The vault storage for Vertex {project, region} and the renderer Keys step fields stay on this issue for a later run. CLI parity incli/src/lib/probes.tswill be a separate PR (independent release flow).johnnyhuy commentedon Nov 8, 2026
Progress from the work loop - the probe layer landed in #48 (closes #47), and the vault and renderer slice for Vertex project and region is up in #50 (closes #49). Still open on this issue - CLI probe parity in cli/src/lib/probes.ts as a separate CLI PR, and live-key verification against real Bedrock and Vertex accounts.
johnnyhuy commentedon Nov 8, 2026
CLI probe parity landed in #52 (the CLI half of this issue, split out as #51). The app probe layer from #48, the vault wiring from #50 and the CLI dispatch now cover Bedrock, Vertex and Azure with mocked HTTP tests on both sides. What remains here needs live provider accounts: a Bedrock probe against a working AWS account, a Vertex probe against a real GCP project, and an Azure probe against a real deployment. Parking the live verification with this note rather than closing, since no agent run can supply those credentials.
johnnyhuy commentedon Nov 12, 2026
The scheduled run rechecked origin/main at 93c2a42. The implementation slices are already merged in #48, #50 and #52. The remaining live-provider acceptance checks need Johnny or a reviewer with working cloud accounts. This run made no live provider calls and used no real credentials.
Added
needs-live-provider-verificationso this issue is explicitly parked and stays open. Existing implementation limits still apply as recorded in #48 and #52: Bedrock reads environment credentials rather than the full AWS SDK default chain, and Vertex takes an access token rather than loading ADC itself. The checks below do not establish live-provider compatibility.Both repository gates passed against the exact origin/main commit.
mise run ci:app- exit 0; catalog check, typecheck, lint, all unit suites and the main/preload/renderer builds passed.mise run ci:cli- exit 0; typecheck, 25 tests with 0 failures, and the Linux x64 compiled binary build passed.git diff --check- exit 0; worktree clean after both gates.Representative proof from the app gate, using mocked HTTP only:
The CLI gate passed
aws mocked 200 returns ok and sends a SigV4 signed request,gcp mocked 200 returns ok with project and region in the path,azure missing endpoint returns error, mocked 200 validates, and the 16 discovery-cache fixture cases.No user-facing change, new commit or PR was needed in this run. The dependency install reported 23 audit findings (12 moderate, 9 high, 2 critical); no dependency versions were changed.
Sweep completed with 0 merged PRs, 0 closed PRs, 0 remote branches deleted and 4 leftover local agent branches deleted after confirming their work had merged. Release PR #46 is excluded from the sweep. The schedule remains active because the exit condition requires a clear PR list and #46 is still open.
johnnyhuy commentedon Nov 13, 2026
Scheduled run reverified origin/main at 93c2a42 on 13 November 2026. No user-facing change or new PR was needed. #16 and #17 are closed; this issue remains explicitly parked with needs-live-provider-verification.
Both repository gates passed with exit 0.
mise run ci:apppassed catalog freshness, typecheck, lint, all unit suites and main/preload/renderer builds.mise run ci:clipassed typecheck, 25 tests with 0 failures and the Linux x64 compiled binary build.git diff --checkpassed and the worktree remained clean.Representative proof from the app gate:
CLI proof includes
aws mocked 200 returns ok and sends a SigV4 signed request,gcp mocked 200 returns ok with project and region in the path,azure missing endpoint returns error, mocked 200 validatesand all 16 discovery-cache fixture tests:No real cloud credentials were used and no live provider calls were made. Johnny or a reviewer with working cloud accounts still needs to perform the acceptance checks. Existing credential-source limits remain as recorded in #48 and #52.
Sweep completed with 0 merged, 0 closed and 0 deleted branches. Only release PR #46 remains open and was excluded from the sweep. The schedule stays active because the exit condition requires a clear PR list.
The app dependency install still reports 23 audit findings (12 moderate, 9 high, 2 critical). No dependencies were changed.
johnnyhuy commentedon Nov 13, 2026
Reverified origin/main at 93c2a42 on 13 November 2026. No user-facing change or new PR was needed. The implementation is already merged in #48, #50 and #52; #16 and #17 are closed.
Both repository gates passed with exit 0.
mise run ci:apppassed catalog freshness, typecheck, lint, all unit suites and the main/preload/renderer builds.mise run ci:clipassed typecheck, all 25 tests and the Linux x64 compiled binary build.git diff --checkpassed; the worktree is clean.Representative app proof from mocked HTTP fixtures:
CLI proof includes
aws mocked 200 returns ok and sends a SigV4 signed request,gcp mocked 200 returns ok with project and region in the path,azure missing endpoint returns error, mocked 200 validatesand all 16 discovery-cache fixture tests.No real provider credentials or live provider calls were used. This issue stays parked with
needs-live-provider-verificationuntil Johnny or a reviewer can perform the live acceptance checks. Existing credential-source limits remain as recorded in #48 and #52.Sweep completed with 0 merged PRs, 0 closed PRs and 0 deleted branches. Only release PR #46 remains open and was excluded from the sweep. The schedule stays active because the exit condition requires a clear PR list. No release commands were run.
The app dependency install still reports 23 audit findings (12 moderate, 9 high, 2 critical). No dependencies were changed.
johnnyhuy commentedon Nov 13, 2026
Scheduled run verified origin/main at
93c2a420c5c1e63f687e64f1cf0681b14b7dfaf2on 13 November 2026. No user-facing change or new PR was needed. #16 and #17 are closed; #15 stays parked withneeds-live-provider-verificationfor Johnny or a reviewer with cloud accounts.Both repository gates passed with exit 0.
mise run ci:apppassed catalog freshness, typecheck, lint, all unit suites and the main/preload/renderer builds.mise run ci:clipassed typecheck, 25 tests and the Linux x64 compiled binary build.git diff --checkpassed and the worktree is clean.Representative proof from the app gate, using mocked responses only:
CLI proof includes
aws mocked 200 returns ok and sends a SigV4 signed request,gcp mocked 200 returns ok with project and region in the path,azure missing endpoint returns error, mocked 200 validates, and the discovery-cache fixture tests.No real provider credentials or live provider calls were used. Credential-source limitations remain as previously recorded: Bedrock reads environment credentials rather than the full SDK default chain; Vertex takes an access token rather than loading ADC itself.
Sweep completed with 0 merged PRs, 0 closed PRs and 0 deleted branches. Only release PR #46 remains open and was excluded. The schedule stays active because the exit condition requires a clear PR list. No release commands were run.
The app install reported the existing 23 dependency audit findings (12 moderate, 9 high, 2 critical). No dependency versions were changed.
johnnyhuy commentedon Nov 13, 2026
Reverified origin/main at
93c2a420c5c1e63f687e64f1cf0681b14b7dfaf2on 13 November 2026. No user-facing change or new PR was needed. This issue remains parked pending live-provider checks by Johnny or a reviewer with working cloud accounts.Both repository gates passed with exit 0.
mise run ci:apppassed catalog freshness, typecheck, lint, all unit suites and main/preload/renderer builds.mise run ci:clipassed typecheck, all 25 tests and the Linux x64 compiled binary build.git diff --checkpassed and the worktree is clean.Representative app proof from mocked HTTP fixtures:
CLI proof includes
aws mocked 200 returns ok and sends a SigV4 signed request,gcp mocked 200 returns ok with project and region in the path,azure missing endpoint returns error, mocked 200 validatesand all 16 discovery-cache fixture cases.No real credentials or live provider calls were used. The previously documented credential-source limits remain. Implementation PRs #48, #50 and #52 are merged.
Sweep completed with 0 merged PRs, 0 closed PRs and 2 stale local agent branches deleted after confirming they were ancestors of origin/main. Release PR #46 remains excluded from the sweep. The schedule remains active because the exit condition requires a clear PR list. No release commands were run.
The app dependency install reported 23 audit findings (12 moderate, 9 high, 2 critical). No dependencies were changed.
johnnyhuy commentedon Nov 13, 2026
Reverified
origin/mainat93c2a420c5c1e63f687e64f1cf0681b14b7dfaf2on 13 November 2026. No user-facing change or new PR was needed.mise run ci:apppassed with exit 0, including catalog freshness, typecheck, lint, all unit suites and the app builds.mise run ci:clipassed with exit 0, including typecheck, 25 tests, 0 failures and the Linux x64 binary build.git diff --checkpassed and the worktree stayed clean.Mocked provider proof from the gates:
No live provider calls or real credentials were used. This issue remains parked for a reviewer with working cloud accounts; the previously documented credential-source limitations remain.
Sweep completed with 0 merged, 0 closed and 0 branches deleted. Release PR #46 remains excluded. The schedule stays active because the pause condition requires a clear PR list. No release commands were run.
johnnyhuy commentedon Nov 13, 2026
Reverified origin/main at
93c2a420c5c1e63f687e64f1cf0681b14b7dfaf2on 14 November 2026. No user-facing change, commit or new PR was needed; the implementation slices are already merged.mise run ci:apppassed with exit 0, covering catalog freshness, typecheck, lint, all unit suites and the main/preload/renderer builds.mise run ci:clipassed with exit 0, covering typecheck, 25 tests with 0 failures and the Linux x64 compiled binary build.git diff --checkpassed; the worktree stayed clean.Representative proof from mocked provider tests:
No real credentials or live provider calls were used. #15 remains parked with
needs-live-provider-verificationfor Johnny or a reviewer with cloud accounts. The existing credential-source limits remain: Bedrock reads environment credentials rather than the full SDK default chain; Vertex takes an access token rather than loading ADC itself.Sweep completed with 0 merged PRs, 0 closed PRs and 0 branches deleted. Release PR #46 remains excluded; the schedule stays active because the pause condition requires a clear PR list. No release commands were run. The app dependency install reported the existing 23 audit findings; no dependencies were changed.