Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
34 changes: 34 additions & 0 deletions docs/pr-evidence/custom-provider-web-flow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Custom Provider Web Flow

## Reproduced failures

A production-style local instance exposed four separate breaks in the same flow:

1. `PUT /v1/admin/custom-providers/:id` returned `404` because the production entrypoint did not pass the custom-provider dependencies to the server.
2. A configured custom-provider key did not satisfy Portal or Admin readiness checks.
3. A custom model was rejected on Web turns unless the administrator explicitly saved a Web UI model allowlist.
4. The runtime config advertised the custom model, but the Web picker silently discarded it because the browser lacked its protocol metadata.

## Automated verification

- The production entrypoint is covered by a regression assertion for both required dependencies.
- Core route tests cover custom-provider readiness, status hygiene, runtime catalog metadata, and a Web turn without an explicit picker allowlist.
- Web UI tests cover generic OpenAI-compatible custom-model construction and selection.
- Admin tests cover custom-provider readiness, initial table loading, and readiness refresh after save/remove mutations.
- The full root suite passed once at 3,818 tests with 0 failures and 135 skipped. A later post-review run hit one unrelated OpenCode startup-output timing failure; that test passed immediately on an isolated rerun (10/10 OpenCode harness tests).
- Prettier, TypeScript, ESLint, and Oxlint checks pass.

## Live DeepSeek verification

The current branch was loaded through the repository's production-style dev supervisor and tested through Chrome against the saved `deepseek` provider. No credential is present in these artifacts.

- Portal `/` returned the Web application instead of redirecting to onboarding.
- Admin reported `Ready`, identified `deepseek-v4-flash · custom provider key`, and listed the provider with a write-only key status.
- The Web picker displayed `DeepSeek V4 Flash` under the Pi harness without requiring an explicit model allowlist.
- Text turn: `DS_PR_TEXT_OK_20260810`.
- Tool turn: the model called `execute` with `pwd`; Admin recorded exit code `0`, the next model request contained the tool result `/root/workspace`, and the model returned `DS_PR_TOOL_OK`.
- Session: `4de03b0b-6ec4-426d-aa72-8fd066cf9246`.

![Admin reports the custom provider as ready](custom-provider-onboarding-ready.png)

![DeepSeek completes text and tool turns in the Web UI](custom-provider-web-turn.png)
Binary file added docs/pr-evidence/custom-provider-web-turn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
28 changes: 22 additions & 6 deletions plugins/admin/public/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -6977,6 +6977,7 @@ <h2 id="governance-review-title">Confirm governance change</h2>

let onboardingModels = {};
let onboardingModelStatuses = [];
let onboardingCustomProviderStatuses = [];
function onboardingBadge(id, text, ready, optional = false) {
const el = $(id);
el.textContent = text;
Expand All @@ -6998,6 +6999,15 @@ <h2 id="governance-review-title">Confirm governance change</h2>
""
);
}
function onboardingModelStatus(modelId) {
const provider = onboardingProviderForModel(modelId);
const managed = onboardingModelStatuses.find((item) => item.provider === provider);
if (managed?.configured) return managed;
const custom = onboardingCustomProviderStatuses.find(
(item) => item.id === provider && !item.disabled && item.hasKey,
);
return custom ? { provider, configured: true, source: "custom" } : managed;
}
function renderOnboardingProviderOptions(preferred) {
const select = $("onboarding-model-provider");
select.textContent = "";
Expand Down Expand Up @@ -7043,13 +7053,14 @@ <h2 id="governance-review-title">Confirm governance change</h2>
return;
}
onboardingModelStatuses = models.data.providers || [];
onboardingCustomProviderStatuses = models.data.customProviders || [];
onboardingModels = {};
(models.data.models || []).forEach((model) => {
(onboardingModels[model.provider] ||= []).push(model);
});
const baseModel = config.data.baseModel || config.data.baseModelDefault || "";
const baseProvider = onboardingProviderForModel(baseModel);
const baseStatus = onboardingModelStatuses.find((item) => item.provider === baseProvider);
const baseStatus = onboardingModelStatus(baseModel);
onboardingBadge(
"onboarding-model-badge",
baseStatus?.configured ? "Ready" : "Needs a key",
Expand All @@ -7058,7 +7069,13 @@ <h2 id="governance-review-title">Confirm governance change</h2>
$("onboarding-model-summary").textContent = !baseModel
? "No base model is configured yet — pick a provider and model below."
: baseStatus?.configured
? baseModel + " · " + (baseStatus.source === "admin" ? "admin-managed key" : "deployment key")
? baseModel +
" · " +
(baseStatus.source === "admin"
? "admin-managed key"
: baseStatus.source === "custom"
? "custom provider key"
: "deployment key")
: baseModel +
" cannot run until its " +
(MODEL_PROVIDER_LABELS[baseProvider] || connectorName(baseProvider)) +
Expand Down Expand Up @@ -7087,6 +7104,7 @@ <h2 id="governance-review-title">Confirm governance change</h2>
$("onboarding-oauth-summary").textContent = configured.length
? configured.map((item) => connectorName(item.provider)).join(", ") + " available in the web UI."
: "Add Google, GitHub, Notion, or other OAuth clients to make those connectors available.";
await loadCustomProviders();
viewLoadedAt.onboarding = Date.now();
}
$("onboarding-model-provider").onchange = () => renderOnboardingModelOptions();
Expand All @@ -7110,7 +7128,6 @@ <h2 id="governance-review-title">Confirm governance change</h2>
$("onboarding-model-save").disabled = false;
$("onboarding-model-key").value = "";
await loadOnboarding();
await loadCustomProviders();
setStatus(
"st-onboarding-model",
selected.ok ? "Key and base model saved." : "Key saved, but the base model could not be changed.",
Expand All @@ -7132,7 +7149,6 @@ <h2 id="governance-review-title">Confirm governance change</h2>
return;
}
await loadOnboarding();
await loadCustomProviders();
setStatus("st-onboarding-model", "Provider disabled.", "ok");
};
let customProvidersLoaded = [];
Expand Down Expand Up @@ -7196,7 +7212,7 @@ <h2 id="governance-review-title">Confirm governance change</h2>
setStatus("st-custom-provider", removed.data?.message || "Could not remove this provider.", "err");
return;
}
await loadCustomProviders();
await loadOnboarding();
setStatus("st-custom-provider", "Provider removed.", "ok");
};
actions.appendChild(edit);
Expand Down Expand Up @@ -7232,7 +7248,7 @@ <h2 id="governance-review-title">Confirm governance change</h2>
return;
}
$("custom-provider-key").value = "";
await loadCustomProviders();
await loadOnboarding();
setStatus("st-custom-provider", "Provider saved. Its models are now in the picker.", "ok");
};
function openOnboardingTarget(target) {
Expand Down
29 changes: 29 additions & 0 deletions plugins/admin/test/onboarding-view.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,32 @@ test("?view=onboarding resolves to the onboarding view", () => {
test("unknown views still fall back to the default view", () => {
assert.equal(resolveView("/admin/no-such-view", ""), "history");
});

test("a keyed custom provider makes its base model ready", () => {
const src = [
slice("function onboardingProviderForModel(modelId) {", "function renderOnboardingProviderOptions"),
"onboardingModelStatus('acme-large');",
].join("\n");
const context = vm.createContext({
onboardingModels: { "acme-gateway": [{ id: "acme-large", name: "Acme Large" }] },
onboardingModelStatuses: [],
onboardingCustomProviderStatuses: [{ id: "acme-gateway", disabled: false, hasKey: true }],
});
assert.deepEqual(JSON.parse(JSON.stringify(vm.runInContext(src, context))), {
provider: "acme-gateway",
configured: true,
source: "custom",
});
});

test("onboarding loads the custom provider table on entry", () => {
assert.match(
slice("async function loadOnboarding() {", '$("onboarding-model-provider").onchange'),
/await loadCustomProviders\(\)/,
);
});

test("custom provider mutations refresh onboarding readiness", () => {
const handlers = slice("async function loadCustomProviders() {", "function openOnboardingTarget");
assert.equal(handlers.match(/await loadOnboarding\(\)/g)?.length, 2);
});
5 changes: 4 additions & 1 deletion plugins/web-ui/src/core-bridge.ts
Original file line number Diff line number Diff line change
Expand Up @@ -473,7 +473,10 @@ export interface RuntimeConfig {
scopeId: string;
approvedHarnesses: string[];
modelsByHarness: Record<string, string[]>;
modelCatalog: Record<string, { name: string; provider: string }>;
modelCatalog: Record<
string,
{ name: string; provider: string; api?: Api; contextWindow?: number; maxTokens?: number }
>;
orgDefault: { harnessId: string; modelId: string; effortLevel?: string; fastMode?: boolean; revision: number };
scopeOverride: {
harnessId: string;
Expand Down
16 changes: 12 additions & 4 deletions plugins/web-ui/src/model-options.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,14 @@ interface ModelMeta {
buttonLabel: string;
}

interface CatalogModel {
name: string;
provider: string;
api?: Api;
contextWindow?: number;
maxTokens?: number;
}

const MODEL_CATALOG: Record<string, ModelMeta> = {
"claude-opus-5": {
label: "Opus 5",
Expand Down Expand Up @@ -78,7 +86,7 @@ function buildOption(
id: string,
harnessId = "pi",
qualified = false,
catalog: Readonly<Record<string, { name: string; provider: string }>> = {},
catalog: Readonly<Record<string, CatalogModel>> = {},
): ModelOption | null {
try {
const dynamic = catalog[id];
Expand All @@ -101,7 +109,7 @@ function buildOptions(
ids: readonly string[],
harnessId = "pi",
qualified = false,
catalog: Readonly<Record<string, { name: string; provider: string }>> = {},
catalog: Readonly<Record<string, CatalogModel>> = {},
): ModelOption[] {
const seen = new Set<string>();
const out: ModelOption[] = [];
Expand Down Expand Up @@ -154,7 +162,7 @@ export function applyPickerModelIds(ids: readonly string[] | null | undefined, b
export function runtimeModelOptions(
approvedHarnesses: readonly string[],
modelsByHarness: Readonly<Record<string, readonly string[]>>,
catalog: Readonly<Record<string, { name: string; provider: string }>> = {},
catalog: Readonly<Record<string, CatalogModel>> = {},
): ModelOption[] {
const options = approvedHarnesses.flatMap((harnessId) => {
const configured = buildOptions(modelsByHarness[harnessId] ?? [], harnessId, true, catalog);
Expand All @@ -170,7 +178,7 @@ export function applyRuntimeOptions(
approvedHarnesses: readonly string[],
modelsByHarness: Readonly<Record<string, readonly string[]>>,
effective: { harnessId: string; modelId: string },
catalog: Readonly<Record<string, { name: string; provider: string }>> = {},
catalog: Readonly<Record<string, CatalogModel>> = {},
): void {
const options = runtimeModelOptions(approvedHarnesses, modelsByHarness, catalog);
const applied = { options, defaultValue: `${effective.harnessId}:${effective.modelId}` };
Expand Down
21 changes: 20 additions & 1 deletion plugins/web-ui/src/pi-models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@ const CLONE_TEMPLATES: Readonly<Record<string, { template: string; name: string
};

type PiModel = Model<Api>;
type DynamicModel = {
name: string;
provider: string;
api?: Api;
contextWindow?: number;
maxTokens?: number;
};

function builtinModel(id: string): PiModel | undefined {
for (const provider of KNOWN_PROVIDERS) {
Expand All @@ -22,7 +29,7 @@ function builtinModel(id: string): PiModel | undefined {
return undefined;
}

export function getBaseModel(id: string, fallback?: { name: string; provider: string }): PiModel {
export function getBaseModel(id: string, fallback?: DynamicModel): PiModel {
const builtin = builtinModel(id);
if (builtin) return builtin;
const clone = CLONE_TEMPLATES[id];
Expand All @@ -34,6 +41,18 @@ export function getBaseModel(id: string, fallback?: { name: string; provider: st
const template = getModel("openrouter", "openrouter/auto" as Parameters<typeof getModel>[1]) as PiModel | undefined;
if (template) return cloneModel(template, id, fallback.name);
}
if (fallback?.api) {
const templateId = fallback.api === "anthropic-messages" ? "claude-opus-4-8" : "gpt-5.5";
const template = builtinModel(templateId);
if (template)
return {
...cloneModel(template, id, fallback.name),
provider: fallback.provider,
api: fallback.api,
...(fallback.contextWindow ? { contextWindow: fallback.contextWindow } : {}),
...(fallback.maxTokens ? { maxTokens: fallback.maxTokens } : {}),
};
}
throw new Error(`Unsupported model: ${id}`);
}

Expand Down
22 changes: 22 additions & 0 deletions plugins/web-ui/test/model-options.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,28 @@ test("runtime options preserve a fetched OpenRouter model as the selected web tu
assert.equal(defaultModelValue(), "pi:anthropic/claude-sonnet-4.5");
});

test("runtime options render and select a custom provider model", () => {
applyRuntimeOptions(
null,
["pi"],
{ pi: ["acme-large"] },
{ harnessId: "pi", modelId: "acme-large" },
{
"acme-large": {
name: "Acme Large",
provider: "acme-gateway",
api: "openai-completions",
contextWindow: 128000,
maxTokens: 8192,
},
},
);
const option = getModelOptions()[0]!;
assert.equal(option.value, "pi:acme-large");
assert.equal(option.model.provider, "acme-gateway");
assert.equal(defaultModelValue(), "pi:acme-large");
});

test("runtime options hide retired persisted model ids", () => {
applyRuntimeOptions(
null,
Expand Down
16 changes: 16 additions & 0 deletions plugins/web-ui/test/pi-models.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,22 @@ test("models this pi-ai build lacks are cloned from a template of their own prov
assert.equal(sol.provider, "openai", "an OpenAI model never resolves through an Anthropic template");
});

test("custom provider models are materialized from protocol metadata", () => {
const model = getBaseModel("acme-large", {
name: "Acme Large",
provider: "acme-gateway",
api: "openai-completions",
contextWindow: 128000,
maxTokens: 8192,
});
assert.equal(model.id, "acme-large");
assert.equal(model.name, "Acme Large");
assert.equal(model.provider, "acme-gateway");
assert.equal(model.api, "openai-completions");
assert.equal(model.contextWindow, 128000);
assert.equal(model.maxTokens, 8192);
});

test("fast-mode support is fed from core's runtime config, not a hardcoded client copy", () => {
setFastModeModelIds(null, []);
assert.equal(modelSupportsFastMode(null, "claude-opus-4-8"), false);
Expand Down
14 changes: 7 additions & 7 deletions src/api/app-turn.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import {
modelProviderAvailabilityFor,
modelServiceable,
} from "../model/pi-models.ts";
import { selectableCatalogForHarness, selectableModelCatalog } from "../model/model-catalog.ts";
import { builtInModelCatalog, selectableCatalogForHarness, selectableModelCatalog } from "../model/model-catalog.ts";
import { resolveRuntimeChoiceDurable } from "../harness/harness-router.ts";
import { errMessage } from "../util/errors.ts";

Expand Down Expand Up @@ -170,16 +170,16 @@ export function createTurnMethods(
};
}
const configuredWebuiModels = await deps.config.getWebuiModelsDurable(org);
let enabledWebuiModels: string[] | null = null;
let enabledWebuiModels: string[];
if (configuredWebuiModels?.length) {
enabledWebuiModels = [...new Set([...configuredWebuiModels, orgRuntime.modelId])];
} else if (providers?.openrouter) {
} else {
const catalog = providers?.openrouter
? await selectableModelCatalog(deps.modelCredentialFetch)
: builtInModelCatalog();
enabledWebuiModels = [
...new Set([
...selectableCatalogForHarness(
await selectableModelCatalog(deps.modelCredentialFetch),
runtime.harnessId,
).map((model) => model.id),
...selectableCatalogForHarness(catalog, runtime.harnessId).map((model) => model.id),
...(orgRuntime.harnessId === runtime.harnessId ? [orgRuntime.modelId] : []),
]),
];
Expand Down
10 changes: 8 additions & 2 deletions src/api/routes/admin/model-providers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,15 @@ export async function getModelProviders(ctx: ApiCtx): Promise<void> {
resource: "model-providers",
scopeLabel: orgScope(ctx.deps),
});
const [providers, models, customProviders] = await Promise.all([
ctx.deps.modelCredentials.statuses(),
selectableModelCatalog(ctx.deps.modelCredentialFetch),
ctx.deps.customProviders?.statuses() ?? [],
]);
return sendJson(ctx.res, 200, {
providers: await ctx.deps.modelCredentials.statuses(),
models: await selectableModelCatalog(ctx.deps.modelCredentialFetch),
providers,
models,
...(ctx.deps.customProviders ? { customProviders } : {}),
});
}

Expand Down
Loading