Skip to content

byok.md

Latest commit

 

History

History
268 lines (195 loc) · 9.38 KB

byok.md

File metadata and controls

268 lines (195 loc) · 9.38 KB
Raw

Bring Your Own Key (BYOK)

Connect your own model providers (Ollama, Azure OpenAI, Anthropic, etc.) alongside GitHub Copilot models.

Quick Start

  1. Add a provider to config.json:
{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "models": [
        { "id": "qwen3:8b", "name": "Qwen 3 8B" }
      ]
    }
  }
}
  1. Reload config: /reload config
  2. Switch to the model: /model ollama:qwen3:8b

Provider Configuration

Each provider is a named entry under the "providers" key in config.json:

{
  "providers": {
    "<name>": {
      "type": "openai",
      "baseUrl": "https://...",
      "apiKeyEnv": "MY_API_KEY",
      "models": [
        { "id": "model-id", "name": "Display Name", "contextWindow": 32768 }
      ]
    }
  }
}

Required Fields

Field Description
baseUrl Provider's API base URL (e.g., http://localhost:11434/v1)
models Array of model entries, each with at least an id

Optional Fields

Field Description
type Provider type: openai (default), azure, anthropic
apiKey Inline API key (discouraged — use apiKeyEnv instead)
apiKeyEnv Environment variable name containing the API key
bearerToken Inline bearer token
bearerTokenEnv Environment variable name containing the bearer token
wireApi Default wire protocol for all models: "completions" (default) or "responses". Can be overridden per model.
azure Azure-specific config: { "apiVersion": "2024-10-21" }

Model Entry Fields

Field Description
id Model identifier as the provider expects it (e.g., qwen3:8b)
name Optional display name
contextWindow Context window size in tokens (used for /context display)
wireApi Override the provider's wire protocol for this model: "completions" or "responses"

Wire API & Model Compatibility

wireApi can be set at the provider level (applies to all models) or per model (overrides the provider default). This lets you mix models that need different protocols on the same endpoint.

  • "completions" (Chat Completions API) — default. Works with most models: GPT-4o, GPT-4.1, Llama, Phi, Qwen, etc.
  • "responses" (Responses API) — required for Codex models (gpt-5.x-codex-*).
  • Models must support structured function calling (OpenAI-compatible tool_calls). Models that emit tool calls as raw text (e.g., DeepSeek, some smaller/fine-tuned models) will not work correctly — the agent will output XML/JSON markup instead of executing tools.

Provider Names

Provider names must be non-empty and cannot contain : or whitespace. They're used as prefixes in model IDs (e.g., ollama:qwen3:8b), so keep them short and descriptive.

Provider Examples

Ollama (Local)

{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "models": [
        { "id": "qwen3:8b", "name": "Qwen 3 8B", "contextWindow": 32768 },
        { "id": "qwen3:14b", "name": "Qwen 3 14B" }
      ]
    }
  }
}

No authentication needed for local Ollama instances.

Azure OpenAI

{
  "providers": {
    "work-azure": {
      "type": "azure",
      "baseUrl": "https://myco.openai.azure.com",
      "apiKeyEnv": "AZURE_OPENAI_KEY",
      "azure": { "apiVersion": "2024-10-21" },
      "models": [
        { "id": "gpt-4o", "name": "GPT-4o" },
        { "id": "gpt-5.2-codex", "name": "GPT-5.2 Codex", "wireApi": "responses" }
      ]
    }
  }
}

Anthropic (Direct)

{
  "providers": {
    "anthropic": {
      "type": "anthropic",
      "baseUrl": "https://api.anthropic.com/v1",
      "apiKeyEnv": "ANTHROPIC_API_KEY",
      "models": [
        { "id": "claude-sonnet-4-20250514", "name": "Claude Sonnet 4" }
      ]
    }
  }
}

Chat Commands

Command Description
/model List all models grouped by provider
/model <provider> List models for a specific provider
/model <provider>:<model> Switch to a specific provider model
/model <bare-model> Switch model (Copilot first, then BYOK)
/provider List configured providers with details
/provider test <name> Test provider connectivity and model availability
/status Shows current model with provider prefix

Model Resolution

When you type a bare model name (e.g., /model qwen3:8b), the bridge resolves it in order:

  1. Exact match against all models (Copilot + BYOK)
  2. Copilot models first — fuzzy match against Copilot models
  3. BYOK providers — check each provider in config order for a bare ID match
  4. Global fuzzy match — fuzzy match across all models

When you include a provider prefix (e.g., /model ollama:qwen3:8b), the search is scoped to that provider's models only.

Model Switching

Switching between models on the same provider uses an in-place model swap (no session restart).

Switching between different providers (e.g., Copilot → Ollama, or Ollama → Azure) creates a fresh session because the underlying endpoint and authentication are different. Conversation history from the previous session is not carried over.

Model Fallback

BYOK models are excluded from automatic fallback chains. If a Copilot model fails, the bridge will only try other Copilot models as fallbacks — not BYOK models.

To include a BYOK model in the fallback chain, add it explicitly to fallbackModels in your channel or default config:

{
  "defaults": {
    "fallbackModels": ["claude-sonnet-4.6", "ollama:qwen3:8b"]
  }
}

Hot Reload

Provider changes are applied immediately via /reload config:

  • Added providers — available for new sessions
  • Removed providers — existing sessions keep working until recreated
  • Updated providers — new sessions use the updated config

No bridge restart is needed for provider changes.

Troubleshooting

Provider unreachable

❌ Provider "ollama" is unreachable at http://localhost:11434/v1

Check that the provider service is running and the baseUrl is correct. Use /provider test <name> to diagnose.

Authentication failure

❌ Provider "azure" rejected authentication

Verify your API key environment variable is set and the value is correct. Check with /provider to see the configured auth method.

Auth headers by provider type:

type Auth method Header sent
"openai" (default) apiKeyEnv / bearerTokenEnv Authorization: Bearer <token>
"azure" apiKeyEnv api-key: <key>
"anthropic" apiKeyEnv x-api-key: <key>

If you get 401 errors, make sure you're using the right type for your provider. Azure endpoints require type: "azure" to send the api-key header — using the default "openai" type sends a Bearer token which Azure rejects.

Model not found

❌ Model "nonexistent" not found on provider "ollama"

Check the model ID matches what the provider expects. For Ollama, model IDs include the tag (e.g., qwen3:8b, not just qwen3). Use /provider test <name> to see available remote models.

Azure 404 Not Found

This is the most common Azure issue. The SDK constructs Azure URLs using a specific pattern, and mismatches cause 404s.

How the SDK builds Azure URLs:

The SDK constructs the final URL as: {baseUrl}/openai/deployments/{model.id}/{endpoint}?api-version={apiVersion}

  • If baseUrl already contains /openai/, the SDK uses it as-is
  • If baseUrl does not contain /openai/, the SDK strips it to just the origin (scheme + host) and appends /openai
  • The model.id in your config becomes the Azure deployment name in the URL path

Common causes:

  1. Wrong baseUrl format — use just the host, no path segments:

    ✅ "baseUrl": "https://myco.openai.azure.com" ❌ "baseUrl": "https://myco.openai.azure.com/v1" ❌ "baseUrl": "https://myco.openai.azure.com/openai/deployments/gpt-4o"

  2. Deployment name mismatch — the model.id must exactly match your deployment name in Azure. If your Azure deployment is named gpt-4o-2024 but your config has "id": "gpt-4o", the SDK will request /deployments/gpt-4o/ which doesn't exist.

  3. Wrong apiVersion — Azure deployments support specific API versions. If omitted, the SDK defaults to "2024-10-21". Check your Azure portal for supported versions and set explicitly:

    "azure": { "apiVersion": "2025-04-01-preview" }

  4. Non-standard Azure gateways — some Azure AI Foundry gateways don't use the /deployments/{model}/ URL pattern. The SDK always constructs this path for type: "azure", so these gateways may need type: "openai" instead (though this changes the auth header — see Authentication above).

Tools not executing (raw XML/JSON output)

If the agent outputs raw tool call markup instead of executing tools, the model doesn't support structured function calling. Known incompatible models: DeepSeek, some smaller/fine-tuned models. Use models that support OpenAI-compatible tool_calls (GPT-4o, GPT-4.1, Llama 3.3, Phi-4, Qwen 3).

Context window showing wrong value

The SDK reports its own token limit (typically 90k–200k), not the model's actual context window. Set contextWindow on the model entry in config to override:

{ "id": "qwen3:8b", "name": "Qwen 3 8B", "contextWindow": 32768 }