Repository navigation
feat(llm): Settings picker for LLM provider + model (per-session pinning) - #55
Merged
Merged
Conversation
…ons only)
Replaces the single-LLM-at-startup pattern with a runtime picker. Users go to
Settings -> Agent config -> LLM, pick from any provider whose credentials are
present in .env, and save. The next chat session uses the new model; in-flight
chats keep their original model.
Core
- agent/llm/backend.py: resolve_model_and_key now accepts override_source and
override_model; new available_providers() returns each provider's configured
status and a curated model list for the UI dropdown.
- agent/llm/preference.py (new): load/save the user pick to app_config under
"llm_preference" (OSS) or "llm_preference_{org_id}" (product, key reserved).
- agent/core.py: drop the _agent singleton in favor of an lru_cache-backed
_build_agent(model, key_hash); resolve_agent(...) returns (agent, model_name).
init_agent(checkpointer) now just stores the checkpointer + warms the cache
with the deployment default. get_active_model() takes an optional pin.
- agent/db: upsert_session no longer overwrites sessions.model on conflict;
new get_session_model(session_id) helper lets the chat router learn the
pinned model for an existing session.
Backend
- api/routers/settings.py: GET /api/settings/llm returns providers + current
pick + active backend info. PUT /api/settings/llm validates the pick against
available_providers() before persisting. Admin-only.
- api/routers/chat.py: at chat start, prefer the session's pinned model
(db.get_session_model); for new sessions, fall back to the saved preference;
pass through to resolve_agent(). Usage card now labels the per-session model.
Frontend
- pages/SettingsPage.tsx: new LLM card on the Agent config tab — provider
dropdown (grays out providers whose .env keys are missing), model dropdown,
Save button. Toast on success/failure. Helper text clarifying "new chats only".
Tests
- test_agent/test_core.py: switched the two timeout tests from patching the
removed _agent module var to patching get_agent(). 191 pytest green.
Catalog - OpenRouter: added Claude Haiku, GPT-4-Turbo, Gemini 2.5 Flash + Pro, Gemma 3-27B, Llama 3.3 70B, DeepSeek v3, Qwen 2.5 72B, Grok 2, Mistral Large. - New "Google Gemini" provider (direct, gemini/* via GEMINI_API_KEY / GOOGLE_API_KEY): gemini-2.5-pro / -flash / -flash-lite. - Claude Code (subscription): now lists Opus/Sonnet/Haiku so the OAuth token can pick a specific Claude model — resolve_model_and_key honors the explicit model alongside the detector source. - OpenAI: added o1-mini. - Groq: added Mixtral. - Ollama: added DeepSeek R1. Settings views - get_backend_info(pref=) — when a saved LLM preference is present, the returned display reflects what NEW sessions will actually use (source, model, label "picked in Settings"). Previously it only described the .env defaults, so the Environment tab + LLM card showed stale info after a user switched models in the UI. - settings router + init/status router pass the loaded preference through to get_backend_info(), so both surfaces update immediately on save. Tests: 191 pytest green.
Detector - _MODEL_SLOTS: add claude-opus-4-8 (latest); short alias "opus" now points there. Reordered by tier (newest first). - _DEFAULT_MODEL flipped from sonnet-4-6 to opus-4-8 — when ~/.claude/ settings.json doesn't name a model, default to the latest Opus rather than mid-tier Sonnet. Picker catalog - Claude Code (subscription): 7 entries spanning Opus 4.8 / 4.7 / 4.5, Sonnet 4.6 / 4.5, Haiku 4.5, and 3.5 Sonnet (legacy fallback). - Anthropic API: same 7 models, billed via ANTHROPIC_API_KEY instead of OAuth — same effective model lineup, different auth path. - OpenRouter: added openrouter/anthropic/claude-opus-4-8 at the top of the Anthropic block. Tests - test_model_slot_mapping: bump expected values for the "opus" alias and the unknown-slot default (both now Opus 4.8).
The Claude Code branch of available_providers() was overriding the curated catalog with just the single model from ~/.claude/settings.json — so the UI dropdown only ever showed one option (whatever the user had Claude Code set to). Result: users couldn't pick Opus 4.8 / Sonnet / Haiku via the OAuth path from Settings. Now: expose the full curated catalog. If ~/.claude/settings.json names a specific model, promote it to the top of the list so it stays the default.
New apps/backend/tests/test_llm/test_picker.py — 22 focused tests on the
public contract Settings -> LLM relies on. Designed so a contributor
adding a provider or model only has to update the relevant assertion,
and a regression like the recently-fixed "Claude Code dropdown only
shows one model" lands as a named failure here.
Coverage
- available_providers()
- every catalog entry surfaces
- unconfigured providers carry a hint about which env var to add
- Ollama is configured by default (no key needed)
- Claude Code dropdown lists MULTIPLE models (pins the recent fix)
- the user's ~/.claude model is promoted to the top of the list
- CLAUDE_CODE_AUTODETECT=false grays out Claude Code with a clear note
- resolve_model_and_key(override_source, override_model)
- override_model picks the right provider's env-var key
(anthropic / openrouter / gemini via GOOGLE_API_KEY)
- override_source='claude_code' uses the OAuth token + detector model
- override_source + override_model combines OAuth token with chosen model
- disabled detector raises a clear RuntimeError (no silent bypass)
- no overrides + no env => falls to settings.llm_model
- get_backend_info(pref=...)
- pref with source labels "picked in Settings"
- pref with model only infers the provider correctly
- pref=None preserves old env-only behavior
- _provider_of() classification for every catalog provider
Everything mocked: no real ~/.claude, no real env vars. Fast (<1s).
…v tab Custom model option - Settings -> Agent config -> LLM: each provider's model dropdown gains a "Custom — type a model name…" option at the bottom. Selecting it swaps the select for a free-text input prefilled with a provider-prefixed placeholder (e.g. "anthropic/claude-…", "openrouter/your-model-name"). - A small "← back to curated list" button restores the dropdown. - A saved model that isn't in the provider's curated list auto-enters custom mode on page load so users can see / edit what's actually saved. Backend already accepts any model string when the provider is configured, so this is a UI-only change. LiteLLM surfaces invalid model strings at chat time (documented behavior — we can't pre-validate without a provider call). Use cases - A new release lands (Claude 5, Gemini 3, …) before we update the catalog. - A niche OpenRouter variant we didn't curate. - A locally-pulled Ollama tag with a custom name. Environment tab cleanup - Drop the LLM_MODEL row. With the picker overriding the .env default per session, surfacing the raw env var was actively misleading: users would see "openrouter/gemma" while the agent was actually calling "claude-opus-4-8". The Agent config LLM card is now the single source of truth for "what model am I using?".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_agentsingleton with anlru_cache-backed_build_agent(model, key_hash)+resolve_agent(override_source, override_model). Per-request model selection becomes a 2-line lookup; same(model, key)reuses the compiled LangGraph.GET/PUT /api/settings/llmendpoints (admin-only) backed byapp_config["llm_preference"]. UI shows a card on Settings → Agent config with provider + model dropdowns; unconfigured providers are grayed out with a hint about which.envkey to add.sessions.modelis now preserved onupsert_sessionconflict, so existing chats keep the LLM they started with. Newdb.get_session_model(session_id)helper lets the chat router look up the pinned model.CLAUDE_CODE_AUTODETECTis the single kill switch for Claude Code in both the auto-fallback chain and the Settings picker (flipping tofalsegrays out the option in the UI).gemini/*viaGEMINI_API_KEY/GOOGLE_API_KEY), broader OpenRouter selection (Gemma, Gemini, Llama, DeepSeek, Qwen, Grok, Mistral).get_backend_info(pref=...)now reflects the saved pick — Environment tab + LLM card show what new sessions will actually use, not the stale.envdefault.LLM_MODELrow dropped from the Environment tab (was misleading after the picker overrides it).Test plan
cd apps/backend && uv run pytest— 213 passing (191 pre-existing + 22 new intest_llm/test_picker.py)sessions.modelon-conflict change applies equally; tested implicitly via the samesave_turnpath)app_config["llm_preference_{org_id}"])Notes for reviewers
create_deep_agentcompile cost twice per (model, key).enhancement: model allowed-list) for a future admin-policy gate.