Merge pull request webbrain-one#2731 from esokullu/main

Detect vision support and toggle for local model providers

Author: esokullu

.github/workflows/webbrain-cloud-smoke.yml

@@ -2,18 +2,6 @@ name: WebBrain Cloud Smoke
 
 on:
   workflow_dispatch:
-  schedule:
-    - cron: "17 5 * * *"
-  push:
-    branches: [main]
-    paths:
-      - ".github/workflows/webbrain-cloud-smoke.yml"
-      - "ci/**"
-      - "src/chrome/src/agent/**"
-      - "src/chrome/src/background.js"
-      - "src/chrome/src/offscreen/cloud-bridge.js"
-      - "src/chrome/src/cloud-runs.js"
-      - "src/firefox/src/agent/**"
 
 permissions:
   contents: read

CHANGELOG.md

@@ -8,6 +8,7 @@ This changelog was generated from the repository Git history and release tags. V
 
 ### Changed
 - Updated the LM Studio plugin for the current browser-delegation protocol.
+- Added model-bound vision capability detection for llama.cpp, LM Studio, and LocalAI, with Auto / Force on / Off settings and fail-closed Chrome/Firefox request routing.
 
 ### Fixed
 - Hid empty assistant placeholders until response content is ready to render.

README.md

@@ -90,10 +90,11 @@ LM Studio (`:1234/v1`), Jan (`:1337/v1`), and LocalAI (`:8080/v1`) work the same
 way. Load a model with **at least a 16k-token context window** — 8k works only
 with the Compact tier, and 4k is too small for the system prompt plus tool
 schemas. WebBrain auto-detects the real window for llama.cpp, Ollama, and LM
-Studio, and auto-compacts the conversation as it fills up. For Ollama, the
-Vision setting defaults to Auto and reads the selected model's native
-`/api/show` capabilities; Force on and Off remain available as explicit
-overrides. There is also a
+Studio, and auto-compacts the conversation as it fills up. For Ollama,
+llama.cpp, LM Studio, and LocalAI, it also reads native server metadata before
+adding screenshots; Settings provides Auto, Force on, and Off overrides. When
+the optional Model field is blank, the loaded-model capability is rechecked on
+every user turn so a server-side hot swap takes effect. There is also a
 preview `ollama launch webbrain --model <model>` handoff. Details:
 [providers and models](docs/providers-and-models.md#local-providers).
 

docs/architecture.md

@@ -160,10 +160,6 @@ background.js handleMessage('chat_start')
 
 ### Step 3: Enrich First User Message
 ```
-providerManager.prepareActiveProviderCapabilities()
-  → for Ollama Auto, resolve the model/base-URL-scoped `/api/show` check
-  → coalesce concurrent checks and fail closed to text-only on metadata errors
-
 _enrichUserMessageWithCurrentPage(tabId, messages, userMessage)
 
   1. Collect URL + title via chrome.tabs.get(tabId)
@@ -709,11 +705,13 @@ testConnection()              → { ok, error, model }
 
 `promptTier` drives both the action prompt and the normal tool subset. Local providers default to Mid, cloud providers are forced Full, and the legacy `useCompactPrompt` flag maps to Compact for existing configs. Dev mode is a separate conversation mode: Mid/Full Dev uses the selected Act tier plus `SYSTEM_PROMPT_DEV_APPENDIX`; Compact Dev is blocked before an LLM request is sent.
 
-Ollama is the exception to OpenAI-compatible model-name vision inference. Its
-`visionMode` is `auto`, `on`, or `off`; Auto performs one coalesced,
-three-second `/api/show` metadata check per model/base URL and service-worker
-lifetime before turn enrichment. Only successful metadata is persisted, and a
-late result is applied only if the same Ollama identity is still configured.
+Ollama, llama.cpp, LM Studio, and LocalAI resolve `supportsVision` from native
+server metadata before page enrichment. Explicit model/base-URL identities are
+cached and protected by stale-result guards; an empty Model field is treated as
+the server's mutable loaded-model slot, so concurrent checks are coalesced only
+within that turn and the next user turn rechecks it. Detection is bounded to
+three seconds and fails closed without failing the text request. User overrides
+bypass detection. Chrome and Firefox share the same parsers and behavior.
 
 See `docs/providers-and-models.md`.
 

docs/fr/providers-and-models.md

@@ -37,13 +37,13 @@ class BaseLLMProvider {
 | ID Fournisseur | Type | Catégorie | Modèle par défaut | Vision |
 |---|---|---|---|---|
 | `webbrain_cloud` | `openai` | cloud | `webbrain-cloud 1.0` | Oui |
-| `llamacpp` | `llamacpp` | local | (modèle chargé) | Oui (activé par défaut) |
-| `ollama` | `openai` | local | (modèle chargé) | Auto via `/api/show` |
-| `lmstudio` | `openai` | local | (modèle chargé) | Oui (activé par défaut) |
+| `llamacpp` | `llamacpp` | local | (modèle chargé) | Métadonnées auto / surcharge |
+| `ollama` | `openai` | local | (modèle chargé) | Auto via `/api/show` / surcharge |
+| `lmstudio` | `openai` | local | (modèle chargé) | Métadonnées auto / surcharge |
 | `jan` | `openai` | local | (modèle chargé) | Oui (activé par défaut) |
 | `vllm` | `openai` | local | (modèle chargé) | Oui (activé par défaut) |
 | `sglang` | `openai` | local | (modèle chargé) | Oui (activé par défaut) |
-| `localai` | `openai` | local | (modèle chargé) | Oui (activé par défaut) |
+| `localai` | `openai` | local | (modèle chargé) | Métadonnées auto / surcharge |
 | `gpt4all` | `openai` | local | (modèle chargé) | Oui (activé par défaut) |
 | `azure_openai` | `azure_openai` | cloud | (déploiement) | Bascule manuelle |
 | `aws_bedrock` | `aws_bedrock` | cloud | (ID de modèle) | Non |
@@ -125,10 +125,15 @@ serveur local a été démarré avec authentification :
 - **SGLang** : `http://localhost:30000/v1` — le serveur compatible OpenAI de SGLang
 - **LocalAI** : `http://localhost:8080/v1` — le serveur compatible OpenAI de LocalAI
 
-Ollama utilise par défaut `visionMode: "auto"` et vérifie le modèle sélectionné
-avec les métadonnées natives de `/api/show` avant de joindre des images. Les
-autres fournisseurs locaux conservent leur comportement explicite
-`supportsVision` existant.
+Ollama, llama.cpp, LM Studio et LocalAI utilisent `visionMode: auto` par défaut.
+WebBrain lit les métadonnées natives du modèle sélectionné avant l'enrichissement et
+n'envoie des captures que si le serveur déclare explicitement l'entrée image.
+Une détection indisponible ou malformée reste en texte seul pour ce tour et
+sera retentée plus tard. Si le champ Modèle est vide, la capacité du modèle
+chargé est revérifiée à chaque tour afin de suivre les changements côté serveur.
+Les réglages proposent Automatique, Forcer
+l'activation et Désactivé ; les autres fournisseurs locaux conservent leur
+interrupteur explicite actuel.
 
 #### Relais de lancement Ollama (préversion)
 
@@ -180,9 +185,16 @@ Le mode Ask ignore le niveau du fournisseur et reste en lecture seule. Le mode A
 |---|---|
 | Compatible OpenAI | Regex sur le nom du modèle (`gpt-4o`, `gpt-5`, `claude-3`, `claude-sonnet-4`, `gemini-2.0-flash`, etc.) |
 | Anthropic | Patterns `claude-(3\|sonnet-4\|opus-4)` |
-| llama.cpp | Interrupteur explicite `supportsVision` dans la configuration |
-| Ollama | Champ `capabilities` de `POST /api/show`, avec repli historique sur `projector_info` / les métadonnées `.vision.` ; réglage Auto / Forcer l'activation / Désactivé |
-| LM Studio / Jan / vLLM / SGLang / LocalAI | Interrupteur explicite `supportsVision` dans la configuration (via le fournisseur OpenAI) |
+| Ollama | `POST /api/show` `capabilities`, avec replis historiques `projector_info` / `.vision.` |
+| llama.cpp | `GET /props` → `modalities.vision`, avec Automatique / Forcer / Désactivé |
+| LM Studio | `GET /api/v1/models` → `capabilities.vision`, puis ancien `/api/v0/models` `type` |
+| LocalAI | `GET /v1/models/capabilities` → `input_modalities` / `capabilities` |
+| Jan / vLLM / SGLang | Interrupteur explicite `supportsVision` dans la configuration (via le fournisseur OpenAI) |
+
+La détection est liée au fournisseur, au modèle exact et à l'URL de base. Les
+requêtes simultanées sont regroupées et une réponse tardive d'une ancienne
+configuration ne peut pas modifier la configuration actuelle. Un fournisseur
+de vision dédié conserve le routage séparé existant.
 
 ### Conversion Anthropic
 
@@ -286,11 +298,4 @@ myprovider: {
 },
 ```
 
-La vision est normalement auto-détectée via une regex sur le nom du modèle.
-Ollama fait exception : son mode Auto utilise les métadonnées natives de
-`/api/show` et échoue de manière fermée vers le mode texte seul si elles ne
-peuvent pas être vérifiées. Si un autre fournisseur a un ensemble connu de
-modèles de vision, ajoutez-les à la regex dans `openai.js`. Définissez
-`supportsStreamUsageOptions: true` uniquement pour les fournisseurs qui
-acceptent `stream_options.include_usage` de style OpenAI ; laissez-le à false
-lorsqu'un fournisseur retourne l'utilisation sans accepter ce champ de requête.
+La vision est auto-détectée via une regex sur le nom du modèle. Si le fournisseur a un ensemble connu de modèles de vision, ajoutez-les à la regex dans `openai.js`. Définissez `supportsStreamUsageOptions: true` uniquement pour les fournisseurs qui acceptent `stream_options.include_usage` de style OpenAI ; laissez-le à false lorsqu'un fournisseur retourne l'utilisation sans accepter ce champ de requête.

docs/providers-and-models.md

@@ -37,13 +37,13 @@ class BaseLLMProvider {
 | Provider ID | Type | Category | Default Model | Vision |
 |---|---|---|---|---|
 | `webbrain_cloud` | `openai` | cloud | `webbrain-cloud 1.0` | Yes |
-| `llamacpp` | `llamacpp` | local | (loaded model) | Yes (default on) |
-| `ollama` | `openai` | local | (loaded model) | Auto via `/api/show` |
-| `lmstudio` | `openai` | local | (loaded model) | Yes (default on) |
+| `llamacpp` | `llamacpp` | local | (loaded model) | Auto metadata / override |
+| `ollama` | `openai` | local | (loaded model) | Auto via `/api/show` / override |
+| `lmstudio` | `openai` | local | (loaded model) | Auto metadata / override |
 | `jan` | `openai` | local | (loaded model) | Yes (default on) |
 | `vllm` | `openai` | local | (loaded model) | Yes (default on) |
 | `sglang` | `openai` | local | (loaded model) | Yes (default on) |
-| `localai` | `openai` | local | (loaded model) | Yes (default on) |
+| `localai` | `openai` | local | (loaded model) | Auto metadata / override |
 | `gpt4all` | `openai` | local | (loaded model) | Yes (default on) |
 | `azure_openai` | `azure_openai` | cloud | (deployment) | Manual toggle |
 | `aws_bedrock` | `aws_bedrock` | cloud | (model id) | No |
@@ -155,9 +155,15 @@ local server was started with auth:
 - **SGLang**: `http://localhost:30000/v1` — SGLang's OpenAI-compatible server
 - **LocalAI**: `http://localhost:8080/v1` — LocalAI's OpenAI-compatible server
 
-Ollama defaults to `visionMode: "auto"` and verifies the selected model through
-its native `/api/show` metadata before attaching images. The other local
-providers retain their existing explicit `supportsVision` behavior.
+Ollama, llama.cpp, LM Studio, and LocalAI default to `visionMode: auto`. WebBrain asks
+the selected server for model capability metadata before enrichment and sends
+screenshots only when the response explicitly reports image input. A failed or
+malformed metadata request is text-only for that turn and is retried later;
+Settings can override Auto with Force on or Off. For providers whose Model
+field may be blank, WebBrain coalesces concurrent checks but rechecks once per
+user turn, so changing the model loaded by the server cannot reuse a stale
+answer. Other local providers retain
+their existing explicit `supportsVision` setting.
 
 #### Ollama launch handoff (preview)
 
@@ -218,9 +224,16 @@ Ask mode ignores provider tier and stays read-only. Act mode uses the selected t
 |---|---|
 | OpenAI-compatible | Regex against model name (`gpt-4o`, `gpt-5`, `claude-3`, `claude-sonnet-4`, `gemini-2.0-flash`, etc.) |
 | Anthropic | `claude-(3\|sonnet-4\|opus-4)` patterns |
-| llama.cpp | Explicit `supportsVision` config toggle |
-| Ollama | `POST /api/show` `capabilities`, with legacy projector / `.vision.` metadata fallbacks; Auto / Force on / Off setting |
-| LM Studio / Jan / vLLM / SGLang / LocalAI | Explicit `supportsVision` config toggle (via OpenAI provider) |
+| Ollama | `POST /api/show` `capabilities`, with legacy projector / `.vision.` metadata fallbacks; Auto / Force on / Off |
+| llama.cpp | `GET /props` → `modalities.vision`, with Auto / Force on / Off |
+| LM Studio | `GET /api/v1/models` → `capabilities.vision`; legacy `/api/v0/models` `type`, with overrides |
+| LocalAI | `GET /v1/models/capabilities` → `input_modalities` / `capabilities`, with overrides |
+| Jan / vLLM / SGLang | Explicit `supportsVision` config toggle (via OpenAI provider) |
+
+Auto results are keyed by provider, exact selected model, and canonical base
+URL. Concurrent checks share one request, and a late response from an older
+configuration cannot change the current provider. A separately configured
+dedicated vision provider continues to use the existing split-provider path.
 
 ### Anthropic Conversion
 
@@ -331,10 +344,4 @@ myprovider: {
 },
 ```
 
-Vision is normally auto-detected via model-name regex. Ollama is the exception:
-its Auto mode uses native `/api/show` metadata and fails closed to text-only
-when metadata cannot be verified. If another provider has a known set of vision
-models, add them to the regex in `openai.js`. Set
-`supportsStreamUsageOptions: true` only for providers that accept OpenAI-style
-`stream_options.include_usage`; leave it false when a provider returns usage
-without accepting that request field.
+Vision is auto-detected via model-name regex. If the provider has a known set of vision models, add them to the regex in `openai.js`. Set `supportsStreamUsageOptions: true` only for providers that accept OpenAI-style `stream_options.include_usage`; leave it false when a provider returns usage without accepting that request field.

docs/zh-CN/providers-and-models.md

@@ -37,13 +37,13 @@ class BaseLLMProvider {
 | 提供商 ID | 类型 | 类别 | 默认模型 | 视觉能力 |
 |---|---|---|---|---|
 | `webbrain_cloud` | `openai` | 云端 | `webbrain-cloud 1.0` | 是 |
-| `llamacpp` | `llamacpp` | 本地 | （已加载模型） | 是（默认开启） |
-| `ollama` | `openai` | 本地 | （已加载模型） | 通过 `/api/show` 自动检测 |
-| `lmstudio` | `openai` | 本地 | （已加载模型） | 是（默认开启） |
+| `llamacpp` | `llamacpp` | 本地 | （已加载模型） | 自动元数据 / 覆盖 |
+| `ollama` | `openai` | 本地 | （已加载模型） | 通过 `/api/show` 自动检测 / 覆盖 |
+| `lmstudio` | `openai` | 本地 | （已加载模型） | 自动元数据 / 覆盖 |
 | `jan` | `openai` | 本地 | （已加载模型） | 是（默认开启） |
 | `vllm` | `openai` | 本地 | （已加载模型） | 是（默认开启） |
 | `sglang` | `openai` | 本地 | （已加载模型） | 是（默认开启） |
-| `localai` | `openai` | 本地 | （已加载模型） | 是（默认开启） |
+| `localai` | `openai` | 本地 | （已加载模型） | 自动元数据 / 覆盖 |
 | `gpt4all` | `openai` | 本地 | （已加载模型） | 是（默认开启） |
 | `azure_openai` | `azure_openai` | 云端 | （部署） | 手动开关 |
 | `aws_bedrock` | `aws_bedrock` | 云端 | （模型 ID） | 否 |
@@ -118,9 +118,11 @@ WebBrain 会直接记录；若服务省略用量，则记录基于字符数的
 - **SGLang**：`http://localhost:30000/v1` — SGLang 的 OpenAI 兼容服务器
 - **LocalAI**：`http://localhost:8080/v1` — LocalAI 的 OpenAI 兼容服务器
 
-Ollama 默认使用 `visionMode: "auto"`，并在附加图片前通过原生
-`/api/show` 元数据验证所选模型。其他本地提供商保持现有的显式
-`supportsVision` 行为。
+Ollama、llama.cpp、LM Studio 和 LocalAI 默认使用 `visionMode: auto`。WebBrain 在
+页面上下文增强前读取所选模型的原生服务器元数据，只有服务器明确报告支持图像输入时才
+发送截图。元数据请求失败或格式错误时，本回合按纯文本处理，之后会重试。设置中
+可选择自动、强制开启或关闭。模型字段为空时，每个用户回合都会重新检测当前加载
+模型的能力，以便服务端热切换立即生效；其他本地提供商保持现有的显式开关行为。
 
 #### Ollama 启动交接（预览）
 
@@ -170,9 +172,14 @@ Ask 模式忽略提供商层级，保持只读。Act 模式使用所选层级的
 |---|---|
 | OpenAI 兼容 | 根据模型名称进行正则匹配（`gpt-4o`、`gpt-5`、`claude-3`、`claude-sonnet-4`、`gemini-2.0-flash` 等） |
 | Anthropic | `claude-(3\|sonnet-4\|opus-4)` 模式 |
-| llama.cpp | 显式 `supportsVision` 配置开关 |
-| Ollama | `POST /api/show` 的 `capabilities` 字段，并兼容旧版 `projector_info` / `.vision.` 元数据；提供自动 / 强制开启 / 关闭设置 |
-| LM Studio / Jan / vLLM / SGLang / LocalAI | 显式 `supportsVision` 配置开关（通过 OpenAI 提供商） |
+| Ollama | `POST /api/show` 的 `capabilities`，并兼容旧版 `projector_info` / `.vision.` 元数据 |
+| llama.cpp | `GET /props` → `modalities.vision`，支持自动 / 强制开启 / 关闭 |
+| LM Studio | `GET /api/v1/models` → `capabilities.vision`；旧版本回退到 `/api/v0/models` 的 `type` |
+| LocalAI | `GET /v1/models/capabilities` → `input_modalities` / `capabilities` |
+| Jan / vLLM / SGLang | 显式 `supportsVision` 配置开关（通过 OpenAI 提供商） |
+
+检测结果按提供商、精确模型和规范化基础 URL 绑定。并发检测会合并为一次请求，旧
+配置的延迟响应不能修改当前设置。单独配置的视觉提供商继续使用现有的分流路径。
 
 ### Anthropic 转换
 
@@ -269,9 +276,4 @@ myprovider: {
 },
 ```
 
-视觉能力通常通过模型名称正则自动检测。Ollama 是例外：其自动模式使用
-原生 `/api/show` 元数据；无法验证元数据时会安全地回退为纯文本。如果其他
-提供商有已知的视觉模型集，请将它们添加到 `openai.js` 的正则表达式中。仅对
-接受 OpenAI 风格 `stream_options.include_usage` 的提供商设置
-`supportsStreamUsageOptions: true`；当提供商在不接受该请求字段的情况下返回
-使用量时，请将其保持为 false。
+视觉能力通过模型名称正则自动检测。如果提供商有已知的视觉模型集，请将它们添加到 `openai.js` 的正则表达式中。仅对接受 OpenAI 风格 `stream_options.include_usage` 的提供商设置 `supportsStreamUsageOptions: true`；当提供商在不接受该请求字段的情况下返回使用量时，请将其保持为 false。

src/chrome/README.md

@@ -77,6 +77,11 @@ ollama serve
 # LocalAI: http://localhost:8080/v1
 ```
 
+For llama.cpp, LM Studio, and LocalAI, Vision defaults to **Auto**. WebBrain
+reads the selected model's server metadata before a turn and sends screenshots
+only when image input is reported. Settings also offers **Force on** and
+**Off** overrides; metadata failures remain text-only for that turn.
+
 ### Use it
 
 Click the WebBrain icon → the side panel opens. Type a message like: