Problem
When configuring providers (especially fallback providers), there's no way to verify a provider is working without triggering an actual automation or service call. This makes it difficult to:
- Confirm API keys are valid after initial setup
- Verify a provider still works before setting it as the fallback
- Debug connectivity issues (wrong endpoint, expired key, model removed)
- Distinguish between provider config issues and automation issues
Proposed Solution
Add a "Test Connection" button to each provider's config entry in the LLM Vision settings UI.
Behavior
- User clicks "Test" on a provider entry
- Integration sends a minimal request to the provider (e.g., a simple text prompt like "Respond with OK", no images needed)
- UI shows result: success (with model name/version) or failure (with error message)
Implementation Notes
- Could use
config_flow.py's async_step mechanism with a test action
- Minimal token usage — single short text prompt, no images
- Should display the actual error on failure (429 rate limit, 401 auth, timeout, etc.)
- Could also show provider metadata: model name, endpoint, response time
Bonus: Test with image
Optional enhanced test that sends a small test image to verify the full vision pipeline works, not just text completion. This would catch issues like image encoding problems or model-specific limitations.
Use Case
Setting up a fallback provider: User has OpenRouter as primary and Google AI direct as fallback. Before saving the fallback config, they want to click "Test" on the Google provider to confirm the API key is still valid and the model responds correctly. Currently the only way to verify is to trigger an automation and check logs.
Context
Came up while configuring a fallback provider to handle OpenRouter 429 rate limits. Had to manually trigger an automation to verify the Google direct provider was still functional before trusting it as a fallback.
Problem
When configuring providers (especially fallback providers), there's no way to verify a provider is working without triggering an actual automation or service call. This makes it difficult to:
Proposed Solution
Add a "Test Connection" button to each provider's config entry in the LLM Vision settings UI.
Behavior
Implementation Notes
config_flow.py's async_step mechanism with a test actionBonus: Test with image
Optional enhanced test that sends a small test image to verify the full vision pipeline works, not just text completion. This would catch issues like image encoding problems or model-specific limitations.
Use Case
Setting up a fallback provider: User has OpenRouter as primary and Google AI direct as fallback. Before saving the fallback config, they want to click "Test" on the Google provider to confirm the API key is still valid and the model responds correctly. Currently the only way to verify is to trigger an automation and check logs.
Context
Came up while configuring a fallback provider to handle OpenRouter 429 rate limits. Had to manually trigger an automation to verify the Google direct provider was still functional before trusting it as a fallback.