provider_test_report.md

ExLLM Provider Test Report

Summary

Testing conducted on: 2025-06-30

Working Providers (7/11)

• ✅ OpenAI - All features working, cost tracking enabled
• ✅ Anthropic - All features working, embeddings unavailable
• ✅ Groq - Working (using deepseek-r1-distill model), vision unavailable
• ✅ X.AI (Grok) - Working with Grok-3 model, embeddings unavailable
• ✅ OpenRouter - Working with deepseek model
• ✅ Ollama - Local provider working
• ✅ Mock - Test provider working as expected

Failed Providers (4/11)

• ❌ Gemini - Error: :unexpected_pipeline_state
• ❌ Mistral - Setup error: key :name not found
• ❌ Perplexity - Provider setup issue
• ❌ LM Studio - Connection/configuration issue

Detailed Test Results

Basic Chat Test

All working providers successfully completed the "2+2" calculation test with proper responses.

Cost Tracking

• OpenAI: $0.000009 (26 tokens)
• Anthropic: $0.000103 (30 tokens)
• X.AI: $0.000490 (42 tokens)
• OpenRouter: $0.000031 (26 tokens)
• Groq: No pricing data for deepseek-r1-distill model
• Ollama/Mock: No cost (local)

Key Findings

1. Streaming Timeout Fix: The streaming timeout configuration fix from our debugging session is working correctly - no timeout issues observed during testing.

2. Provider-Specific Issues:

   • Gemini has a pipeline state error that needs investigation
   • Mistral provider has configuration/setup issues
   • Some providers like Groq are using newer models (deepseek-r1) that don't have pricing data

3. Feature Availability:

   • Vision is unavailable for Groq
   • Embeddings are unavailable for Anthropic, X.AI, and OpenRouter
   • Cost tracking works for most cloud providers

Recommendations

1. Investigate Gemini's unexpected_pipeline_state error - likely related to response parsing
2. Fix Mistral provider setup to handle missing :name key
3. Update pricing data for newer models like deepseek-r1-distill
4. Add proper error handling for provider configuration issues