Support pluggable embedding models (voyage-code-3, local models)

[clafollett]

Intent

Codetriever is currently locked to a single embedding model (Jina v2-base-code, 768-dim) via Candle. The code embedding landscape has advanced significantly — Voyage-code-3 (now owned by Anthropic) is the current SOTA, outperforming OpenAI's text-embedding-3-large by 13.8% on 32 code retrieval benchmarks. It also supports Matryoshka dimensions (256/512/1024/2048) and int8/binary quantization, enabling storage-quality tradeoffs at scale.

Codetriever needs a pluggable embedding architecture that supports both local models (via Candle, for privacy-first deployments) and API-based models (for maximum retrieval quality without local GPU infrastructure). This is not about supporting every model — it is about supporting the right models well.

Business Rules

• The system MUST support at minimum two embedding backends: local (Candle) and API-based
• The embedding backend MUST be selectable via configuration, not code changes
• When switching embedding models, the system MUST require a full re-index — mixing embeddings from different models in the same vector collection is prohibited
• The system MUST track which embedding model was used to generate each collection's vectors
• Vector dimensions MUST be configurable to match the selected model's output dimensions
• API-based embedding backends MUST handle rate limiting, retries, and batch sizing appropriate to the provider's constraints
• API keys for cloud embedding providers MUST be provided via environment variables, never hardcoded or stored in config files
• The system SHOULD support Matryoshka dimension selection for models that offer it, allowing operators to trade storage cost for retrieval quality
• Local model inference MUST continue to support Metal (Apple Silicon), CUDA, and CPU fallback

Competitive Context

• Voyage-code-3 (Anthropic/Voyage AI): SOTA code retrieval, available via API and AWS SageMaker
• CodeSage-large-v2: Strong open-source baseline on HuggingFace
• LoRACode (ICLR 2025): LoRA rank-32 adapters that significantly boost MRR — potential for fine-tuned models
• Matryoshka embeddings are increasingly important for storage/cost tradeoffs at scale
• Anthropic's ownership of Voyage AI creates natural alignment with the MCP ecosystem

Acceptance Criteria

• Embedding backend is selectable via configuration
• At least one local model and one API-based model are supported
• Model identity is tracked per vector collection
• Dimension configuration matches the selected model
• API backends handle rate limiting and retries gracefully
• Existing local Candle inference continues to work with Metal/CUDA/CPU
• Re-indexing is enforced when the embedding model changes