Core redesign + full backend build-out: facade for ~14 vectorDBs, provider registry, setup assist

[thorwhalen]

Goal

Build vd into a complete vectorDB facade per the new report misc/docs/11 -- VectorDB Selection & Setup Guide. Three deliverables:

1. Choose — provider registry + decision framework so agents/devs can pick a backend.
2. Set up — check_requirements(backend) per deployment archetype, with "next step" guidance.
3. Facade — implement every vectorDB adapter, vendor-lock-in-free, with per-backend escape hatches.

Core redesign (decided with maintainer)

Redesign the core; refactor ef to match (ef is the only consumer, no end users yet).

• Decouple embedding from the facade core. Embedding becomes an optional convenience layer (connect(backend, embedder=None)); the core deals in vectors. Drop the hard imbed dependency. (Matches the existing .claude/CLAUDE.md §6.)
• Client becomes a Mapping[str, Collection]; Collection stays MutableMapping[str, Document] + search(); batch ops demoted to SupportsBatch.
• Shared AbstractClient / AbstractCollection bases to kill the memory/chroma duplication; backend adapters become thin.
• Per-backend filter translators (_compile_filter) over the canonical Mongo-style AST; UnsupportedFilterError wired to each backend's documented operator subset.
• Escape hatches (.client, .native) mandatory on every adapter.

Backend scope

Deep + tested here (pip-installable / embedded): chroma, faiss, sqlite-vec, duckdb-vss, lancedb, qdrant (local), memory.
Adapters written correct-by-construction (need a server/cloud account to run): pgvector, pinecone, weaviate, milvus, redis, elasticsearch, mongodb-atlas, turbopuffer.

Tracking

Dev journal — appended as work proceeds.

[thorwhalen]

Done — PR #12

The redesign + full backend build-out is complete and on PR #12 (8 commits).

Core redesign: embedding decoupled from the facade (vector-first; embedder
optional; imbed/dol deps dropped — core is stdlib + pyyaml). Client is a
Mapping, Collection a MutableMapping + search. AbstractClient /
AbstractCollection bases collapsed the per-backend duplication.

15 backends. Tested end-to-end here: memory, chroma, faiss, duckdb, lancedb,
qdrant. Correct-by-construction (need a server/account): sqlite_vec, pgvector,
pinecone, weaviate, milvus, redis, elasticsearch, mongodb, turbopuffer.

Choose: data/providers.yaml (~21-provider registry) + recommend_backend.
Set up: check_requirements / setup_guide / install_backend.

Tests: 204 pass (test_core.py parametrized over all 6 testable backends).

ef: adapted on its adapt-to-vd-0.2 branch — it dropped its dummy-embedder
workaround; 111 ef tests pass. Merge that branch only after vd 0.2 publishes.

Deferred (not in scope of this PR)

• Exercising the 9 server/managed adapters against real instances.
• SupportsHybrid implementations + client-side RRF wiring.
• AsyncClient/AsyncCollection, LangChain adapter shims, F() filter builder.

[thorwhalen]

Deferred item — live verification of the 9 server/managed adapters — done in #14 (closed #13). 7 verified against real engines (Docker: pgvector/redis/elasticsearch/weaviate/mongodb; embedded Milvus Lite for milvus; loadable-extension Python for sqlite_vec); 8 bugs fixed; all 344 contract tests pass across 13 backends. pinecone and turbopuffer remain deferred (need cloud accounts), along with SupportsHybrid+RRF, async protocols, and the LangChain shims / F() builder.