Add TurboVecHybridAdapter (turbovec POC)

[dev-nid]

Context

Third and final slice of the turbovec ANN POC (3-repo coordinated change). Adds a new database adapter to @qvac/rag that uses fabric's native vector index for ANN search while keeping HyperDB / Corestore for document rows and replication. Sibling to HyperDBAdapter — HyperDBAdapter is not modified, existing users keep working unchanged.
Depends on the IdMapIndex JS class from the companion embed-llamacpp PR (sub-export @qvac/embed-llamacpp/idMapIndex).

What was done

• src/adapters/database/TurboVecHybridAdapter.js — hybrid adapter extending BaseDBAdapter. Documents stored in the existing @rag/documents table with the unchanged dbSpec (same schema as HyperDBAdapter for read-path parity). Vectors stored in a single .tvim binary file at indexPath. @rag/vectors / @rag/centroids / @rag/ivfBuckets are deliberately untouched so a Corestore created by HyperDBAdapter can be opened by TurboVecHybridAdapter without data loss.
• src/utils/stringIdToU64.js — deterministic SHA-256-prefix mapping from string doc id to uint64 (little-endian first 8 bytes). Collision math + the loud-fail behavior on duplicates documented inline.
• Export wiring: TurboVecHybridAdapter is exposed two ways: a lazy property getter on the main entry (require('@qvac/rag').TurboVecHybridAdapter), and a dedicated sub-export @qvac/rag/turbovec. Both resolve to the same class. Lazy load is load-bearing: require('@qvac/rag') alone now loads zero @qvac/embed-llamacpp modules (verified 0 → 4 on first property touch). HyperDBAdapter-only consumers retain the zero-native-dep status quo.
• test/unit/turbovec-hybrid-adapter.test.js — brittle-bare test against a temporary Corestore. Covers save → search → delete → close → reopen → search; verifies reverse-map rebuild from doc metadata, content/metadata round-trip, and self-match scores. 25 assertions.
• examples/turbovec-poc.js — end-to-end smoke that synthesizes (or loads from the bench harness, if present) 200 × 1024 vectors and runs the full sequence: ingest, search, delete, persist + reopen.
• examples/turbovec-perf.js + examples/turbovec-vs-hyperdb.js — synthetic-corpus perf probes. See the "Findings" section below.

Why these choices

• Why hybrid (HyperDB for docs, separate .tvim for vectors)? Preserves replication-over-hyperswarm for the document corpus (HyperDB strength) while moving the ANN search to a format that's actually fit for it. Keeping the document schema identical to HyperDBAdapter means a Corestore is interchangeable for the read path; you can swap adapters without migrating data.
• Why a reverse map (metadata.__turbovec_u64Id) inside the doc rows? We can't extend the existing dbSpec without regenerating it (per prompt direction); we need a u64 → string-id lookup for search-result resolution; stashing the hash in the freeform metadata JSON is the smallest-blast-radius option (16 B/doc memory, rebuilt at open). Bigint doesn't survive JSON so the value is stored as a base-10 string.
• Why reject per-doc embeddingModelId overrides? The adapter is bound to one model at construction. Earlier draft warned + silently picked one of them, which produced contradictory behavior. Now mixed-model batches are rejected at validation — silent picking would cause hard-to-debug recall problems downstream.
• Why a lazy getter + sub-export for TurboVecHybridAdapter? Mirrors the IdMapIndex sub-export pattern in embed-llamacpp. Unconditional require would force every @qvac/rag consumer (including HyperDBAdapter-only users) to load the native .bare binary — a hard regression.
• Why atomic add + rollback? idx.addWithIds is atomic; we do the native insert first, then HyperDB writes. If the HyperDB transaction fails we remove the freshly-added u64s from the native index so the two stores stay consistent.
• Why a parallelized search? Each top-k slot is an independent HyperDB read. Serial awaits were wasted round-trip latency on synthetic perf measurements; Promise.all brings it to parity with HyperDBAdapter's shape.
• Why no reindex? The native index is data-oblivious — no centroid clustering, no IVF buckets — so reindexing has no meaning. Returns { reindexed: false, reason: 'data-oblivious; no rebuild needed' }.

What's left (out of POC scope)

• Atomic .tvim write. idx.write() is full-rewrite, not write-to-tmp + rename. A crash mid-write leaves a truncated file. Fix belongs in fabric (ggml_vec_index_write should do the rename dance) so all consumers benefit. Documented in-source.
• Cross-store atomicity. tx.flush() + idx.write() are not transactional relative to each other. A crash between them leaves HyperDB one batch ahead of the index. Fix belongs in fabric (journaled / delta-log .tvim format). Documented in-source.
• reindex for quantized layouts. Once fabric's bit_width becomes meaningful (real quantization), reindex may become useful for re-encoding existing vectors at a different precision. Currently a no-op.
• Filtered search. Not in POC scope (and the IdMapIndex C API doesn't expose a filter parameter yet).

Verification

• Unit test (test/unit/turbovec-hybrid-adapter.test.js): 25/25 asserts across stringIdToU64 correctness + adapter save → search → delete → close → reopen → search lifecycle.
• End-to-end smoke (examples/turbovec-poc.js): 6/6 sequence steps pass; deletion survives reopen; top-1 self-match score ≈ 1.0.
• Lazy-load invariant (verified at the REPL): require('@qvac/rag') loads 0 embed-llamacpp modules; touching rag.TurboVecHybridAdapter lazily pulls in 4.
• Lifecycle isolation (covered authoritatively in the embed-llamacpp PR's id-map-index.test.js): require.cache spy confirms the IdMapIndex sub-export never loads the GGMLBert require chain.

Findings from examples/turbovec-vs-hyperdb.js

Synthetic 1024-dim corpus, top-k=10, Apple Silicon macOS. Recall not measured (no ground truth on synthetic data; recall comparison lives in benchmarks/turbovec-eval/REPORT.md against real corpora).

N	Adapter	Queryable	Ingest	RSS	Query mean
3,633	TurboVecHybridAdapter (POC scalar f32)	3,633 / 3,633	142 ms	263 MB	1.69 ms
3,633	HyperDBAdapter (retain-full)	3,633 / 3,633	680 ms	507 MB	25.9 ms
10,000	TurboVecHybridAdapter	10,000 / 10,000	367 ms	490 MB	4.44 ms
10,000	HyperDBAdapter (retain-full)	10,000 / 10,000	2.0 s	794 MB	118.5 ms
50,000	TurboVecHybridAdapter	50,000 / 50,000	2.5 s	1,430 MB	21.7 ms
50,000	HyperDBAdapter (retain-full)	50,000 / 50,000	13.5 s	1,411 MB	1,466 ms
POC scales linearly (~450 ns/vector). At 50k the POC is ~68× faster than well-configured HyperDB on query and ~5× faster on ingest.

Note for future benchmarkers: the vs-hyperdb script writes unique contentHash per row. An earlier draft with identical contentHash values surfaced a real HyperDBAdapter issue — its internal _filterDuplicates collapses the entire batch to a single survivor, producing misleadingly fast benchmark numbers because nothing is actually indexed.

Companion PRs

#2590
tetherto/qvac-fabric-llm.cpp#153

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1215645648418280

[github-actions]

This draft PR is stale because it has been open 21 days and the author has not commented since opening. It is flagged for removal. Remove the stale label or comment on the PR or this will be closed in one day.

[github-actions]

This draft PR was closed because it has been stalled for 22 days with no author comment since opening. You can reopen this PR later if it is still necessary.

[github-actions]

This draft PR is stale because it has been open 21 days and the author has not commented since opening. It is flagged for removal. Remove the stale label or comment on the PR or this will be closed in one day.

[github-actions]

This draft PR was closed because it has been stalled for 22 days with no author comment since opening. You can reopen this PR later if it is still necessary.