Add ggml-vector-index C API (turbovec POC)

[dev-nid]

Context

Part of the turbovec ANN POC (3-repo coordinated change: this repo + tetherto/qvac packages embed-llamacpp and rag). Goal of the POC is to land the final cross-package architecture for a TurboQuant-style ANN vector index inside the QVAC stack, even with naive internals — so subsequent quantization / SIMD / GPU work can be swapped in behind a stable C API without touching downstream packages.

This PR is the fabric-side slice: a new public C API + scalar reference implementation + standalone C-API test. Strictly additive — no existing ggml type, llama symbol, or quantization code path is modified.

What was done

• ggml/include/ggml-vector-index.h — new public C API. Lifecycle (create/free), mutation (add/remove/contains/prepare), search, persistence (write/load), stats (len/dim/bit_width), error enum. Documented little-endian .tvim v1 on-disk format. Added to GGML_PUBLIC_HEADERS so consumers can #include "ggml-vector-index.h".
• ggml/src/ggml-vector-index.cpp — naive scalar reference implementation. Flat std::vector<float> storage, std::unordered_map<uint64_t, size_t> id-to-slot map with swap-with-last remove, std::priority_queue min-heap top-k. add() is atomic (rejects duplicate ids — both vs existing entries and within the batch — without mutating state). Compiled into ggml-base.
• tests/test-vector-index.cpp — standalone C-API smoke. Exercises create/stats, add with non-trivial 64-bit ids (42, (1ULL<<40)+7, (1ULL<<62)+11, UINT64_MAX-13) to flush out sign-extension / typed-array pitfalls, atomic duplicate rejection, top-1 self-match search (score ≈ 1.0 on L2-normalized inputs), sentinel-padded tail when k > len (UINT64_MAX ids), remove, and write→free→load→re-search round-trip with the deletion preserved. Wired into the existing llama_build_and_test target so it runs under fabric's LLAMA_BUILD_TESTS=ON test suite.

Why these choices

• Why a separate C-API module rather than a new GGML_TYPE_TBQVEC* tensor type? Top-k search is not a tensor op; the existing GGML_TYPE_TBQ* family is for matmul-shaped operations on the KV-cache path. Index ops have different lifecycle (long-lived mutable store, file-backed) and different consumers (a JS addon, not a graph node). Keeping them as a sibling C module avoids polluting ggml.h with non-tensor surface and keeps the index entirely off the LLM inference path.
• Why full f32 + scalar? POC scope. The brief explicitly defers quantization, SIMD, and GPU to a later phase. Internals can be swapped without changing the C API, the addon binding, the JS class, or the on-disk format's structural fields (the bit_width byte in the header is already reserved for the quantized v2 layout — POC accepts any [1, 32] value, validates the range, and stores full f32 regardless).
• Why a versioned file format with a reserved bit_width field? So the quantized v2 implementation lands as a version bump rather than a breaking change. v1 readers stay valid; v2 readers honor bit_width != 32.
• Why min-heap top-k rather than partial sort? Same algorithmic complexity O(N log k), but heap-based top-k is the shape the SIMD / Vulkan optimization phase will inherit (scratch arrays match what fused matmul+top-k kernels expect).
• Why atomic add? Mirrors the upstream turbovec Rust crate's add_with_ids semantics so the JS-side addWithIds can rely on "either all inserted or none" for the duplicate-id case. Also catches in-batch duplicates, not just collisions against the existing index.

What's left (explicitly out of POC scope, all behind this same C API)

• Quantization: replace std::vector<float> storage with packed TBQ codes (fabric already has the codebook generator at scripts/compute_tq_codebooks.py). Honors the bit_width field that is currently ignored.
• Random rotation (RHT) at ingest using fabric's existing primitive.
• RaBitQ length-correction scalar (for quantization-noise debiasing — not needed at full f32).
• SIMD kernels (NEON / AVX-512) for dot product + top-k.
• Vulkan / Metal compute kernels.
• Filtered search (allowlist / mask).

None of the above require changes to ggml-vector-index.h, the persistence format's structural fields, the addon binding, or any downstream JS code. They live entirely inside ggml/src/ggml-vector-index.cpp and (optionally) sibling files like ggml-vector-index-neon.cpp, ggml-vector-index.metal, etc.

Verification

• Standalone test-vector-index is built via llama_build_and_test and runs under fabric's standalone test suite (cmake -DLLAMA_BUILD_TESTS=ON).
• End-to-end coverage of the same C symbols via the addon binding (companion PR in embed-llamacpp) passes 3 tests / 69 assertions, including the sign-bit case ((1n << 63n) + 7n) that the C test deliberately leaves to the JS layer where the BigInt↔uint64_t boundary actually lives.
• No existing ggml/llama tests are touched; build remains green at upstream defaults.

Companion PRs

tetherto/qvac#2589
tetherto/qvac#2590