Enhancement: Promote HNSW vector indexes

[smiggleworth]

Audit classification

• Type: Enhancement
• Affected subsystem: Vector indexing and search
• User/operator impact: The capability may exist only as Experimental or an unverified path, limiting operator confidence and making production support boundaries unclear.
• Current behavior: The relevant implementation or workflow is not yet promoted to the requested support level; existing behavior remains bounded by its documented status and evidence.
• Expected behavior/outcome: Verify the behavior against explicit promotion criteria, document supported and unsupported cases, and promote it only when the acceptance criteria are met.
• Deterministic evidence: Source-of-truth: the issue's linked repository documentation, tests, workflows, and benchmark contracts; baseline commit afb23d0. The existing scope and acceptance criteria below are the deterministic work record.
• Confidence and support status: Medium to high, based on repository documentation and existing contracts.; Experimental or promotion candidate; not a claimed bug.
• Related tracker: Audit: Cassie engine and Admin UI/UX gap backlog at afb23d0 #1

Scope

Complete HNSW promotion evidence while preserving exact source-row reranking and deterministic fallback.

Source of truth

docs/experimental-promotion-criteria.md:20-29.

Acceptance criteria

• Cover option validation, restart hydration, exact rerank correctness, recall/latency curves, and fallback diagnostics.
• Test provider/dimension mismatch handling and concurrent mutation behavior.
• Retain representative benchmark artifacts with fixture and deployment metadata.
• Update support/readiness status only after common promotion gates pass.

Tracking: #1.

[smiggleworth]

Grouped in PR #44 (#44): promotion evidence matrix records existing local tests, missing same-commit or hosted evidence, and explicit unsupported boundaries. The surface remains not promoted until its row's gates are met.

[smiggleworth]

Local-only acceptance update: HNSW option validation, persistence, restart hydration, exact rerank, mutation safety, corruption/missing/stale fallback, and dimension checks pass locally.

[smiggleworth]

Full-history re-audit at current main d0874ff: docs/promotion-evidence-matrix.md explicitly marks issue #24 Not promoted and names missing closure evidence. The prior close recorded only a local correctness slice, while the original issue requires promotion evidence and a support/readiness update. Remaining criteria: same-commit retained recall and latency benchmarks plus provider-specific deployment evidence. Dependency: #7 and/or the relevant compatibility or deployment-profile evidence lane. Next action: retain the named evidence, re-run the focused and full validation gates, and update Feature Support only if every common promotion criterion passes.

[smiggleworth]

Current-main re-audit at : option validation, persisted graph hydration, exact source-row reranking, missing/stale/corrupt fallback, dimension mismatch handling, deletion visibility, and concurrent mutation safety are covered by current tests. The remaining promotion gate is retained same-commit recall/latency evidence with fixture and deployment metadata. Canonical run 32505662069 failed earlier in Tier 3, so it cannot support HNSW promotion; PR #158 repairs that upstream blocker but is not yet merged or evidenced.\n\nDisposition: keep #24 open and HNSW Experimental. Dependency: #7 / PR #158 followed by a passing unfiltered canonical artifact. Inspect HNSW recall, latency, access-path, candidate/read bounds, fallback counts, noise/trust, and deployment profile before any support/readiness change.

[smiggleworth]

Correction to the immediately preceding comment: shell rendering stripped the current-main SHA. The audited revision is 8b84ee936986f586567530a2decee949d2b8f14c.

At that revision, option validation, persisted graph hydration, exact source-row reranking, missing/stale/corrupt fallback, dimension mismatch handling, deletion visibility, and concurrent mutation safety are covered by current tests. The remaining promotion gate is retained same-commit recall/latency evidence with fixture and deployment metadata. Canonical run 32505662069 failed earlier in Tier 3, so it cannot support HNSW promotion; PR #158 repairs that upstream blocker but is not yet merged or evidenced.

Disposition: keep #24 open and HNSW Experimental. Dependency: #7 / PR #158 followed by a passing unfiltered canonical artifact. Inspect HNSW recall, latency, access path, candidate/read bounds, fallback counts, noise/trust, and deployment profile before any support/readiness change.