feat(web): expose incremental indexing controls and status

[fishmingyu]

Summary

The current web demo presents repositories as already indexed snapshots. Users can browse wiki pages, CodeGraph, and Ask, but the UI does not expose CodeMiner's incremental indexing capabilities. The backend already has most of the machinery, especially for vector embeddings, but the web surface only reads a prebuilt qa_registry.json and loads static RepoBundles.

We should add a web-facing indexing workflow that lets users choose what to index, run full or incremental updates, inspect update status, and keep the Ask/wiki experience aligned with the latest indexed commit.

Reference pattern: Karpathy's LLM Wiki gist frames this as raw sources -> maintained derived artifact -> schema/workflow. For CodeMiner, source repos are the raw source of truth, while BM25/vector/graph indexes plus generated wiki pages are maintained derived artifacts that should be incrementally refreshed instead of rebuilt or re-derived from scratch on every question.

https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f

Current State

Relevant code today:

• web/app/page.tsx shows repo cards and an Add repo placeholder, but no index selection or update action.
• web/app/[repoId]/ask/page.tsx supports multi-turn Ask, but every turn calls POST /api/chat against the already-loaded runtime. It does not trigger or reflect index refreshes.
• codeminer/web/app.py exposes read/query endpoints only: repos, wiki, page graph, codemap, source, chat.
• codeminer/web/repo_registry.py loads manifests from qa_registry.json, lazily loads BM25/vector runtime, and does not reload bundles after an index changes.
• scripts/build_qa_index.py is the offline full-build path for demo repos.
• codeminer/compiler/index_builders.py already defines builder-level incremental_update():
  • VectorIndexBuilder has real incremental update using IncrementalState, IncrementalChunkStore, EmbeddingsCache, GitDiffDetector, and IncrementalIndexUpdater.
  • BM25IndexBuilder, SymbolGraphBuilder, and ZoektIndexBuilder currently fall back to rebuild behavior at the builder protocol level.
• codeminer/index/incremental/ persists last commit, chunk state, and content-hash keyed embedding cache. This is the capability the web UI should make visible.
• docs/incremental_graph/index.md documents graph patching separately; if we surface symbol graph status, the UI should distinguish true graph patching from builder-level rebuild fallback.

Proposed Product Shape

Expose three index surfaces in the web UI:

1. BM25 / keyword index

   • Fast lexical retrieval.
   • Builder-level incremental action currently rebuilds the BM25 artifact.
   • UI copy should say rebuild on update unless true delta BM25 is later added.

2. Embedding / semantic index

   • Hierarchical vector store, L0/L2 where configured.
   • True incremental path: git diff -> rechunk changed files -> reuse cached embeddings by content hash -> embed only cache misses -> update FAISS.
   • Show metrics such as changed files, chunks re-embedded, chunks from cache, cache hit rate, and indexed commit.

3. Symbol graph / CodeGraph index

   • Powers codemap/wiki graph views.
   • Current builder route is rebuild-style, while graph patching exists separately for supported languages.
   • UI should show capability and update mode explicitly: patch, rebuild, or unavailable depending on what the backend can safely execute.

The repo card or repo header should show concise index badges:

• built / missing / stale / updating / failed
• last indexed commit and current HEAD when available
• update mode: incremental, rebuild fallback, or unavailable

Proposed Backend API

Add a small indexing job layer under codeminer/web/ rather than calling scripts directly from the frontend.

Suggested endpoints:

• GET /api/repos/{repo_id}/index-status

  • Returns per-index state, last indexed commit, current HEAD, stale flag, capabilities, and update mode.

• POST /api/repos/{repo_id}/index-jobs

  • Body: { indexes: ["bm25", "vector", "symbol_graph"], mode: "incremental" | "full", force?: boolean }
  • Creates a background job and returns job_id.
  • If mode=incremental is requested for an index without true delta support, either reject with a clear message or run the documented rebuild fallback with update_mode: "rebuild" in the job result.

• GET /api/index-jobs/{job_id}

  • Returns job status, stage, logs/events, started/finished timestamps, and per-index result metadata.

• Optional later: GET /api/index-jobs/{job_id}/events via SSE for live progress. Polling is enough for the first implementation.

Implementation notes:

• Wrap IndexBuilderRegistry, IndexCompiler, and builder incremental_update() in a web service object, for example IndexJobManager.
• Keep one active indexing job per repo to avoid concurrent writes to the same cache directory.
• Update RepoManifest and qa_registry.json metadata after a successful full or incremental update.
• Add a RepoRegistry.reload(repo_id) or bundle invalidation path so Ask/wiki/codemap use the new artifacts without restarting codeminer-web.
• Preserve the existing prebuilt-index workflow: if a vector index path points into a read-only prebuilt_dir, the job layer must reject writes or copy-on-write into data_dir before updating.

Proposed Frontend Flow

On the landing page:

• Turn the current Add repo placeholder into an index workflow entry point.
• Each repo card shows the three index badges and whether updates are available.

On the repo/wiki header:

• Add an Indexes control with three selectable rows/cards: BM25, Embeddings, Symbol graph.
• Let users run Update changed files when a previous commit exists, and Rebuild selected when missing or forced.
• Show incremental result stats after completion, especially for embeddings.

On the Ask page:

• Keep the existing multi-turn thread behavior, but add index-awareness:
  • Before asking, show if the selected retrieval indexes are stale.
  • Offer Update indexes first vs Ask with current index.
  • After an index job finishes, Ask should use the reloaded bundle automatically.
• Consider a Karpathy-inspired file answer back to wiki follow-up later, but do not block this issue on that. The immediate goal is indexed artifacts that stay current.

Acceptance Criteria

• The web UI exposes exactly three primary index surfaces: BM25, Embeddings, and Symbol graph/CodeGraph.
• Users can see per-index status for a repo without reading logs or manifest JSON.
• Users can trigger an update for selected indexes from the web UI.
• Embedding updates use the existing incremental path and surface chunks_reembedded, chunks_from_cache, cache_hit_rate, and new_commit.
• Indexes without true delta support clearly report rebuild fallback rather than being described as true incremental.
• After a successful update, /api/repos, /api/chat, wiki pages, and codemap read the refreshed manifest/artifacts without a server restart.
• Failed jobs expose a useful error state in the UI and do not corrupt the previous working manifest.
• Unit/API tests cover status calculation, job lifecycle, vector incremental metadata propagation, and registry/bundle reload behavior.

Open Questions

• Should the three frontend options be raw index types (bm25, vector, symbol_graph) or higher-level presets (keyword, semantic, full code intelligence)? I recommend raw index types with human-readable names because they map cleanly to existing manifests.
• For symbol graphs, should the first milestone use rebuild fallback only, or wire supported languages to LSIndexer.graph_patch() when a base graph and target commit are available?
• Should wiki page generation/cache invalidation be included in the same job, or treated as a separate derived artifact after indexes update?
• How should local repo ingestion work for Add repo: path input only for local dev, Git URL clone, or both?

Non-Goals For First PR

• No redesign of retrieval quality or ranking.
• No new embedding model selection UI beyond displaying the configured model/dimensions.
• No requirement to implement true delta updates for BM25 or symbol graph if not already available through a safe backend path.
• No need to persist Ask answers into wiki pages in the first iteration; that can be a follow-up once index freshness is visible.

[fishmingyu]

@YashJayswal24 @boqiny
We can discuss about this feature this week :)

[fishmingyu]

Triage:

• Type: type/feature — new web UI surface over existing backend indexing capabilities; no behavior is broken.
• Scope: scope/compiler (IndexCompiler, IndexBuilderRegistry, RepoManifest in codeminer/compiler/index_builders.py) and scope/indexing (IncrementalState, EmbeddingsCache, GitDiffDetector, IncrementalIndexUpdater in codeminer/index/incremental/).
• Priority: priority/P2 — meaningful improvement to the web demo; workaround exists via offline scripts (scripts/build_qa_index.py, builder-level incremental_update()).

Routing to @fishmingyu (compiler) and @siriuxyu (indexing).

---

Generated by Claude Code