Implement incremental indexing at commit/diff level

[clafollett]

Intent

Codetriever currently tracks file content hashes via SHA256 in PostgreSQL, which provides basic change detection. However, the indexing model requires awareness of what changed at a commit or diff level rather than re-evaluating entire file sets. In the 2026 competitive landscape, users expect near-real-time index updates — GitHub Copilot achieved seconds-level indexing latency in March 2025, and Augment Code processes thousands of files per second via event-driven architecture.

Our PostgreSQL file queue and content hash tracking provide a solid foundation, but the system needs to operate at the granularity of git diffs and commit deltas rather than full file scans.

Business Rules

• The system MUST detect which files changed between the last indexed state and the current state using git diff or equivalent mechanism
• Only changed, added, or deleted files MUST be processed during an incremental index update
• Deleted files MUST have their chunks and vectors removed from both Qdrant and PostgreSQL
• Renamed files MUST be handled correctly — old entries removed, new entries created (or updated in place if content unchanged)
• The system MUST maintain a persistent record of the last indexed commit/state per repository per branch
• A full re-index MUST remain available as a fallback/recovery mechanism
• Incremental indexing MUST be safe for concurrent access — multiple index requests for the same repo MUST NOT produce duplicate or orphaned entries
• The system SHOULD support webhook-triggered indexing (e.g., git push hooks) for event-driven updates
• Index staleness (time between code change and searchable result) SHOULD be measurable and reportable

Competitive Context

• GitHub Copilot: seconds-level indexing latency (GA March 2025)
• Augment Code: real-time via GCP PubSub + BigTable, branch switches reflected in seconds
• Cursor: Merkle tree-based incremental diffing
• Any system requiring full re-indexing on every commit is uncompetitive in 2026

Acceptance Criteria

• Changed files are identified via git diff or content hash comparison
• Only changed files are re-parsed, re-embedded, and re-stored
• Deleted files have their data fully cleaned up across all storage layers
• Last indexed state is tracked per repository per branch
• Full re-index remains available as an explicit operation
• Concurrent indexing requests for the same repo are handled safely