Repository navigation
feat: automatic sync of the index inside docgrep serve - #63
Merged
Merged
Conversation
A removed source that pairs with an added file of the same content hash and size (the one-to-one pairing of ADR-0027) loses no content, so it no longer counts against the guard of its old root, whether or not its passages are reused. Unpaired removals count as before, empties_root included: a root whose sources all moved elsewhere is not emptied. The plan and the guard message show the paired and unpaired counts separately; the guard details and the plan JSON carry `paired` when it is not zero. Amends ADR-0027. Story: batch 28, #45.
UpdateIndex.run and PlanUpdate.detect take an UpdateScope: only its stored paths are compared with the corpus and changed, every other recorded source counts as unchanged and no other file is added. A corpus that can stat single files (PathScanner, implemented by the corpus adapter as scan_paths) does not walk its roots, and a root that cannot be read raises instead of reporting its files gone. In a scoped update the roots need not all be readable, the removals of a root that trips the removal guard are held back and named in the summary instead of refusing the run (paired removals and all additions and modifications still apply), and the run kind recorded in the journal can be `sync`. Status and cancel treat a sync run like an update. Story: batch 28, #45.
SyncIndex scans every readable root from directory entries alone, keeps the changes it saw in memory until they have settled, and applies the settled ones with an update scoped to their paths, recorded as a `sync` run. A file is applied when its size and modification time were identical in two consecutive scans and its modification time is older than the settle time; a removal when the file was absent in two consecutive scans. Partial downloads, temporary, lock and swap files and the .obsidian, .git and .trash folders are never touched. A cycle that finds the index lock held ends and keeps its entries; a failed file is retried only when it changes or at a reconciliation. A reconciliation with the update plan runs at start-up, daily and for a root that is readable again; it settles additions and changes at once, and the targets of moves wait for the removal of their old path so that both are applied together. A root that cannot be read is paused, removals that trip the removal guard are held back (additions and changes still apply, and moves pass), a root without a completed manual update is left alone, and a configuration that drifted or an unfinished build or re-encode keeps sync idle with its reason. Story: batch 28, #45.
With DOCGREP_SYNC=true the lifespan of `docgrep serve` starts a background task that runs one sync cycle after the other on a worker thread, every DOCGREP_SYNC_INTERVAL_SECONDS, and stops it cleanly on shutdown: no new cycle, and an applied run finishes the source it is on. The task shares the container of search, so the models are loaded once, and asks for it on a worker thread, so the server answers while they load. Sync is off by default and cannot be enabled, triggered or configured over HTTP or MCP. New settings: DOCGREP_SYNC, DOCGREP_SYNC_INTERVAL_SECONDS (120), DOCGREP_SYNC_SETTLE_SECONDS (60) and DOCGREP_SYNC_RECONCILE_HOURS (24). The status sync reports is written to a small file in the data directory after every cycle, for other processes. Story: batch 28, #45.
`docgrep status`, `GET /info` and the MCP `get_index_status` body gain a `sync` section: enabled, running, interval and settle time, the last scan, the last applied run, pending counts by kind, the number of changes still settling (and of those waiting over 30 minutes), held removals per root, paused roots, roots awaiting a first update, the reason sync is idle, the last error and the progress of an applied run. `docgrep status` reads the section from the file the server writes after every cycle, so it works from another process, and adds a few stored paths of each root with held removals. `GET /info` and the MCP tool served by `docgrep serve` read the live state and never include a path; the stdio MCP server reads the file and shows no path either. The latest-run line and the cancel message treat a sync run like an update. The README documents the new keys of the status report. Story: batch 28, #45.
Local mode keeps its collections in arrays that nothing synchronizes, so a search that reads while an upsert grows them failed with an index error (`index N is out of bounds`) and surfaced as `store_unavailable`. A server process searches from several threads, and automatic sync writes from another, which made it likely. Every call of the local client now takes one lock; a server client is unchanged. Story: batch 28, #45.
An applied sync run gave a waiting search the models before and after every source, which made it about five times slower than needed. It now pauses once, after each finished source. Measured against a Qdrant server with 200 new files: the run takes 15 s instead of 27 s, search p50 rises 8% during it (33% when the run does not pause at all). Story: batch 28, #45.
README: enabling sync and its settings, what it does and never does (OCR, new roots, held removals, paused roots), how to read its status and logs, and the removal guard section for unpaired removals. ADR-0029 records sync inside `docgrep serve` (polling with settling, the scoped update, the holds) and the amendment of ADR-0027: moves do not count against the removal guard. Closes #45
A sync run that was cancelled dropped all its paths from the pending list and marked them as tried, so the files it had not reached were not applied until they changed. The paths now stay pending and are applied by the next cycle. An unexpected failure inside a cycle is recorded in the status by category only (a message can carry a path), and the MCP status scrubs paths from the idle reason and the last error. Story: batch 28, #45.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #45