feat(core): diversify retrieval results by document

Release highlights:
- Return broader document coverage without reducing the requested result count.
- Keep lexical fallback complete when the corpus exceeds one scan batch.

Release details:
- Add a configurable document cap with internal over-retrieval and ranked backfill.
- Apply diversification to CLI, MCP, ask, evaluation, portable status, and explanations.
- Scan fallback batches across the full eligible corpus and document MMR tradeoffs.

Verification:
- pnpm validate
- Quality benchmark: 100/100 cases, reproducible, recall@10 1.000.
- Manual CLI: 10/10 distinct documents by default versus 6/10 uncapped.

Author: jb-thery

README.md

@@ -224,7 +224,10 @@ pnpm exec rgr research "release obligations" --compact --timeout-ms 10000
 ```
 
 `preview` inspects redacted chunks without writing an index. `audit` compares sources with indexed
-state. `research` combines bounded query variants with deterministic cross-query ranking. Use
+state. Search keeps one primary passage per document by default, over-retrieves before applying the
+cap, and preserves ranked backfill when fewer distinct documents are available. Override the cap
+with `--max-chunks-per-document`; context-radius neighbors remain attached to their primary result.
+`research` combines bounded query variants with deterministic cross-query ranking. Use
 `rgr doctor --deep` only when you need a live O(corpus) inventory; normal status and doctor checks
 read the compact activation manifest.
 

docs/api-reference.md

@@ -58,13 +58,13 @@ instead of presenting an unverifiable line claim.
 
 With `explain: true`, `score` includes the vector and lexical ranks, their reciprocal-rank-fusion
 contributions, matched terms, backend scores, FTS or complete-fallback activation and reason,
-candidate materialization, query-variant count, indexed/unindexed rows, coverage, queue wait as
-`workloadQueueMs`, and
-`rankingPolicyFingerprint`. The fingerprint
-identifies the provider, retrieval profile, fusion parameters, and abstention threshold used by
-the result. Equal scores are ordered by stable source and chunk keys, so identical indexes return
-the same order regardless of backend row order. Search returns an empty array when every candidate
-fails the active provider's evidence threshold.
+fallback scan batches, candidate materialization, query-variant count, indexed/unindexed rows,
+coverage, document-cap and ranked-backfill state, queue wait as `workloadQueueMs`, and
+`rankingPolicyFingerprint`. The fingerprint identifies the provider, retrieval profile, document
+cap, fusion parameters, and abstention threshold used by the result. Equal scores are ordered by
+stable source and chunk keys, so identical indexes return the same order regardless of backend row
+order. Search returns an empty array when every candidate fails the active provider's evidence
+threshold.
 
 ### Persistent client for Node.js workers
 
@@ -236,13 +236,17 @@ to perform an external action.
 
 One evaluation pins a single configuration, connection, manifest generation, table handle, and
 embedding model. Cases run with bounded concurrency, preserve file order in the report, and release
-all scoped resources when evaluation finishes.
+all scoped resources when evaluation finishes. The report records the configured
+`maxChunksPerDocument` beside the ranking-policy fingerprint so reference results are reproducible.
 
-`SearchOptions` accepts `cwd`, `topK`, `contextRadius`, `includePaths`, `excludePaths`,
-`contextPaths`, `explain`, `vectorSearchMode`, `signal`, and `timeoutMs`. Set
+`SearchOptions` accepts `cwd`, `topK`, `maxChunksPerDocument`, `contextRadius`, `includePaths`,
+`excludePaths`, `contextPaths`, `explain`, `vectorSearchMode`, `signal`, and `timeoutMs`. Set
 `vectorSearchMode: "exact"` to bypass ANN for diagnostic comparison; the default `"adaptive"`
-uses the compatible strategy recorded in the manifest. `topK` is limited to 100 and
-`contextRadius` is clamped to three chunks. `IngestOptions` also accepts `rebuild`, a
+uses the compatible strategy recorded in the manifest. `topK` and `maxChunksPerDocument` are
+limited to 100. The document cap defaults to one, applies after scoring, and is preceded by internal
+over-retrieval. Ranked backfill preserves the requested result count when too few distinct
+documents are available. `contextRadius` is clamped to three chunks and attaches neighbors after
+primary-result diversification. `IngestOptions` also accepts `rebuild`, a
 positive `batchSize` that defaults to 25 files and is capped at 128, `incrementalFailurePolicy`, and
 an optional `onProgress` callback. Set `collectMetrics: true` to include privacy-safe phase,
 throughput, cache-state, RSS, OCR subprocess, fallback, error, timeout, and bound-activation metrics

docs/cli-reference.md

@@ -40,25 +40,29 @@ rgr sources add "docs/**/*.md" "!docs/archive/**"
 rgr sources list
 rgr preview --path docs --max-files 5 --max-chunks 3
 rgr search "migration" --top-k 5 --context-radius 1
+rgr search "migration" --top-k 5 --max-chunks-per-document 2
 rgr search "migration" --include-path docs --exclude-path docs/archive
 rgr search "migration" --context-path "Guide > Migration" --explain
 rgr search "migration" --exact-vector-search
 ```
 
 `sources add` accepts paths, globs, and `!` exclusions. Search, ask, and research accept `--top-k`,
 `--include-path`, `--exclude-path`, and repeatable `--context-path`. Search and ask accept
-`--explain`; the optional score object reports RRF contributions, retriever ranks, raw backend
-scores, FTS or complete-fallback activation and reason, candidate and index coverage, queue wait,
-and matched query terms without changing ranking. Use `--compact` on search or research when
+`--max-chunks-per-document` and `--explain`. The document cap defaults to one, applies after scoring,
+and over-retrieves internally before final truncation. Ranked backfill keeps the requested result
+count when the corpus has too few distinct documents. The optional score object reports RRF
+contributions, retriever ranks, raw backend scores, document-cap and backfill state, FTS or
+complete-fallback activation and reason, fallback scan batches, candidate and index coverage, queue
+wait, and matched query terms without changing ranking. Use `--compact` on search or research when
 agent context is limited. This remains explicit for CLI automation; MCP search, ask, and research
-are compact by default. Search and ask accept `--exact-vector-search` to bypass an active ANN
-index for diagnostics against exhaustive vector search. `--top-k` is limited to 100 and
-`--context-radius` is clamped to three chunks.
+are compact by default. Search and ask accept `--exact-vector-search` to bypass an active ANN index
+for diagnostics against exhaustive vector search. `--top-k` and `--max-chunks-per-document` are
+limited to 100, and `--context-radius` is clamped to three chunks.
 
 The explanation also contains a ranking-policy fingerprint so a stored quality report can be tied
-to the exact provider, profile, fusion, and abstention settings. Equal backend scores have a stable
-source-and-chunk tie-break. Search returns no result when all candidates fail the provider-aware
-evidence threshold; it does not force a low-confidence passage into the response.
+to the exact provider, profile, document cap, fusion, and abstention settings. Equal backend scores
+have a stable source-and-chunk tie-break. Search returns no result when all candidates fail the
+provider-aware evidence threshold; it does not force a low-confidence passage into the response.
 
 `preview` uses the active redaction and chunking configuration but never writes storage. `audit`
 reports min, mean, p50, p95, and max chunk sizes plus structural-context coverage.

docs/configuration.md

@@ -24,6 +24,7 @@ edit JSON only for a real need.
 | `embeddingModelRevision` | Pinned commit for bundled profiles | Use an immutable 40-character commit for reproducible model artifacts. Unknown custom models default to the mutable `main` revision until explicitly pinned. |
 | `embeddingModelDigest` | `null` | `rgr models pull --enable` records a SHA-256 identity for the resolved local artifact tree. Do not set it by hand unless the local files were verified independently. |
 | `topK` | `8` | Change the CLI and TypeScript default, up to the hard limit of 100. MCP requests without `topK` start at the lower of this value and 3. |
+| `maxChunksPerDocument` | `1` | Limit primary ranked passages from one document before final `topK` truncation, up to 100. Ranked backfill preserves the requested result count when the candidate set has too few documents. |
 | `mcpMaxTopK` | `10` | Bound explicit MCP passage requests; values above 100 are rejected. |
 | `mcpMaxOutputBytes` | `32768` | Cap variable-size MCP tool and resource JSON; the server also enforces an absolute 1 MiB ceiling. |
 | `chunkSize` / `chunkOverlap` | `1200` / `200` | Tune chunking, then rebuild the index. Chunk size is capped at 1,000,000 characters. |
@@ -32,7 +33,7 @@ edit JSON only for a real need.
 | `embeddingBatchSize` | `32` | Bound one model call; values above `128` are rejected. |
 | `sourceFingerprintMode` | `fast` | Use `strict` to hash every source on every inventory instead of reusing unchanged private fingerprints. |
 | `incrementalFailurePolicy` | `preserve-last-good` | Use `remove-stale` only when failed changed files must disappear immediately. |
-| `hybridTextScanLimit` | `5000` | Bound only the complete-scan fallback used when FTS is unavailable; values above 10,000 are rejected. A fallback smaller than the active corpus is rejected instead of returning silently truncated lexical evidence. |
+| `hybridTextScanLimit` | `5000` | Set the batch size for the complete lexical scan used when FTS is unavailable; values above 10,000 are rejected. The fallback scans every matching chunk across as many batches as the corpus requires. |
 | `workloadLimits` | See below | Bound active search, embedding, and ingestion work plus their queues and queue deadlines. |
 | `includeExtensions` | `[]` | Add safe custom text extensions. |
 
@@ -81,21 +82,33 @@ Profiles bound retrieval work. They are candidate and diversification budgets, n
 a larger budget improves every corpus. Evaluate the profile against a representative golden set
 before changing production configuration.
 
-| Profile | Quality intent | Latency intent | Default `topK` | Fallback scan cap | Vector candidates | FTS candidates | First-pass chunks per source | Context radius |
+| Profile | Quality intent | Latency intent | Default `topK` | Fallback batch size | Vector candidates | FTS candidates | Document cap | Context radius |
 | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| `fast` | Narrow, diverse evidence | Lowest work budget | 5 | 2,000 | max(40, 3 x `topK`) | max(100, 10 x `topK`) | 1, then backfill | 0 |
-| `balanced` | General-purpose evidence | Default work budget | 8 | 5,000 | max(80, 4 x `topK`) | max(250, 20 x `topK`) | 2, then backfill | 0 |
-| `quality` | Broader multi-section evidence | Highest work budget | 12 | 10,000 | max(200, 8 x `topK`) | min(4,000, max(500, 40 x `topK`)) | 4, then backfill | 1 |
-| `custom` | Golden-set validated | Operator-defined | configured | configured | max(80, 4 x `topK`) | max(250, 20 x `topK`) | 2, then backfill | 0 |
+| `fast` | Narrow, diverse evidence | Lowest work budget | 5 | 2,000 | max(40, 3 x `demand`) | max(100, 10 x `demand`) | configured, default 1 | 0 |
+| `balanced` | General-purpose evidence | Default work budget | 8 | 5,000 | max(80, 4 x `demand`) | max(250, 20 x `demand`) | configured, default 1 | 0 |
+| `quality` | Broader multi-section evidence | Highest work budget | 12 | 10,000 | max(200, 8 x `demand`) | min(4,000, max(500, 40 x `demand`)) | configured, default 1 | 1 |
+| `custom` | Golden-set validated | Operator-defined | configured | configured | max(80, 4 x `demand`) | max(250, 20 x `demand`) | configured, default 1 | 0 |
 
+`demand` is `topK * ceil(4 / maxChunksPerDocument)`, with a minimum multiplier of one. This internal
+over-retrieval gives the diversity pass enough lower-ranked documents before final truncation.
 Vector candidates are capped at 1,000. The FTS pool is profile-aware and capped at 4,000,
-independently from `hybridTextScanLimit`. Structural context and body text feed the primary local
-index. Exact file paths use a bounded scalar variant. Controlled exact-phrase, identifier, and fuzzy
-rare-term queries expand only a primary pool that cannot fill `topK`, preserving established ranks.
-The diversity pass prefers distinct sources first, then backfills ranked non-duplicate, non-overlapping
-chunks to `topK`. Hybrid ranking uses deterministic reciprocal-rank fusion with `k = 60` and equal
-vector and lexical weights. Stable source and chunk keys break score ties before ranks are assigned.
-The active provider, profile, and ranking parameters form a policy fingerprint stored in quality
+independently from the complete-scan batch size. Structural context and body text feed the primary
+local index. Exact file paths use a bounded scalar variant. Controlled exact-phrase, identifier,
+and fuzzy rare-term queries expand only a primary pool that cannot fill the demand, preserving
+established ranks.
+
+After scoring and abstention, the deterministic diversity pass keeps at most
+`maxChunksPerDocument` primary passages per relative path while preserving rank order. It then
+backfills ranked, non-duplicate, non-overlapping chunks only when too few documents or candidates
+remain to fill `topK`. Neighbor chunks requested through `contextRadius` are attached afterward and
+do not consume the primary document cap. MMR is not part of the default policy: pairwise similarity
+adds more work and corpus-dependent ordering, and the current golden benchmark does not show an
+advantage over the simple cap default. It can be evaluated later as an explicit opt-in strategy
+without changing this predictable default.
+
+Hybrid ranking uses deterministic reciprocal-rank fusion with `k = 60` and equal vector and lexical
+weights. Stable source and chunk keys break score ties before ranks are assigned. The active
+provider, profile, document cap, and ranking parameters form a policy fingerprint stored in quality
 reports and exposed by score explanations.
 
 Abstention is provider-aware. `local-hash` requires lexical evidence and gives query identifiers

packages/ragmir-core/benchmarks/quality.mjs

@@ -129,6 +129,7 @@ async function evaluateRankingVariants(client, goldenQueries) {
     const rows = await client.search(testCase.query, {
       topK: 100,
       explain: true,
+      maxChunksPerDocument: 100,
       ...(testCase.includePaths === undefined ? {} : { includePaths: testCase.includePaths }),
       ...(testCase.excludePaths === undefined ? {} : { excludePaths: testCase.excludePaths }),
       ...(testCase.contextPaths === undefined ? {} : { contextPaths: testCase.contextPaths }),

packages/ragmir-core/src/cli.ts

@@ -655,6 +655,11 @@ program
   .description("Retrieve the most relevant passages without calling an LLM.")
   .argument("<query>", "Search query.")
   .option("-k, --top-k <number>", "Number of passages to return.", parsePositiveInt)
+  .option(
+    "--max-chunks-per-document <number>",
+    "Maximum primary passages per document before ranked backfill.",
+    parsePositiveInt,
+  )
   .option(
     "--context-radius <number>",
     "Include neighboring chunks around each matched passage.",
@@ -687,6 +692,7 @@ program
       query: string,
       options: {
         topK?: number
+        maxChunksPerDocument?: number
         contextRadius?: number
         includePath: string[]
         excludePath: string[]
@@ -729,7 +735,7 @@ program
           const lexicalRank = result.score.lexicalRank ?? "n/a"
           console.log(
             pc.dim(
-              `score=${result.score.combinedScore.toFixed(6)} fusion=${result.score.fusion} vector=${result.score.vectorContribution.toFixed(6)} lexical=${result.score.lexicalContribution.toFixed(6)} vectorRank=${vectorRank} lexicalRank=${lexicalRank} matchedTerms=${result.score.matchedTerms.join(",") || "n/a"}`,
+              `score=${result.score.combinedScore.toFixed(6)} fusion=${result.score.fusion} vector=${result.score.vectorContribution.toFixed(6)} lexical=${result.score.lexicalContribution.toFixed(6)} vectorRank=${vectorRank} lexicalRank=${lexicalRank} documentCap=${result.score.maxChunksPerDocument} diversityBackfill=${result.score.diversityBackfillActivated} matchedTerms=${result.score.matchedTerms.join(",") || "n/a"}`,
             ),
           )
         }
@@ -742,6 +748,11 @@ program
   .description("Return cited retrieval context for a question without calling an LLM.")
   .argument("<query>", "Question to answer.")
   .option("-k, --top-k <number>", "Number of passages to use.", parsePositiveInt)
+  .option(
+    "--max-chunks-per-document <number>",
+    "Maximum primary passages per document before ranked backfill.",
+    parsePositiveInt,
+  )
   .option(
     "--context-radius <number>",
     "Include neighboring chunks around each matched passage.",
@@ -773,6 +784,7 @@ program
       query: string,
       options: {
         topK?: number
+        maxChunksPerDocument?: number
         contextRadius?: number
         includePath: string[]
         excludePath: string[]
@@ -1919,6 +1931,7 @@ function withSearchOptions(
   cwd: string,
   options: {
     topK?: number
+    maxChunksPerDocument?: number
     contextRadius?: number
     includePath?: string[]
     excludePath?: string[]
@@ -1929,6 +1942,7 @@ function withSearchOptions(
 ): {
   cwd: string
   topK?: number
+  maxChunksPerDocument?: number
   contextRadius?: number
   includePaths?: string[]
   excludePaths?: string[]
@@ -1939,6 +1953,7 @@ function withSearchOptions(
   const result: {
     cwd: string
     topK?: number
+    maxChunksPerDocument?: number
     contextRadius?: number
     includePaths?: string[]
     excludePaths?: string[]
@@ -1947,6 +1962,7 @@ function withSearchOptions(
     vectorSearchMode?: "exact"
   } = { cwd }
   addOption(result, "topK", options.topK)
+  addOption(result, "maxChunksPerDocument", options.maxChunksPerDocument)
   addOption(result, "contextRadius", options.contextRadius)
   addPathFilters(result, options)
   addOption(result, "explain", options.explain)

packages/ragmir-core/src/config.test.ts

@@ -45,6 +45,7 @@ describe("loadConfig", () => {
     expect(config.accessLog).toBe(true)
     expect(config.mcpMaxTopK).toBe(10)
     expect(config.mcpMaxOutputBytes).toBe(32_768)
+    expect(config.maxChunksPerDocument).toBe(1)
     expect(config.workloadLimits).toEqual({
       search: { concurrency: 8, maxQueue: 64, queueTimeoutMs: 30_000 },
       embedding: { concurrency: 1, maxQueue: 64, queueTimeoutMs: 30_000 },
@@ -538,10 +539,20 @@ describe("loadConfig", () => {
 
     await writeFile(
       path.join(root, ".ragmir", "config.json"),
-      JSON.stringify({ retrievalProfile: "fast", topK: 7, hybridTextScanLimit: 3_000 }),
+      JSON.stringify({
+        retrievalProfile: "fast",
+        topK: 7,
+        maxChunksPerDocument: 2,
+        hybridTextScanLimit: 3_000,
+      }),
     )
     expect(await loadConfig(root)).toEqual(
-      expect.objectContaining({ retrievalProfile: "fast", topK: 7, hybridTextScanLimit: 3_000 }),
+      expect.objectContaining({
+        retrievalProfile: "fast",
+        topK: 7,
+        maxChunksPerDocument: 2,
+        hybridTextScanLimit: 3_000,
+      }),
     )
   })
 
@@ -560,6 +571,9 @@ describe("loadConfig", () => {
     await writeFile(configPath, JSON.stringify({ topK: 101 }))
     await expect(loadConfig(root)).rejects.toThrow(/topK.*at most/i)
 
+    await writeFile(configPath, JSON.stringify({ maxChunksPerDocument: 101 }))
+    await expect(loadConfig(root)).rejects.toThrow(/maxChunksPerDocument.*at most/i)
+
     await writeFile(configPath, JSON.stringify({ mcpMaxTopK: 101 }))
     await expect(loadConfig(root)).rejects.toThrow(/mcpMaxTopK.*at most/i)