Merge pull request #163 from jcode-works/feature/agentic-rag-performance-ci

feat(core): diversify and accelerate agentic retrieval

Author: jb-thery

README.md

@@ -10,8 +10,10 @@
 
 Ragmir turns specifications, Word files, PDFs, spreadsheets, code, and local exports into cited
 evidence indexed and retrieved on your machine. Core works offline by default, never uploads your
-corpus, and calls no model. Connect the agent or automation you already use through CLI, MCP, or a
-typed TypeScript API, or keep the complete workflow local with the optional Chat package.
+corpus, and calls no model. Ragmir is the local retrieval and evidence layer for agentic RAG
+workflows: the host agent plans and acts, while Ragmir supplies bounded, cited project evidence.
+Connect the agent or automation you already use through CLI, MCP, or a typed TypeScript API, or keep
+the complete workflow local with the optional Chat package.
 
 [Website](https://ragmir.com) · [npm](https://www.npmjs.com/package/@jcode.labs/ragmir) ·
 [Documentation](https://github.com/jcode-works/jcode-ragmir/wiki) ·
@@ -45,7 +47,7 @@ Outcome: Core installed with the repository's package manager; useful sources se
 
 3. Implement after approval:
 - Install @jcode.labs/ragmir as a dev dependency with the detected manager. Install Chat/TTS only if selected, at a compatible version.
-- Run the matching rgr setup --no-ingest --agents <selected> command. Keep project scope. If a same-name skill is unmanaged, show the diff and ask before --force.
+- Run rgr setup --no-ingest --agents <selected> via the detected manager. Keep project scope. Show unmanaged skill diffs and ask before --force-agent-skills.
 - Build a narrow .ragmir/config.json. Prefer stable relative globs for root guidance, docs/specs/ADRs, package READMEs/manifests, useful app config, and source/tests that explain behavior. Include locales only when useful.
 - Exclude .env*, credentials, keys, unapproved dumps/customer data, dependencies, generated/build/cache/coverage/log folders, vendored code, binaries/media, and .ragmir storage/models. In monorepos, keep nested bases scoped and shared knowledge at root.
 - Run preview and audit --unsupported before ingest. Review redactions, unsupported/oversized files, duplicates, chunks, and sensitive paths. Fix config first, then ingest.

context7.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://context7.com/schema/context7.json",
   "projectTitle": "Ragmir",
-  "description": "Confidential local RAG for coding agents and scripts, with offline cited retrieval. The TypeScript CLI, library, read-focused MCP server, and portable skills keep the corpus and index under ignored local state. Optional local Chat and TTS remain separate packages.",
+  "description": "Confidential local retrieval and evidence for agentic RAG workflows, with offline cited retrieval. The TypeScript CLI, library, read-focused MCP server, and portable skills keep the corpus and index under ignored local state. Optional local Chat and TTS remain separate packages.",
   "excludeFolders": [
     "node_modules",
     "packages/ragmir-core/dist",
@@ -21,7 +21,7 @@
   "rules": [
     "Ragmir v3.0.0 and later are licensed under AGPL-3.0-only, with a separate commercial licensing option from JCode Works for proprietary use. Earlier releases keep their published license.",
     "Use the canonical English setup prompt in docs/quick-start.md for a repository-aware installation; it detects the package manager and existing state, asks before changes or downloads, and verifies retrieval.",
-    "Ragmir Core returns cited retrieval context only; it does not synthesize answers itself.",
+    "Ragmir Core is the local retrieval and evidence layer for agentic RAG workflows. It is not an autonomous agent, planner, or action runtime; planning, tool use, and action authority stay in the host.",
     "Any compatible coding agent, script, CLI, TypeScript application, or MCP client can consume Core results; use a local consumer when no passage may leave the machine.",
     "Use `rgr portable export` to create a frozen relocatable folder containing the active index, any required local embedding model, a restricted launcher, portable skills, MCP adapters, and a SHA-256 inventory. Raw source files and access logs are excluded, but indexed passages remain sensitive. Verify the folder after moving it and keep action authority in the host agent or automation.",
     "Use one `createRagmirClient()` per project root for repeated work in a stateful Node.js process; it caches one immutable read snapshot until atomic generation replacement and flushes metadata-only access logs during close. Use top-level functions for one-shot scripts.",

docs/agent-integration.md

@@ -1,8 +1,10 @@
 # Agent integration
 
-Ragmir indexes the selected project files locally and gives the AI or automation you choose cited
-passages through CLI or one stdio MCP server. The default `local-hash` path keeps ingestion and
-retrieval offline. Core is model-agnostic, never uploads the corpus, and never calls a model itself.
+Ragmir is the local retrieval and evidence layer for agentic RAG workflows. It indexes the selected
+project files locally and gives the AI or automation you choose cited passages through CLI or one
+stdio MCP server. It is not an autonomous agent, planner, or action runtime: planning, tool use, and
+action authority stay in the host. The default `local-hash` path keeps ingestion and retrieval
+offline. Core is model-agnostic, never uploads the corpus, and never calls a model itself.
 
 For an interactive repository-aware installation, paste the canonical prompt from the
 [quick-start guide](./quick-start.md) into the coding agent. It detects the package manager and

docs/api-reference.md

@@ -281,9 +281,11 @@ clamped to three chunks.
 `ResearchReport.budgets` records configured and consumed budgets; `audit.mode` distinguishes
 `manifest` from `full`. Evidence exposes a weighted cross-query RRF `researchScore` and `bestRank`.
 The original query has a protected weight so language-aware expansions can add evidence without
-removing direct-search results from the same candidate depth.
+removing direct-search results from the same candidate depth. After fusion, research reapplies
+`maxChunksPerDocument` with ranked backfill; the report records the cap and whether backfill was
+needed.
 
-Golden evaluation files are limited to 1 MiB and 100 cases. Each query is limited to 20,000
+Golden evaluation files are limited to 16 MiB and 1,000 cases. Each query is limited to 20,000
 characters, with at most 100 expected paths or citations of 500 characters each.
 `AccessLogUsageOptions.days` accepts an integer from 1 to 3650.
 

docs/quick-start.md

@@ -26,7 +26,7 @@ Outcome: Core installed with the repository's package manager; useful sources se
 
 3. Implement after approval:
 - Install @jcode.labs/ragmir as a dev dependency with the detected manager. Install Chat/TTS only if selected, at a compatible version.
-- Run the matching rgr setup --no-ingest --agents <selected> command. Keep project scope. If a same-name skill is unmanaged, show the diff and ask before --force.
+- Run rgr setup --no-ingest --agents <selected> via the detected manager. Keep project scope. Show unmanaged skill diffs and ask before --force-agent-skills.
 - Build a narrow .ragmir/config.json. Prefer stable relative globs for root guidance, docs/specs/ADRs, package READMEs/manifests, useful app config, and source/tests that explain behavior. Include locales only when useful.
 - Exclude .env*, credentials, keys, unapproved dumps/customer data, dependencies, generated/build/cache/coverage/log folders, vendored code, binaries/media, and .ragmir storage/models. In monorepos, keep nested bases scoped and shared knowledge at root.
 - Run preview and audit --unsupported before ingest. Review redactions, unsupported/oversized files, duplicates, chunks, and sensitive paths. Fix config first, then ingest.

llms.txt

@@ -30,6 +30,8 @@ the corpus, and calls no model.
   `compact: false` only for an explicit full payload. CLI compact output remains opt-in.
 - Core behavior: cited retrieval only. `ask` returns evidence, while synthesis belongs to the calling
   agent or optional local chat add-on.
+- Agentic RAG boundary: Ragmir supplies bounded, cited evidence to a host agent or automation;
+  planning, tool use, and action authority remain in that host.
 - Consumer choice: use the AI or automation you already have through CLI, TypeScript, or MCP. Use a
   local consumer or optional Chat when no retrieved passage may leave the machine.
 - Portable knowledge: `rgr portable export` creates a frozen relocatable folder with the active

packages/ragmir-chat/README.md

@@ -40,7 +40,7 @@ Outcome: Core installed with the repository's package manager; useful sources se
 
 3. Implement after approval:
 - Install @jcode.labs/ragmir as a dev dependency with the detected manager. Install Chat/TTS only if selected, at a compatible version.
-- Run the matching rgr setup --no-ingest --agents <selected> command. Keep project scope. If a same-name skill is unmanaged, show the diff and ask before --force.
+- Run rgr setup --no-ingest --agents <selected> via the detected manager. Keep project scope. Show unmanaged skill diffs and ask before --force-agent-skills.
 - Build a narrow .ragmir/config.json. Prefer stable relative globs for root guidance, docs/specs/ADRs, package READMEs/manifests, useful app config, and source/tests that explain behavior. Include locales only when useful.
 - Exclude .env*, credentials, keys, unapproved dumps/customer data, dependencies, generated/build/cache/coverage/log folders, vendored code, binaries/media, and .ragmir storage/models. In monorepos, keep nested bases scoped and shared knowledge at root.
 - Run preview and audit --unsupported before ingest. Review redactions, unsupported/oversized files, duplicates, chunks, and sensitive paths. Fix config first, then ingest.

packages/ragmir-chat/package.json

@@ -67,9 +67,9 @@
     "node-llama-cpp": "3.20.0"
   },
   "devDependencies": {
-    "@types/node": "^24.10.1",
+    "@types/node": "^22.20.1",
     "publint": "^0.3.23",
-    "typescript": "^5.9.3",
+    "typescript": "^6.0.2",
     "vitest": "^4.1.10"
   }
 }

packages/ragmir-core/README.md

@@ -41,7 +41,7 @@ Outcome: Core installed with the repository's package manager; useful sources se
 
 3. Implement after approval:
 - Install @jcode.labs/ragmir as a dev dependency with the detected manager. Install Chat/TTS only if selected, at a compatible version.
-- Run the matching rgr setup --no-ingest --agents <selected> command. Keep project scope. If a same-name skill is unmanaged, show the diff and ask before --force.
+- Run rgr setup --no-ingest --agents <selected> via the detected manager. Keep project scope. Show unmanaged skill diffs and ask before --force-agent-skills.
 - Build a narrow .ragmir/config.json. Prefer stable relative globs for root guidance, docs/specs/ADRs, package READMEs/manifests, useful app config, and source/tests that explain behavior. Include locales only when useful.
 - Exclude .env*, credentials, keys, unapproved dumps/customer data, dependencies, generated/build/cache/coverage/log folders, vendored code, binaries/media, and .ragmir storage/models. In monorepos, keep nested bases scoped and shared knowledge at root.
 - Run preview and audit --unsupported before ingest. Review redactions, unsupported/oversized files, duplicates, chunks, and sensitive paths. Fix config first, then ingest.

packages/ragmir-core/benchmarks/README.md

@@ -22,9 +22,14 @@ and explicit reranker, compression, hashing, and content-dedup experiments.
 
 ## Claim rules
 
-- `bench:quality` requires clean indexes with matching corpus and quality fingerprints. It separates
-  p50/p95 latency from deterministic quality and evaluates vector-only, lexical-only, current
-  hybrid, and experimental lexical weights.
+- `bench:quality` requires clean indexes with matching corpus and outcome fingerprints. It separates
+  p50/p95 latency from deterministic quality and evaluates vector-only, lexical-only, an
+  undiversified hybrid baseline, the default one-chunk document cap, a two-chunk cap, MMR, and
+  experimental lexical weights. Ranking variants report the mean number of distinct documents in
+  the first ten results.
+- `bench:compare` recognizes scale and quality reports separately. It validates both quality runs,
+  their absolute gates, and explicit workload versions. Missing identities or metrics are invalid,
+  incompatible workloads are inconclusive, and only complete comparable reports can pass.
 - `bench:vector-index` uses deterministic 384-dimensional tables, 10 warm-ups, 100 samples, and five
   measured repetitions. A candidate fails if it loses at least 0.01 Recall@10, has incomplete
   coverage, misses the M/L latency gate, or does not improve p95. `--quick` is calibration only.