Concept: Local insights reports with model-backed enrichment

[tony]

Summary

• Add an agentgrep insights command tree that turns local agent history into a report — deterministic by default, with an opt-in ladder of HTML, classical ML, embeddings, a persistent hybrid index, and a local-LLM summary.
• Add a curated model registry that auto-downloads a sentence-embedding model through the same urllib + manifest artifact path used for local LLM models, into the platform cache.
• Add a pluggable Level-4 index — tantivy (BM25) + sqlite-vec (vectors) by default, with LanceDB as a drop-in single-store alternative behind one capability probe.
• Stream enrichment progress — phase lines, download bytes, and live LLM token deltas — to stderr, so long-running levels are visible without polluting machine-readable stdout.
• Keep the base install lean: importing agentgrep loads none of the optional backends, and every missing rung degrades to a precise install command instead of a traceback.

This is a concept branch demonstrating the ADR 0005 architecture end-to-end; it is not intended for direct merge.

Changes by area

Engine — src/agentgrep/insights/

• report.py: builds a report from a SearchRecord stream — resolves the effective level, attaches enrichment, and records diagnostics and grounded next actions.
• activity.py: deterministic level-0 analysis (per-agent/store counters, daily timeline, frequent terms, repeated instructions, an open-thread heuristic, coverage) emitting RecordRef drilldown handles.
• loader.py: lazy backend loading behind an injectable import_module seam; availability is probed with importlib.util.find_spec so a builtin report never imports a heavy backend just to list levels.
• models.py: the curated registry, an atomic urllib downloader, and a manifest sidecar; a torch-free model2vec model and the LLM artifacts share one cache layout.
• enrichers/: html (jinja2), ml (scikit-learn TF-IDF + KMeans), embeddings (sentence-transformers preferred, model2vec fallback), index (tantivy+sqlite-vec or LanceDB), llm (Ollama over httpx, grounded in compact facts).
• cache.py / model.py / progress.py: ADR-0005 cache-directory precedence, the typed report model with JSON payloads, and the progress protocol.

CLI — src/agentgrep/cli/

• parser.py: the insights subcommand tree (report, levels, doctor, setup, models, cache) with typed argument dataclasses.
• insights_render.py: dispatchers, the text/markdown/html/json/ndjson renderer, and the stderr console progress sink.
• __init__.py: main() dispatch for the new argument types.

Packaging & docs

• pyproject.toml: insights-html / -ml / -embeddings / -embeddings-st / -index / -index-lancedb / -llm / -all extras.
• docs/cli/insights.md: the CLI guide, registered in the CLI index toctree and card grid.

Design decisions

• Deterministic first, models opt-in: level 0 is always available and network-free; richer levels attach as enrichment only when their backend is installed and selected. best-installed never auto-selects the LLM level, since its runtime depends on an external daemon the import probe cannot see.
• One download pattern for every model: a sentence-embedding model provisions through the identical urllib + agentgrep-manifest.json path as a local LLM artifact, so "fetch an embedding model" and "fetch a local LLM" are the same mechanism with one cache layout.
• Pluggable index, default chosen for portability: tantivy + sqlite-vec is the default (tiny, ABI-stable wheels); LanceDB is offered as a single-store alternative rather than the default to keep the base index footprint small.
• Ollama is the only wired LLM runtime: LiteRT-LM and llama.cpp are fetch-only registry entries whose runtime returns a clear configuration error, rather than shipping half-working in-process bindings.
• Loaded backends are typed Any: an optionally-present third-party module cannot be statically typed, so EnricherContext.modules is dict[str, Any] — which also lets the tests inject plain fakes through the same seam.

Verification

No optional backend is imported at module top of the insights package (all loads are lazy):

rg -n "^import (torch|sklearn|tantivy|lancedb|httpx|jinja2|sentence_transformers|model2vec|numpy)" src/agentgrep/insights

The insights package itself is not imported by import agentgrep except function-locally inside the dispatchers:

rg -n "agentgrep.insights" src/agentgrep/__init__.py

Test plan

• Builtin report renders text/markdown/html/json/ndjson from the record stream
• Every enrichment level runs through injected fake backends — tests/test_insights_enrichers.py
• Model registry downloads via a monkeypatched urlopen, writes a manifest, and is a no-op when cached — tests/test_insights_models.py
• Level resolution, fallback diagnostics, and status classification — tests/test_insights_report.py
• Import-time guard: import agentgrep loads no optional backend or the insights package — tests/test_import_time.py
• Full gate: ruff check, ruff format, ty check, pytest, just build-docs
• Manual end-to-end: real model2vec download → semantic clustering, and a real tantivy + sqlite-vec index with working sample queries

---

Recreated from #65, which auto-closed during a rebase onto master; this branch (workflow-00) is the same work rebased onto current master.

Goal

Idempotent completion condition for /goal. It asserts an end state, not an action: if the state already holds, the first evaluation passes and nothing is edited.

/goal Branch workflow-00 (PR #87) has a recorded, evidence-backed disposition, and no branch, file or GitHub object is modified. This runs read-only from the shared master checkout: report git rev-parse --abbrev-ref HEAD (master) and do not check out, create, fetch or switch any branch or worktree; judge the local workflow-00 and insights-graph-engine refs. FIRST run and report both checks: comm -23 <(git diff --name-only 12542866 workflow-00 | sort) <(git diff --name-only 12542866 insights-graph-engine | sort); and git diff --numstat workflow-00 insights-graph-engine -- src docs pyproject.toml. The comm prints nothing today, so the empty output is evidence, not the deliverable. End state: the comm is empty, meaning workflow-00 touches no file insights-graph-engine does not, and for every path the numstat reports as removed you have stated whether it is a refactor on the #69 side or a capability that exists only on workflow-00 - concluding that workflow-00 is superseded by PR #69, is a self-declared concept branch not intended for direct merge, and carries no unique user-visible capability (CLI subcommand, packaging extra, docs page, MCP tool or enrichment level). If any unique capability does exist, name it and the file on insights-graph-engine that would carry it. Reporting that per-path verdict is the whole deliverable. Constraints: read-only - do not check out, rebase, edit or delete either branch, run no gh write, and merge or close nothing; the disposition is text for a human to apply. Do not settle the insights storage question here: #69's substrate may still move with the #47 versus #142 decision, so state the subsumption without endorsing a storage design, and do not recommend deleting the branch's concept material. Or stop after 6 turns and report what remains.

• Proof — comm -23 <(git diff --name-only 12542866 workflow-00 | sort) <(git diff --name-only 12542866 insights-graph-engine | sort) followed by git diff --numstat workflow-00 insights-graph-engine -- src docs pyproject.toml, with a per-path verdict written for every removed path
• Idempotence — Both commands are read-only and print the same output on every run (the comm is empty on the local refs, verified this pass), so a re-run reproduces the same evidence and the verdict text without mutating anything.
• Blocked by — nothing
• Blocks — nothing
• Preserves — The insights topic stays alive on Insights graph similarity engine + non-gated transformers LLM defaults #69; the concept material is not deleted; the Persistent SQLite DB index with cache-aware search #47-versus-Durable prompt corpus and derived search indexes (ADR 0019) #142 storage decision stays open because the verdict is about branch subsumption, not storage design; no gh state changes, so the maintainer keeps the close/keep call.

[tony]

Closing as superseded by #69, not as abandoned.

Verified by diffing the two branches directly: moving from #69's branch to this one would delete 5,175 lines and add 56. Every path this PR introduces exists on #69, and the 56 "additions" are the pre-refactor forms of functions #69 rewrote (build_search_record, the embeddings enricher, the insights CLI parser block) — not capability #69 lacks.

By patch-id, 4 of this branch's 7 commits are byte-identical on #69. The other 3 (agentgrep(style[ruff]): Meet new lint ratchets, agentgrep(test[insights]): Cover the report ladder with injected fakes, agentgrep(feat[insights]): Wire the insights CLI command tree) differ only because #69 evolved them further; their files are all present there.

#69 additionally carries the graph similarity engine, rerank, skills, sequences, the MCP insights tools, and the human-typed-prompt tests that this branch does not have.

Nothing is lost by closing: the branch is not deleted, and the nine workflow-00 stashes holding the v3-v9 iterations (graph engine, HDBSCAN, LanceDB IVF-PQ, cross-encoder rerank, the transformers/CUDA backend) are untouched and remain recoverable.

Track the work in #69.