Refactor batch write to use concurrent ThreadPoolExecutor

.claude/skills/reflect/SKILL.md

@@ -22,7 +22,7 @@ hooks:
           command: "${CLAUDE_PLUGIN_ROOT}/hooks/session-start.sh"
 metadata:
   author: shashwatjain
-  version: '0.5.2'
+  version: '0.6.0'
 ---
 
 # Reflect — Repo-Owned Memory

.reflect/.last_run

@@ -1 +1 @@
-{"last_checkpoint": "af09a9533c27", "last_git_sha": "52ac785", "timestamp": "2026-04-04T18:19:00.728855"}
+{"last_checkpoint": "", "last_git_sha": "2f5ad53", "timestamp": "2026-04-09T13:44:40.267220"}

.reflect/context.md

@@ -1,35 +1,16 @@
 # Project Context
-<!-- GENERATED by reflect — do not edit, regenerate with: reflect context -->
-<!-- Last updated: 2026-04-04 18:19 | 12 sessions analyzed -->
+<!-- GENERATED by reflect (wiki) — do not edit, regenerate with: reflect context -->
+<!-- Last updated: 2026-04-09 13:44 -->
 
 ## Key Decisions & Rationale
 
-- **Declarative format.yaml + Claude subagent synthesis** — Replaced executable harness with format.yaml-driven context generation to decouple schema from script maintenance and enable format reuse. Decided 2026-04-04 as core pivot. (checkpoint 68bae57a699d)
-- **Zero-storage, read-on-demand architecture** — Fetch all evidence from Entire/git at synthesis time rather than persisting. Avoids staleness/sync at cost of latency. (checkpoint 90e2641946db)
-- **Replaceable harness interface** — Design harness as swappable script, enabling future language alternatives (Python, bash, Ruby). (commit 46c8781)
-- **GitHub Releases distribution with skill/agents/ bundling** — skill/SKILL.md is canonical source of truth; skill/agents/ is parallel directory for bundled subagents. Both copied into target repo during `reflect init`. (checkpoint af09a9533c27)
-- **SessionStart auto-refresh via .last_run state tracking** — Automatically refresh context.md on session start without user action, using per-repo freshness tracking. (checkpoint 90e2641946db)
-- **Split 'Open Work' from 'Abandoned Approaches' with revisit_when field** — Separated unfinished items (7d recency) from dead ends (90d recency). Revisit_when makes entries self-expiring rather than permanent blockers. (checkpoint 68bae57a699d)
+- **Critical Pitfalls Detection in Context Pipeline** — Agents working with codebases need early, actionable awareness of project pitfalls—common mistakes, architectural constraints, and known failure modes (commit 9cdf813)
+- **High-Water Mark Tracking for Incremental Ingest** — High-water mark tracking is a checkpoint-based mechanism that prevents redundant processing when ingesting repository evidence across multiple session (commit 2f5ad53)
+- **Keeper Agent: Focused Repo Memory** — The Keeper Agent evolved from a **broad repo memory** approach to a **focused** one, deliberately narrowing its scope to deliver higher-signal knowled (commit 34626f0)
+- **Wiki Layer for Persistent Compounding Knowledge** — The wiki layer (v0.6.0) introduces a structured mechanism for agents to build and reference persistent knowledge across sessions, extending the sessio (commit 1be188b)
+- **Zero Storage Architecture (v4)** — The Zero Storage Architecture (v4) is a foundational design pattern that eliminates external storage dependencies by storing all agent-generated evide (commit 46c8781)
 
 ## Gotchas & Friction
 
-- **Dual SKILL.md copy sync is unautomated** — skill/SKILL.md (source) and .claude/skills/reflect/SKILL.md (installed) must be manually kept in sync with no enforcement or warnings on divergence. (checkpoint 90e2641946db)
-- **Entire CLI is hard undocumented dependency** — reflect's real-world value is near-zero without Entire installed, but README does not adequately warn users of this bottleneck. (checkpoint 90e2641946db)
-- **Many context.md entries are truncated, vague, or stale** — Signal filtering for "hard-to-derive" facts is aspirational. Results in low-confidence briefings. (checkpoint 90e2641946db)
-- **Checker agent returns silent '[CLI error: unknown]'** — No diagnostic content; blocks root cause diagnosis when agent fails. (checkpoint ed2a5817517f)
-- **Checker.py lacks markdown fence stripping on JSON parsing** — Model wraps JSON in ```json...``` blocks; parser fails to strip before json.decode(). (checkpoint ed2a5817517f)
-- **YAML-lite parser scoping bug discovered post-edit** — in_entry_fields reset on every loop iteration, skipping all but first entry_fields item. Required second read-and-fix pass. (checkpoint 68bae57a699d)
-
-## Open Work
-
-- **Merge/overwrite policy for conflicting agent files** — skill/agents/ install may overwrite user-customized agents in .claude/agents/; no conflict resolution defined. (checkpoint af09a9533c27)
-- **Verify install.sh includes skill/agents/ in release tarball** — GitHub Releases pipeline may omit agents directory from distributed artifact. (checkpoint af09a9533c27)
-- **Strip markdown fences in checker.py before JSON parsing** — Add explicit stripping of ```json...``` blocks before json.loads(). (checkpoint ed2a5817517f)
-- **Review CHECKER_SYSTEM_PROMPT formatting instructions** — Explicitly forbid markdown code fences in JSON responses to prevent parser failures. (checkpoint ed2a5817517f)
-- **Decouple from Entire CLI dependency** — Add native Claude Code session logs, git history, PR descriptions, and issue trackers as optional evidence sources. (checkpoint 90e2641946db)
-
-## Critical Pitfalls
-
-- **mistake**: Declared loop-state variable (in_entry_fields) inside iteration, resetting on every cycle. **consequence**: Only first entry_fields item parsed; rest silently dropped, breaking Abandoned Approaches section. **rule**: Declare state variables outside loop blocks; test parser output independently before using in agents. (checkpoint 68bae57a699d)
-- **mistake**: Allowed model to wrap structured JSON in markdown code fences ```json...```. **consequence**: Parser failed to decode, producing default scores (2,2,2,2) and blocking all diagnosis. **rule**: Explicitly strip markdown fences before json.loads(); prompt models to return raw JSON without code block wrapping. (checkpoint ed2a5817517f)
-- **mistake**: Error handling returned placeholder '[CLI error: unknown]' with zero diagnostic content. **consequence**: Ground truth signals completely missed, making root cause investigation impossible. **rule**: All error paths must include diagnostic info; never return opaque error placeholders. (checkpoint ed2a5817517f)
+- **Repo-Specific Collection Names Prevent Collisions** — When multiple repositories use reflect in a shared environment (monorepo, local dev machine, or shared CI runners), QMD collections with identical nam (commit 7b17590)
+- **Symlink Hooks Directory Handling** — When `.claude/hooks` is a symlink to another directory — a common configuration for shared setups or templated deployments — hook installation logic m (commit bddce4c)

.reflect/wiki/decisions/critical-pitfalls-detection.md

@@ -0,0 +1,38 @@
+---
+created: 2026-04-09
+updated: 2026-04-09
+sources: [commit 9cdf813, commit 7aa4e33, commit 1be188b]
+tags: [pitfalls, context-pipeline, agents, decision]
+status: active
+---
+
+# Critical Pitfalls Detection in Context Pipeline
+
+## Motivation
+
+Agents working with codebases need early, actionable awareness of project pitfalls—common mistakes, architectural constraints, and known failure modes—to avoid costly errors. Manual pitfall documentation is incomplete and stale. The Critical Pitfalls Detection feature (commit 9cdf813) automates identification and surfacing of project-specific pitfalls directly into the context pipeline, ensuring agents receive dynamic, session-aware warnings before attempting changes.
+
+## Integration with Context Pipeline
+
+This decision was implemented as part of the broader refactoring toward declarative, subagent-driven context generation (commit 7aa4e33). Rather than static docs, the system analyzes project state and prior evidence to systematically detect:
+
+- Common architectural violations and antipatterns
+- Patterns that have caused issues in previous sessions
+- Configuration constraints and fragile dependencies
+- Known regressions in specific code paths
+
+Pitfalls are assembled into context during session start, enriching semantic context without requiring manual updates. This complements the wiki layer (commit 1be188b) by distinguishing persistent, compounding knowledge (wiki) from dynamic, task-aware pitfall warnings (context pipeline).
+
+## Implementation Details
+
+Detection logic integrates with the declarative `format.yaml` configuration system and the Claude subagent for context generation. When agents request context, the pipeline evaluates:
+
+1. Prior checkpoint evidence and session history
+2. Repository-specific patterns and failure modes  
+3. Current codebase state and interdependencies
+
+High-priority pitfalls are elevated to the top of context blocks, ensuring agents see warnings before proceeding with changes.
+
+## Status
+
+Active. Pitfalls detection is part of the standard context pipeline as of v0.6.0 and continues to evolve as more pitfall patterns are identified and catalogued across sessions.

.reflect/wiki/decisions/high-water-mark-incremental-ingest.md

@@ -0,0 +1,29 @@
+---
+created: 2026-04-07
+updated: 2026-04-09
+sources: [commit 2f5ad53, commit 1be188b, commit 7b17590]
+tags: [incremental-ingest, persistence, session-chaining, performance]
+status: active
+---
+
+# High-Water Mark Tracking for Incremental Ingest
+
+## Overview
+
+High-water mark tracking is a checkpoint-based mechanism that prevents redundant processing when ingesting repository evidence across multiple sessions. Rather than re-scanning and re-processing identical code or checkpoints on each session start, the system records the last successfully processed state and resumes from that point on subsequent runs.
+
+## Why It Matters
+
+Without high-water mark tracking, each session would re-index all repository content, redundantly extract evidence, and recompute knowledge artifacts. This creates two problems: wasted computational effort and stale or duplicative knowledge in the wiki layer. High-water mark tracking (commit 2f5ad53) solves this by enabling **incremental, stateful ingest** — only new commits, modified files, or novel checkpoints are processed and merged into the persistent knowledge base.
+
+## Mechanism
+
+The high-water mark approach records the ID or timestamp of the last successfully ingested item (commit, checkpoint, or collection batch). On the next session, the ingest pipeline queries from that mark onward, filtering out already-processed content. This is particularly important for repo-specific metadata: commit 7b17590 introduced repo-specific qmd (query metadata) collection names to avoid collisions across projects, meaning the high-water mark must also be scoped per repository.
+
+## Integration with Wiki Layer
+
+The wiki layer (commit 1be188b, v0.6.0) provides the persistent storage for compounding knowledge. High-water mark tracking ensures that sessions don't just accumulate redundant facts — they add incremental, novel insights. When a new commit is ingested, its evidence contributes fresh context to the wiki; older commits are skipped entirely, reducing memory overhead and search noise.
+
+## Session Chaining Benefit
+
+This mechanism directly enables session chaining: because state is tracked across sessions, an agent can run multiple focused tasks in sequence without losing accumulated context. Commits 04dd4dd and 33f3add document session chaining guidance; high-water mark tracking is the underlying infrastructure that makes this efficient rather than wasteful.

.reflect/wiki/decisions/keeper-agent-focused-design.md

@@ -0,0 +1,43 @@
+---
+created: 2026-04-09
+updated: 2026-04-09
+sources:
+  - commit 34626f0
+  - commit fd5f2a2
+  - commit 0500d79
+  - commit 1be188b
+  - commit 2f5ad53
+  - commit 7b17590
+tags: [keeper-agent, memory-scope, architecture, decisions]
+status: active
+---
+
+# Keeper Agent: Focused Repo Memory
+
+The Keeper Agent evolved from a **broad repo memory** approach to a **focused** one, deliberately narrowing its scope to deliver higher-signal knowledge that agents can reliably act on. This shift emerged after testing revealed that comprehensive tracking produces stale, unfocused artifacts.
+
+## The Pivot: From Breadth to Depth
+
+Keeper was initially rewritten as a broad repository chronicler (commit 34626f0), attempting to track all state changes and generate comprehensive summaries. However, practical testing showed this approach generated low-signal noise—agents struggled to distinguish critical insights from routine updates.
+
+The refined Keeper agent (commit fd5f2a2) instead constrains its mandate to **high-impact, decision-driven knowledge**: architecture decisions, critical pitfalls, session learnings, and evidence snapshots. Generic file inventories and broad summaries were traded for focused depth.
+
+## Implementation: Incremental and Scoped
+
+The focused design leverages several technical patterns:
+
+- **Incremental ingestion** with high-water mark tracking (commit 2f5ad53) prevents duplicate work across sessions, allowing Keeper to resume where it left off
+- **Persistent wiki compilation** (commit 1be188b) transforms evidence into indexed markdown pages that compound knowledge over time
+- **Critical pitfall detection** surfaces hard blockers early in the context pipeline (related to commit 0500d79)
+- **Repo-specific namespacing** via MongoDB collection names (commit 7b17590) isolates data across multi-project setups, avoiding collisions
+
+## What "Focused" Means
+
+Rather than attempting to be a universal repository chronicle, Keeper now functions as a **scoped memory layer**:
+
+1. **Selective ingestion**: captures only patterns with lasting impact or cross-session relevance
+2. **Low maintenance overhead**: incremental state tracking eliminates reprocessing
+3. **Compounded learning**: wiki pages persist and grow richer across sessions
+4. **Decision-centric**: prioritizes actionable insights over comprehensive documentation
+
+This focused scope makes Keeper maintainable, reliable, and genuinely useful to consuming agents—a decision log rather than an encyclopedia.

.reflect/wiki/decisions/wiki-layer-persistent-knowledge.md

@@ -0,0 +1,42 @@
+---
+created: 2026-04-09
+updated: 2026-04-09
+sources: [commit 1be188b, commit 2f5ad53, commit 7b17590, commit 33f3add]
+tags: [wiki, persistence, knowledge-accumulation, v0.6.0]
+status: active
+related: [decisions/keeper-agent-repo-memory, features/cross-session-learning]
+---
+
+# Wiki Layer for Persistent Compounding Knowledge
+
+## Overview
+
+The wiki layer (v0.6.0) introduces a structured mechanism for agents to build and reference persistent knowledge across sessions, extending the session-based evidence capture model established in earlier versions. Rather than treating knowledge as ephemeral checkpoints scoped to individual sessions, the wiki layer enables agents to compile, curate, and incrementally refine knowledge artifacts that compound over time (commit 1be188b).
+
+## Problem Statement
+
+Prior to v0.6.0, cross-session knowledge relied on checkpoint references within session logs. Agents could query past evidence but had no formal way to synthesize insights into evolving, canonical documents. This created friction when:
+- Multiple agents need shared, authoritative facts about the repository
+- Knowledge evolves through repeated refinement across sessions
+- Agents must manually re-discover or re-synthesize insights from scattered checkpoints
+
+## Architecture
+
+The wiki layer uses **incremental ingest with high-water mark tracking** (commit 2f5ad53). Rather than re-processing all historical evidence on each session, agents maintain a pointer to the last processed checkpoint and only ingest new evidence since that point. This enables:
+
+1. **Efficiency**: Avoid redundant processing of old sessions
+2. **Idempotency**: Agents can safely re-run without duplicating prior work
+
+To prevent collision across multiple repositories using the same centralized storage, the wiki uses **repo-specific QMD collection names** (commit 7b17590), qualifying each repository's knowledge artifacts by a stable repository identifier.
+
+## Integration with Keeper Agent
+
+The wiki layer complements the Keeper agent (repo memory agent). While Keeper operates within a single session to understand the current repository state, the wiki layer allows Keeper and other agents to deposit curated knowledge for future sessions to build upon. Cross-session learning enhancements (commit 33f3add) document how agents should use both mechanisms in concert.
+
+## Key Distinctions
+
+- **Checkpoints**: Session-scoped, event-driven evidence capture (existing)
+- **Wiki**: Repository-scoped, agent-curated synthesis of patterns, decisions, and lessons (new)
+- **High-water mark**: Enables incremental reads without re-processing history
+
+The wiki layer deliberately separates the *collection* of facts (via checkpoints) from the *curation* of knowledge (via wiki articles), allowing human reviewers and agents to manage knowledge quality independently.

.reflect/wiki/decisions/zero-storage-architecture.md

@@ -0,0 +1,35 @@
+---
+created: 2026-04-03
+updated: 2026-04-09
+sources: [commit 46c8781, commit 1be188b, commit 2f5ad53, commit 7b17590]
+tags: [architecture, storage, harness, design]
+status: active
+related: [decisions/wiki-layer-v06, decisions/evidence-format-spec]
+---
+
+# Zero Storage Architecture (v4)
+
+The Zero Storage Architecture (v4) is a foundational design pattern that eliminates external storage dependencies by storing all agent-generated evidence and metadata directly within the repository as first-class artifacts (commit 46c8781). This approach enables portable, agent-agnostic knowledge persistence without requiring separate databases, cloud services, or sidecar storage systems.
+
+## Core Principle: Replaceable Harness
+
+The "replaceable harness" pattern decouples the evidence format and schema from the runtime that generates it (commit 46c8781). The harness—any CLI, script, or agent framework that executes workflows—can be upgraded, swapped, or reimplemented entirely without breaking downstream consumers of the evidence. All information is serialized to stable formats (YAML checkpoints, structured metadata) that persist independently of the harness implementation.
+
+This enables Claude or other agents to read evidence written by the current CLI, and future agents to read evidence written today, regardless of how the harness evolves.
+
+## Storage Location and Structure
+
+Evidence is stored in `.reflect/` directory at the repository root, organized as checkpoints and session metadata. Each session creates a timestamped record; each checkpoint captures execution state, reasoning, and outputs. By storing everything in the repo, the evidence benefits from version control, snapshots, and distributed access—the same mechanisms protecting source code now protect agent memory (commit 1be188b).
+
+## Incremental Ingest and High-Water Marks
+
+Later enhancements introduced high-water mark tracking (commit 2f5ad53) to enable efficient, incremental evidence collection across sessions. Rather than re-processing all historical checkpoints on every run, the system tracks which evidence has been ingested into the active knowledge layer. Repo-specific MongoDB collection names prevent collisions when multiple projects share a single embedding store (commit 7b17590).
+
+## Why Zero Storage
+
+- **Repo Portability**: Clone the repo, get all evidence immediately—no external sync required
+- **Version Control Integration**: Evidence evolves with commits, enabling audit trails and recovery
+- **No Infrastructure**: Works offline, in CI/CD, on local machines without cloud dependencies
+- **Agent Agnostic**: Any tool reading `.reflect/` formats understands the evidence
+
+This architecture is active and used as the foundation for cross-session learning, the wiki layer (v0.6.0), and ongoing memory management enhancements.