diff --git a/.claude/skills/reflect/SKILL.md b/.claude/skills/reflect/SKILL.md index 86199fe..66da567 100644 --- a/.claude/skills/reflect/SKILL.md +++ b/.claude/skills/reflect/SKILL.md @@ -22,7 +22,7 @@ hooks: command: "${CLAUDE_PLUGIN_ROOT}/hooks/session-start.sh" metadata: author: shashwatjain - version: '0.5.2' + version: '0.6.0' --- # Reflect — Repo-Owned Memory diff --git a/.reflect/.last_run b/.reflect/.last_run index cd6be6e..4de246c 100644 --- a/.reflect/.last_run +++ b/.reflect/.last_run @@ -1 +1 @@ -{"last_checkpoint": "af09a9533c27", "last_git_sha": "52ac785", "timestamp": "2026-04-04T18:19:00.728855"} \ No newline at end of file +{"last_checkpoint": "", "last_git_sha": "2f5ad53", "timestamp": "2026-04-09T13:44:40.267220"} \ No newline at end of file diff --git a/.reflect/context.md b/.reflect/context.md index d6d0b7b..04ee4b5 100644 --- a/.reflect/context.md +++ b/.reflect/context.md @@ -1,35 +1,16 @@ # Project Context - - + + ## Key Decisions & Rationale -- **Declarative format.yaml + Claude subagent synthesis** — Replaced executable harness with format.yaml-driven context generation to decouple schema from script maintenance and enable format reuse. Decided 2026-04-04 as core pivot. (checkpoint 68bae57a699d) -- **Zero-storage, read-on-demand architecture** — Fetch all evidence from Entire/git at synthesis time rather than persisting. Avoids staleness/sync at cost of latency. (checkpoint 90e2641946db) -- **Replaceable harness interface** — Design harness as swappable script, enabling future language alternatives (Python, bash, Ruby). (commit 46c8781) -- **GitHub Releases distribution with skill/agents/ bundling** — skill/SKILL.md is canonical source of truth; skill/agents/ is parallel directory for bundled subagents. Both copied into target repo during `reflect init`. (checkpoint af09a9533c27) -- **SessionStart auto-refresh via .last_run state tracking** — Automatically refresh context.md on session start without user action, using per-repo freshness tracking. (checkpoint 90e2641946db) -- **Split 'Open Work' from 'Abandoned Approaches' with revisit_when field** — Separated unfinished items (7d recency) from dead ends (90d recency). Revisit_when makes entries self-expiring rather than permanent blockers. (checkpoint 68bae57a699d) +- **Critical Pitfalls Detection in Context Pipeline** — Agents working with codebases need early, actionable awareness of project pitfalls—common mistakes, architectural constraints, and known failure modes (commit 9cdf813) +- **High-Water Mark Tracking for Incremental Ingest** — High-water mark tracking is a checkpoint-based mechanism that prevents redundant processing when ingesting repository evidence across multiple session (commit 2f5ad53) +- **Keeper Agent: Focused Repo Memory** — The Keeper Agent evolved from a **broad repo memory** approach to a **focused** one, deliberately narrowing its scope to deliver higher-signal knowled (commit 34626f0) +- **Wiki Layer for Persistent Compounding Knowledge** — The wiki layer (v0.6.0) introduces a structured mechanism for agents to build and reference persistent knowledge across sessions, extending the sessio (commit 1be188b) +- **Zero Storage Architecture (v4)** — The Zero Storage Architecture (v4) is a foundational design pattern that eliminates external storage dependencies by storing all agent-generated evide (commit 46c8781) ## Gotchas & Friction -- **Dual SKILL.md copy sync is unautomated** — skill/SKILL.md (source) and .claude/skills/reflect/SKILL.md (installed) must be manually kept in sync with no enforcement or warnings on divergence. (checkpoint 90e2641946db) -- **Entire CLI is hard undocumented dependency** — reflect's real-world value is near-zero without Entire installed, but README does not adequately warn users of this bottleneck. (checkpoint 90e2641946db) -- **Many context.md entries are truncated, vague, or stale** — Signal filtering for "hard-to-derive" facts is aspirational. Results in low-confidence briefings. (checkpoint 90e2641946db) -- **Checker agent returns silent '[CLI error: unknown]'** — No diagnostic content; blocks root cause diagnosis when agent fails. (checkpoint ed2a5817517f) -- **Checker.py lacks markdown fence stripping on JSON parsing** — Model wraps JSON in ```json...``` blocks; parser fails to strip before json.decode(). (checkpoint ed2a5817517f) -- **YAML-lite parser scoping bug discovered post-edit** — in_entry_fields reset on every loop iteration, skipping all but first entry_fields item. Required second read-and-fix pass. (checkpoint 68bae57a699d) - -## Open Work - -- **Merge/overwrite policy for conflicting agent files** — skill/agents/ install may overwrite user-customized agents in .claude/agents/; no conflict resolution defined. (checkpoint af09a9533c27) -- **Verify install.sh includes skill/agents/ in release tarball** — GitHub Releases pipeline may omit agents directory from distributed artifact. (checkpoint af09a9533c27) -- **Strip markdown fences in checker.py before JSON parsing** — Add explicit stripping of ```json...``` blocks before json.loads(). (checkpoint ed2a5817517f) -- **Review CHECKER_SYSTEM_PROMPT formatting instructions** — Explicitly forbid markdown code fences in JSON responses to prevent parser failures. (checkpoint ed2a5817517f) -- **Decouple from Entire CLI dependency** — Add native Claude Code session logs, git history, PR descriptions, and issue trackers as optional evidence sources. (checkpoint 90e2641946db) - -## Critical Pitfalls - -- **mistake**: Declared loop-state variable (in_entry_fields) inside iteration, resetting on every cycle. **consequence**: Only first entry_fields item parsed; rest silently dropped, breaking Abandoned Approaches section. **rule**: Declare state variables outside loop blocks; test parser output independently before using in agents. (checkpoint 68bae57a699d) -- **mistake**: Allowed model to wrap structured JSON in markdown code fences ```json...```. **consequence**: Parser failed to decode, producing default scores (2,2,2,2) and blocking all diagnosis. **rule**: Explicitly strip markdown fences before json.loads(); prompt models to return raw JSON without code block wrapping. (checkpoint ed2a5817517f) -- **mistake**: Error handling returned placeholder '[CLI error: unknown]' with zero diagnostic content. **consequence**: Ground truth signals completely missed, making root cause investigation impossible. **rule**: All error paths must include diagnostic info; never return opaque error placeholders. (checkpoint ed2a5817517f) \ No newline at end of file +- **Repo-Specific Collection Names Prevent Collisions** — When multiple repositories use reflect in a shared environment (monorepo, local dev machine, or shared CI runners), QMD collections with identical nam (commit 7b17590) +- **Symlink Hooks Directory Handling** — When `.claude/hooks` is a symlink to another directory — a common configuration for shared setups or templated deployments — hook installation logic m (commit bddce4c) \ No newline at end of file diff --git a/.reflect/wiki/decisions/critical-pitfalls-detection.md b/.reflect/wiki/decisions/critical-pitfalls-detection.md new file mode 100644 index 0000000..a1a0823 --- /dev/null +++ b/.reflect/wiki/decisions/critical-pitfalls-detection.md @@ -0,0 +1,38 @@ +--- +created: 2026-04-09 +updated: 2026-04-09 +sources: [commit 9cdf813, commit 7aa4e33, commit 1be188b] +tags: [pitfalls, context-pipeline, agents, decision] +status: active +--- + +# Critical Pitfalls Detection in Context Pipeline + +## Motivation + +Agents working with codebases need early, actionable awareness of project pitfalls—common mistakes, architectural constraints, and known failure modes—to avoid costly errors. Manual pitfall documentation is incomplete and stale. The Critical Pitfalls Detection feature (commit 9cdf813) automates identification and surfacing of project-specific pitfalls directly into the context pipeline, ensuring agents receive dynamic, session-aware warnings before attempting changes. + +## Integration with Context Pipeline + +This decision was implemented as part of the broader refactoring toward declarative, subagent-driven context generation (commit 7aa4e33). Rather than static docs, the system analyzes project state and prior evidence to systematically detect: + +- Common architectural violations and antipatterns +- Patterns that have caused issues in previous sessions +- Configuration constraints and fragile dependencies +- Known regressions in specific code paths + +Pitfalls are assembled into context during session start, enriching semantic context without requiring manual updates. This complements the wiki layer (commit 1be188b) by distinguishing persistent, compounding knowledge (wiki) from dynamic, task-aware pitfall warnings (context pipeline). + +## Implementation Details + +Detection logic integrates with the declarative `format.yaml` configuration system and the Claude subagent for context generation. When agents request context, the pipeline evaluates: + +1. Prior checkpoint evidence and session history +2. Repository-specific patterns and failure modes +3. Current codebase state and interdependencies + +High-priority pitfalls are elevated to the top of context blocks, ensuring agents see warnings before proceeding with changes. + +## Status + +Active. Pitfalls detection is part of the standard context pipeline as of v0.6.0 and continues to evolve as more pitfall patterns are identified and catalogued across sessions. diff --git a/.reflect/wiki/decisions/high-water-mark-incremental-ingest.md b/.reflect/wiki/decisions/high-water-mark-incremental-ingest.md new file mode 100644 index 0000000..dfbcfa1 --- /dev/null +++ b/.reflect/wiki/decisions/high-water-mark-incremental-ingest.md @@ -0,0 +1,29 @@ +--- +created: 2026-04-07 +updated: 2026-04-09 +sources: [commit 2f5ad53, commit 1be188b, commit 7b17590] +tags: [incremental-ingest, persistence, session-chaining, performance] +status: active +--- + +# High-Water Mark Tracking for Incremental Ingest + +## Overview + +High-water mark tracking is a checkpoint-based mechanism that prevents redundant processing when ingesting repository evidence across multiple sessions. Rather than re-scanning and re-processing identical code or checkpoints on each session start, the system records the last successfully processed state and resumes from that point on subsequent runs. + +## Why It Matters + +Without high-water mark tracking, each session would re-index all repository content, redundantly extract evidence, and recompute knowledge artifacts. This creates two problems: wasted computational effort and stale or duplicative knowledge in the wiki layer. High-water mark tracking (commit 2f5ad53) solves this by enabling **incremental, stateful ingest** — only new commits, modified files, or novel checkpoints are processed and merged into the persistent knowledge base. + +## Mechanism + +The high-water mark approach records the ID or timestamp of the last successfully ingested item (commit, checkpoint, or collection batch). On the next session, the ingest pipeline queries from that mark onward, filtering out already-processed content. This is particularly important for repo-specific metadata: commit 7b17590 introduced repo-specific qmd (query metadata) collection names to avoid collisions across projects, meaning the high-water mark must also be scoped per repository. + +## Integration with Wiki Layer + +The wiki layer (commit 1be188b, v0.6.0) provides the persistent storage for compounding knowledge. High-water mark tracking ensures that sessions don't just accumulate redundant facts — they add incremental, novel insights. When a new commit is ingested, its evidence contributes fresh context to the wiki; older commits are skipped entirely, reducing memory overhead and search noise. + +## Session Chaining Benefit + +This mechanism directly enables session chaining: because state is tracked across sessions, an agent can run multiple focused tasks in sequence without losing accumulated context. Commits 04dd4dd and 33f3add document session chaining guidance; high-water mark tracking is the underlying infrastructure that makes this efficient rather than wasteful. diff --git a/.reflect/wiki/decisions/keeper-agent-focused-design.md b/.reflect/wiki/decisions/keeper-agent-focused-design.md new file mode 100644 index 0000000..8b41e5f --- /dev/null +++ b/.reflect/wiki/decisions/keeper-agent-focused-design.md @@ -0,0 +1,43 @@ +--- +created: 2026-04-09 +updated: 2026-04-09 +sources: + - commit 34626f0 + - commit fd5f2a2 + - commit 0500d79 + - commit 1be188b + - commit 2f5ad53 + - commit 7b17590 +tags: [keeper-agent, memory-scope, architecture, decisions] +status: active +--- + +# Keeper Agent: Focused Repo Memory + +The Keeper Agent evolved from a **broad repo memory** approach to a **focused** one, deliberately narrowing its scope to deliver higher-signal knowledge that agents can reliably act on. This shift emerged after testing revealed that comprehensive tracking produces stale, unfocused artifacts. + +## The Pivot: From Breadth to Depth + +Keeper was initially rewritten as a broad repository chronicler (commit 34626f0), attempting to track all state changes and generate comprehensive summaries. However, practical testing showed this approach generated low-signal noise—agents struggled to distinguish critical insights from routine updates. + +The refined Keeper agent (commit fd5f2a2) instead constrains its mandate to **high-impact, decision-driven knowledge**: architecture decisions, critical pitfalls, session learnings, and evidence snapshots. Generic file inventories and broad summaries were traded for focused depth. + +## Implementation: Incremental and Scoped + +The focused design leverages several technical patterns: + +- **Incremental ingestion** with high-water mark tracking (commit 2f5ad53) prevents duplicate work across sessions, allowing Keeper to resume where it left off +- **Persistent wiki compilation** (commit 1be188b) transforms evidence into indexed markdown pages that compound knowledge over time +- **Critical pitfall detection** surfaces hard blockers early in the context pipeline (related to commit 0500d79) +- **Repo-specific namespacing** via MongoDB collection names (commit 7b17590) isolates data across multi-project setups, avoiding collisions + +## What "Focused" Means + +Rather than attempting to be a universal repository chronicle, Keeper now functions as a **scoped memory layer**: + +1. **Selective ingestion**: captures only patterns with lasting impact or cross-session relevance +2. **Low maintenance overhead**: incremental state tracking eliminates reprocessing +3. **Compounded learning**: wiki pages persist and grow richer across sessions +4. **Decision-centric**: prioritizes actionable insights over comprehensive documentation + +This focused scope makes Keeper maintainable, reliable, and genuinely useful to consuming agents—a decision log rather than an encyclopedia. diff --git a/.reflect/wiki/decisions/wiki-layer-persistent-knowledge.md b/.reflect/wiki/decisions/wiki-layer-persistent-knowledge.md new file mode 100644 index 0000000..b339d90 --- /dev/null +++ b/.reflect/wiki/decisions/wiki-layer-persistent-knowledge.md @@ -0,0 +1,42 @@ +--- +created: 2026-04-09 +updated: 2026-04-09 +sources: [commit 1be188b, commit 2f5ad53, commit 7b17590, commit 33f3add] +tags: [wiki, persistence, knowledge-accumulation, v0.6.0] +status: active +related: [decisions/keeper-agent-repo-memory, features/cross-session-learning] +--- + +# Wiki Layer for Persistent Compounding Knowledge + +## Overview + +The wiki layer (v0.6.0) introduces a structured mechanism for agents to build and reference persistent knowledge across sessions, extending the session-based evidence capture model established in earlier versions. Rather than treating knowledge as ephemeral checkpoints scoped to individual sessions, the wiki layer enables agents to compile, curate, and incrementally refine knowledge artifacts that compound over time (commit 1be188b). + +## Problem Statement + +Prior to v0.6.0, cross-session knowledge relied on checkpoint references within session logs. Agents could query past evidence but had no formal way to synthesize insights into evolving, canonical documents. This created friction when: +- Multiple agents need shared, authoritative facts about the repository +- Knowledge evolves through repeated refinement across sessions +- Agents must manually re-discover or re-synthesize insights from scattered checkpoints + +## Architecture + +The wiki layer uses **incremental ingest with high-water mark tracking** (commit 2f5ad53). Rather than re-processing all historical evidence on each session, agents maintain a pointer to the last processed checkpoint and only ingest new evidence since that point. This enables: + +1. **Efficiency**: Avoid redundant processing of old sessions +2. **Idempotency**: Agents can safely re-run without duplicating prior work + +To prevent collision across multiple repositories using the same centralized storage, the wiki uses **repo-specific QMD collection names** (commit 7b17590), qualifying each repository's knowledge artifacts by a stable repository identifier. + +## Integration with Keeper Agent + +The wiki layer complements the Keeper agent (repo memory agent). While Keeper operates within a single session to understand the current repository state, the wiki layer allows Keeper and other agents to deposit curated knowledge for future sessions to build upon. Cross-session learning enhancements (commit 33f3add) document how agents should use both mechanisms in concert. + +## Key Distinctions + +- **Checkpoints**: Session-scoped, event-driven evidence capture (existing) +- **Wiki**: Repository-scoped, agent-curated synthesis of patterns, decisions, and lessons (new) +- **High-water mark**: Enables incremental reads without re-processing history + +The wiki layer deliberately separates the *collection* of facts (via checkpoints) from the *curation* of knowledge (via wiki articles), allowing human reviewers and agents to manage knowledge quality independently. diff --git a/.reflect/wiki/decisions/zero-storage-architecture.md b/.reflect/wiki/decisions/zero-storage-architecture.md new file mode 100644 index 0000000..0c464e4 --- /dev/null +++ b/.reflect/wiki/decisions/zero-storage-architecture.md @@ -0,0 +1,35 @@ +--- +created: 2026-04-03 +updated: 2026-04-09 +sources: [commit 46c8781, commit 1be188b, commit 2f5ad53, commit 7b17590] +tags: [architecture, storage, harness, design] +status: active +related: [decisions/wiki-layer-v06, decisions/evidence-format-spec] +--- + +# Zero Storage Architecture (v4) + +The Zero Storage Architecture (v4) is a foundational design pattern that eliminates external storage dependencies by storing all agent-generated evidence and metadata directly within the repository as first-class artifacts (commit 46c8781). This approach enables portable, agent-agnostic knowledge persistence without requiring separate databases, cloud services, or sidecar storage systems. + +## Core Principle: Replaceable Harness + +The "replaceable harness" pattern decouples the evidence format and schema from the runtime that generates it (commit 46c8781). The harness—any CLI, script, or agent framework that executes workflows—can be upgraded, swapped, or reimplemented entirely without breaking downstream consumers of the evidence. All information is serialized to stable formats (YAML checkpoints, structured metadata) that persist independently of the harness implementation. + +This enables Claude or other agents to read evidence written by the current CLI, and future agents to read evidence written today, regardless of how the harness evolves. + +## Storage Location and Structure + +Evidence is stored in `.reflect/` directory at the repository root, organized as checkpoints and session metadata. Each session creates a timestamped record; each checkpoint captures execution state, reasoning, and outputs. By storing everything in the repo, the evidence benefits from version control, snapshots, and distributed access—the same mechanisms protecting source code now protect agent memory (commit 1be188b). + +## Incremental Ingest and High-Water Marks + +Later enhancements introduced high-water mark tracking (commit 2f5ad53) to enable efficient, incremental evidence collection across sessions. Rather than re-processing all historical checkpoints on every run, the system tracks which evidence has been ingested into the active knowledge layer. Repo-specific MongoDB collection names prevent collisions when multiple projects share a single embedding store (commit 7b17590). + +## Why Zero Storage + +- **Repo Portability**: Clone the repo, get all evidence immediately—no external sync required +- **Version Control Integration**: Evidence evolves with commits, enabling audit trails and recovery +- **No Infrastructure**: Works offline, in CI/CD, on local machines without cloud dependencies +- **Agent Agnostic**: Any tool reading `.reflect/` formats understands the evidence + +This architecture is active and used as the foundation for cross-session learning, the wiki layer (v0.6.0), and ongoing memory management enhancements. diff --git a/.reflect/wiki/gotchas/repo-specific-collection-names.md b/.reflect/wiki/gotchas/repo-specific-collection-names.md new file mode 100644 index 0000000..2244182 --- /dev/null +++ b/.reflect/wiki/gotchas/repo-specific-collection-names.md @@ -0,0 +1,33 @@ +--- +created: 2026-04-09 +updated: 2026-04-09 +sources: [commit 7b17590, commit 1be188b, commit 2f5ad53] +tags: [multi-repo, collections, naming] +status: active +--- + +# Repo-Specific Collection Names Prevent Collisions + +## The Problem + +When multiple repositories use reflect in a shared environment (monorepo, local dev machine, or shared CI runners), QMD collections with identical names will collide. Two separate projects, each with their own `.reflect/` directory, might both create a collection named `docs` or `codebase`. Without namespacing, evidence from different repos mixes in the same collection, causing cross-contamination during ingestion, search, and wiki compounding. + +## The Solution + +Reflect applies a repo-identifier prefix to all QMD collection names (commit 7b17590). This ensures each repository's collections are namespaced uniquely: `project-a/docs` vs. `project-b/docs`, keeping evidence strictly isolated. + +This is critical for workflows relying on persistent collection state: +- **High-water mark tracking** (commit 2f5ad53): incremental ingestion must not conflict across repos; if two repos wrote to the same collection, already-ingested evidence would be incorrectly re-marked +- **Wiki layer** (commit 1be188b): compounding knowledge must draw only from the correct repo's collected evidence, not leaked evidence from parallel runs + +## When This Matters + +**Monorepos**: multiple services or components may each run reflect agents; collections must be scoped per service to avoid mixing evidence. + +**Local dev machines**: a developer working on multiple projects simultaneously will have separate `.reflect/` directories; QMD engines must not cross-pollinate collections between them. + +**Parallel CI/CD**: test runs or deployments across repos executing concurrently must maintain isolated collection state. Without repo-specific names, concurrent writes to the same collection cause race conditions and data loss. + +## Implementation Detail + +The repo identifier (typically derived from the repo slug or hash) prefixes every collection name before it reaches the QMD backend. All subsequent operations—ingestion, search, wiki compounding—work within the scoped namespace transparently. diff --git a/.reflect/wiki/gotchas/symlink-hooks-directory.md b/.reflect/wiki/gotchas/symlink-hooks-directory.md new file mode 100644 index 0000000..40dffa8 --- /dev/null +++ b/.reflect/wiki/gotchas/symlink-hooks-directory.md @@ -0,0 +1,36 @@ +--- +created: 2026-04-09 +updated: 2026-04-09 +sources: [commit bddce4c] +tags: [hooks, symlink, installation, gotcha] +status: active +--- + +# Symlink Hooks Directory Handling + +When `.claude/hooks` is a symlink to another directory — a common configuration for shared setups or templated deployments — hook installation logic must explicitly resolve the symlink to its real path. Naive path comparisons or file operations will either follow the symlink silently and write to the wrong location, or fail to recognize hooked paths as valid. (commit bddce4c) + +## The Issue + +If your project symlinks `.claude/hooks` to an external or shared directory, installer code that doesn't canonicalize paths will create inconsistent state: + +- File operations follow the symlink transparently, writing hooks to the target directory instead of maintaining the `.claude/hooks` interface. +- Path existence or membership checks using string matching (e.g., `if '.claude/hooks' in path`) fail when the actual resolved path differs from the symlink. +- Validation logic sees different paths depending on whether it uses the symlink or the real path, leading to "hooks not found" errors despite correct installation. + +## Solution + +Always resolve symlinks to their canonical filesystem location before any path comparison or installation operation: + +```python +import os +canonical_hooks_dir = os.path.realpath('.claude/hooks') +# Now use canonical_hooks_dir for all file ops and path checks +``` + +This ensures: +- Hook files are written to the correct physical location. +- Path validation consistently recognizes symlinked hooks directories. +- Upgrade and initialization routines find and manage existing hooks correctly. + +The fix is essential if you support `.claude` as a shared package, monorepo submodule, or any setup where users symlink directories for deduplication. diff --git a/.reflect/wiki/log.md b/.reflect/wiki/log.md new file mode 100644 index 0000000..beaf59f --- /dev/null +++ b/.reflect/wiki/log.md @@ -0,0 +1,9 @@ + +## [2026-04-09] ingest | 7 page(s) — 7 created, 0 updated, 0 resolved +- create decisions/high-water-mark-incremental-ingest.md +- create decisions/wiki-layer-persistent-knowledge.md +- create decisions/zero-storage-architecture.md +- create decisions/critical-pitfalls-detection.md +- create decisions/keeper-agent-focused-design.md +- create gotchas/repo-specific-collection-names.md +- create gotchas/symlink-hooks-directory.md diff --git a/lib/context.py b/lib/context.py index ca13a75..f19dd7e 100644 --- a/lib/context.py +++ b/lib/context.py @@ -234,7 +234,9 @@ def _synthesize_context(evidence_doc, fmt, verbose=False): "--max-turns", "1", "--tools", "", "--max-budget-usd", MAX_BUDGET, - "--append-system-prompt", system_prompt, + "--setting-sources", "", + "--no-session-persistence", + "--system-prompt", system_prompt, ] if verbose: diff --git a/lib/ingest.py b/lib/ingest.py index d140352..7a5a216 100644 --- a/lib/ingest.py +++ b/lib/ingest.py @@ -9,6 +9,7 @@ import shutil import subprocess import sys +from concurrent.futures import ThreadPoolExecutor, as_completed from datetime import datetime from pathlib import Path @@ -81,7 +82,9 @@ def _call_subagent(prompt, system_prompt, verbose=False, step_name=""): "--max-turns", "1", "--tools", "", "--max-budget-usd", INGEST_BUDGET, - "--append-system-prompt", system_prompt, + "--setting-sources", "", + "--no-session-persistence", + "--system-prompt", system_prompt, ] label = f"[ingest/{step_name}]" if step_name else "[ingest]" @@ -296,45 +299,58 @@ def _write_page_content(evidence_doc, action_type, page_info, existing_content, def _batch_write(evidence_doc, actions, action_type, wiki_dir, today, verbose=False): - """Write pages in batches of up to 5. + """Write pages concurrently using a thread pool. Returns list of (item, raw_content, effective_action_type) tuples. Each action is a dict from the triage plan. """ - BATCH_SIZE = 5 + MAX_WORKERS = 5 results = [] - for i in range(0, len(actions), BATCH_SIZE): - batch = actions[i:i + BATCH_SIZE] - for item in batch: - action_type_eff = action_type - existing = None - - if action_type in ("update", "resolve"): - page_path = wiki_dir / item["path"] - if page_path.exists(): - existing = page_path.read_text() - else: - if verbose: - print( - f" [ingest/write] page not found: {item['path']}, treating as create", - file=sys.stderr, - ) - action_type_eff = "create" - # Fill in category/title from path for create fallback - parts = item["path"].split("/") - item = dict(item) - if "category" not in item: - item["category"] = parts[0] if parts else "general" - if "title" not in item: - item["title"] = parts[-1].replace(".md", "").replace("-", " ").title() - if "slug" not in item: - item["slug"] = parts[-1].replace(".md", "") if parts else "page" - - raw = _write_page_content( - evidence_doc, action_type_eff, - item, existing, today, verbose=verbose, - ) + def _write_one(item): + action_type_eff = action_type + existing = None + + if action_type in ("update", "resolve"): + page_path = wiki_dir / item["path"] + if page_path.exists(): + existing = page_path.read_text() + else: + if verbose: + print( + f" [ingest/write] page not found: {item['path']}, treating as create", + file=sys.stderr, + ) + action_type_eff = "create" + # Fill in category/title from path for create fallback + parts = item["path"].split("/") + item_copy = dict(item) + if "category" not in item_copy: + item_copy["category"] = parts[0] if parts else "general" + if "title" not in item_copy: + item_copy["title"] = parts[-1].replace(".md", "").replace("-", " ").title() + if "slug" not in item_copy: + item_copy["slug"] = parts[-1].replace(".md", "") if parts else "page" + return item_copy, action_type_eff, existing + + return item, action_type_eff, existing + + # Prepare items (read existing content) — fast, sequential + prepared = [_write_one(item) for item in actions] + + # Call subagent for each page concurrently + def _call_one(prep): + item, action_type_eff, existing = prep + raw = _write_page_content( + evidence_doc, action_type_eff, + item, existing, today, verbose=verbose, + ) + return item, raw, action_type_eff + + with ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool: + futures = {pool.submit(_call_one, p): p for p in prepared} + for future in as_completed(futures): + item, raw, action_type_eff = future.result() if raw: results.append((item, raw, action_type_eff)) else: