Refactor batch write to use concurrent ThreadPoolExecutor - #9
Merged
Conversation
Subagent calls (claude -p) were broken because project/user hooks (Entire CLI, stop-hook-git-check) injected messages about uncommitted changes, causing the model to respond to those instead of the actual triage/write task. Added --setting-sources "" and --no-session-persistence to isolate subagent calls. Switched from --append-system-prompt to --system-prompt to avoid loading the default Claude Code system prompt. Also parallelized wiki write calls using ThreadPoolExecutor (up to 5 concurrent workers) instead of sequential batching. With 7 pages, this reduces write stage from ~56s to ~15s. Measured results: ingest goes from broken/66s to working/77s. Per-call API time drops from ~13s to ~3-7s. https://claude.ai/code/session_017Apv1ffJUpBe6xEmEd45Md Entire-Checkpoint: f5406daa8b23
Generated by running `reflect init` + `reflect ingest` after the subagent isolation and parallel writes fix. 7 wiki pages created covering decisions and gotchas from project history. https://claude.ai/code/session_017Apv1ffJUpBe6xEmEd45Md Entire-Checkpoint: e278aa00b4e6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactored the
_batch_writefunction inlib/ingest.pyto use concurrent execution viaThreadPoolExecutorinstead of sequential batch processing. This enables parallel page writes, improving throughput when processing multiple wiki pages.Key Changes
ThreadPoolExecutor(max_workers=5)to write up to 5 pages in parallel_write_page_contentfor each prepared item concurrently usingas_completed()_write_one()to handle item preparation and_call_one()to handle concurrent page writeslib/ingest.pyandlib/context.py:--append-system-promptto--system-prompt--setting-sources ""and--no-session-persistenceflagsImplementation Details
The refactoring maintains the same logic for handling update/resolve/create actions and fallback behavior when pages don't exist, but executes the expensive
_write_page_contentcalls (which invoke subagents) concurrently rather than sequentially. The preparation phase (reading existing content) remains sequential since it's I/O bound and relatively fast.The concurrent approach preserves result ordering and error handling by collecting futures and processing results as they complete via
as_completed().https://claude.ai/code/session_017Apv1ffJUpBe6xEmEd45Md