Scan resilience: model preflight, shared per-post path, crash-safe memory#16
Merged
Merged
Conversation
…mory Three robustness/maintainability improvements to scan mode: 1. Crash-safe incremental memory (was: saved once at end of run). A local LLM scan can run for minutes; a kill / reboot / OOM mid-run discarded every analysis since the last run, so the next run re-burned the model over all of them. cmd_scan now checkpoints memory every SCAN_SAVE_EVERY (10) newly-analyzed posts AND in a finally, so an interrupted scan keeps its finished work. Writes are atomic (existing _atomic_json_write). 2. Model-availability preflight. The startup check only confirmed the Ollama daemon was reachable; if the requested model wasn't pulled, every call_ollama returned None and the whole run silently no-op'd with per-post errors. ensure_model_available() now checks /api/tags and fails fast with "ollama pull <model>" — wired into both scan and webhook entry points. 3. Extracted _process_post(). The single-post (--post-id) and bulk-scan branches duplicated the analyze -> stamp -> record -> act pipeline; one shared helper removes the drift risk and makes the per-post path unit- testable. Pinned by tests/test_scan_resilience.py (12 tests): preflight match/miss/ daemon-down, _process_post success/dry-run/model-failure, and the checkpoint cadence + that a mid-scan crash still persists completed work. Full suite: 130 passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three scan-mode resilience + maintainability improvements (follow-up to the timeout fix).
1. Crash-safe incremental memory
Scan mode saved
colony_analyzed.jsononce, at the end of a run. A local-LLM scan can run for minutes, so a kill / reboot / OOM mid-run discarded every analysis since the last run — the next run then re-ran the model over all of them.cmd_scannow checkpoints everySCAN_SAVE_EVERY(10) newly-analyzed posts and again in afinally, so an interrupted scan keeps its finished work. Writes stay atomic.2. Model-availability preflight
The startup check only confirmed the Ollama daemon was reachable. If the requested model wasn't pulled, every
call_ollamareturnedNoneand the whole run silently no-op'd.ensure_model_available()now checks/api/tagsand fails fast withollama pull <model>— wired into bothscanandwebhook.3. Extracted
_process_post()The single-post (
--post-id) and bulk-scan branches duplicated the analyze → stamp → record → act pipeline. One shared helper removes drift risk and makes the per-post path unit-testable.Tests
tests/test_scan_resilience.py(12 tests): preflight match/miss/daemon-down,_process_postsuccess/dry-run/model-failure, the checkpoint cadence, and that a mid-scan crash still persists completed work. Full suite: 130 passing.🤖 Generated with Claude Code