Skip to content

docs: repair The monolith section against the current per-chunk streaming architecture - #699

Merged
WilfordGrimley merged 1 commit into
masterfrom
docs-audit-directive-narration
Aug 5, 2026
Merged

docs: repair The monolith section against the current per-chunk streaming architecture#699
WilfordGrimley merged 1 commit into
masterfrom
docs-audit-directive-narration

Conversation

@WilfordGrimley

Copy link
Copy Markdown

Summary

  1. docs/features/stage-e-operations.md's "The monolith" section described run_pipeline's original 2026-07-30 shape (pooled Stage C subprocess via run_image_evidence_cohort, then a separate bulk Stage D pass). PR Streaming pipeline default: micro-batch Stage C/D dispatch and verdict transfer #666 (2026-07-31) replaced this with a per-chunk dispatch_micro_batch streaming loop, but only added a new proposal doc — this existing living reference was never updated. Corrected the pipeline diagram and added a dated note pointing out that _run_stage_c/_run_stage_d_bulk/the run_image_evidence_cohort delegation this section still describes in detail further down are dead code as of this reading (not reachable from handle() under any flag).
  2. Same note records a second, verified finding: the per-chunk loop is still gated by settings.STAGE_E_STREAMING_ENABLED, which still defaults to False in both settings.py and docker-compose.prod.yml. dispatch_micro_batch checks that flag unconditionally for every caller, including run_pipeline — so with the flag at its default, every chunk the monolith dispatches reports status="disabled" and Stage C/D perform no work at all, while Stage 0/E preflight/channel_report still run and the command exits 0. This contradicts run_pipeline's own module docstring ("a complete, working, from-scratch, whole-catalogue run THAT WRITES... None is a precondition").

Test plan

  • python3 .github/scripts/docs_lint.py --strict — clean
  • pre-commit run --files docs/features/stage-e-operations.md — prettier, EOF, trailing-whitespace, readme-parity all passed
  • Every claim in the added note traced to code, not the PR descriptions that reported it: grep -rn STAGE_E_STREAMING_ENABLED across cardpicker/, settings.py (default=False), docker-compose.prod.yml (${STAGE_E_STREAMING_ENABLED:-False}), dispatch_micro_batch's own gate check, and run_pipeline.py's handle() (confirmed _run_stage_c/_run_stage_d_bulk have zero call sites, _run_streaming_stages is the only path taken)

…7-31 per-chunk architecture

PR #666 replaced run_pipeline's pooled-Stage-C-then-bulk-Stage-D shape with
a per-chunk dispatch_micro_batch loop but only added a new proposal doc,
never updating this living reference. Corrects the pipeline diagram and
flags that the loop is still gated by STAGE_E_STREAMING_ENABLED (default
False everywhere, including docker-compose.prod.yml), so run_pipeline
currently performs no Stage C/D work by default despite its own docstring's
write-by-default promise.
@WilfordGrimley
WilfordGrimley merged commit 566a8cb into master Aug 5, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant