docs: repair The monolith section against the current per-chunk streaming architecture - #699
Merged
Merged
Conversation
…7-31 per-chunk architecture PR #666 replaced run_pipeline's pooled-Stage-C-then-bulk-Stage-D shape with a per-chunk dispatch_micro_batch loop but only added a new proposal doc, never updating this living reference. Corrects the pipeline diagram and flags that the loop is still gated by STAGE_E_STREAMING_ENABLED (default False everywhere, including docker-compose.prod.yml), so run_pipeline currently performs no Stage C/D work by default despite its own docstring's write-by-default promise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/features/stage-e-operations.md's "The monolith" section describedrun_pipeline's original 2026-07-30 shape (pooled Stage C subprocess viarun_image_evidence_cohort, then a separate bulk Stage D pass). PR Streaming pipeline default: micro-batch Stage C/D dispatch and verdict transfer #666 (2026-07-31) replaced this with a per-chunkdispatch_micro_batchstreaming loop, but only added a new proposal doc — this existing living reference was never updated. Corrected the pipeline diagram and added a dated note pointing out that_run_stage_c/_run_stage_d_bulk/therun_image_evidence_cohortdelegation this section still describes in detail further down are dead code as of this reading (not reachable fromhandle()under any flag).settings.STAGE_E_STREAMING_ENABLED, which still defaults toFalsein bothsettings.pyanddocker-compose.prod.yml.dispatch_micro_batchchecks that flag unconditionally for every caller, includingrun_pipeline— so with the flag at its default, every chunk the monolith dispatches reportsstatus="disabled"and Stage C/D perform no work at all, while Stage 0/E preflight/channel_reportstill run and the command exits 0. This contradictsrun_pipeline's own module docstring ("a complete, working, from-scratch, whole-catalogue run THAT WRITES... None is a precondition").Test plan
python3 .github/scripts/docs_lint.py --strict— cleanpre-commit run --files docs/features/stage-e-operations.md— prettier, EOF, trailing-whitespace, readme-parity all passedgrep -rn STAGE_E_STREAMING_ENABLEDacrosscardpicker/,settings.py(default=False),docker-compose.prod.yml(${STAGE_E_STREAMING_ENABLED:-False}),dispatch_micro_batch's own gate check, andrun_pipeline.py'shandle()(confirmed_run_stage_c/_run_stage_d_bulkhave zero call sites,_run_streaming_stagesis the only path taken)