Streaming pipeline default: micro-batch Stage C/D dispatch and verdict transfer - #666
Merged
Merged
Conversation
…tch and verdict transfer
This was referenced Aug 3, 2026
Open
WilfordGrimley
added a commit
that referenced
this pull request
Aug 4, 2026
Three defects, previously making stage_d_verdict_transfer_votes sum to 0
across every stage-e-fullcat-b25-* PilotRunLedger row (5,028 measured):
1. stream_full_catalog stamped a distinct run_id per batch
(run_id_prefix-{batch_num}), so _partition_by_md5_verdict's
already-voted read (scoped to run_id) could never see a vote cast by
an earlier batch of the same pass. run_id is now stable for the whole
pass; ledger_run_id (already threaded through dispatch_micro_batch by
PR #666) decouples the PilotRunLedger row identity instead.
2. md5_groups was built from batch members only, so a byte-identical
twin outside the batch (the ordinary case for keyset-paginated
batches) could never be reached. _partition_by_md5_verdict now
returns DB-wide groups for checksums that already have a vote this
pass.
3. members_already_voted was passed as an unconditional empty set, so a
card already holding its own vote for an agent could be purged and
overwritten by the propagated copy. _drain_verdict_transfer_queue now
passes the true per-agent set.
3 tasks
WilfordGrimley
added a commit
that referenced
this pull request
Aug 5, 2026
…7-31 per-chunk architecture (#699) PR #666 replaced run_pipeline's pooled-Stage-C-then-bulk-Stage-D shape with a per-chunk dispatch_micro_batch loop but only added a new proposal doc, never updating this living reference. Corrects the pipeline diagram and flags that the loop is still gated by STAGE_E_STREAMING_ENABLED (default False everywhere, including docker-compose.prod.yml), so run_pipeline currently performs no Stage C/D work by default despite its own docstring's write-by-default promise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run_pipeline.py& snipstage_e_dispatch.py): Promoted streaming Stage E micro-batch dispatch (dispatch_micro_batch) to be the default runtime path formanage.py run_pipeline. Removed theSTAGE_E_STREAMING_ENABLEDgate so streaming runs unconditionally._run_streaming_stages()with fetch-ahead Stage C worker threads and MD5 verdict-transfer optimization (_partition_by_md5_verdict,_drain_verdict_transfer_queue) to avoid redundant extractions for identical card prints.test_run_pipeline.py): Threadeddry_runparameter through micro-batching, Stage C, and Stage D. Added test stubs and updated test suites (test_run_pipeline.py) so all 31 end-to-end monolith integration tests pass successfully.Test plan
test_run_pipeline.py): all 31 tests passed successfully.mypyon touched modules (stage_e_dispatch.py,run_pipeline.py): 0 errors found.ruff,black,isort,prettier): all passed.Task-end checks (CLAUDE.md)
docs/proposals/pipeline-batching-and-verdict-transfer.md.