Add Stage E Phase 2 streaming dispatch loop

[WilfordGrimley]

Description

Stage E Phase 2 (issue #153): the streaming dispatch loop that Phase 1's
envelope primitive (cardpicker/operating_envelope.py, PR #440) had no
caller for. Design authority: docs/proposals/stage-e-streaming.md
(still HOLD as a brief; this is the owner-approved implementation task
for its §3-§5, same posture Phase 1 shipped under).

Ships default-OFF (settings.STAGE_E_STREAMING_ENABLED = False) -
every mechanism below is live code, wired unconditionally, but every
entry point checks the flag first and is a no-op while it's False.
Turning it on in production is the phase-3 shakedown's own polled owner
action, not something this PR does.

What shipped

1. Trigger - event-driven post_save receivers on Card
   (created=True only, "card-create") and ImageEvidence (every save,
   "evidence-change"), queued via django-q2 async_task
   (cardpicker/stage_e_signals.py, wired in apps.py's ready()),
   plus a cron backstop sweep (manage.py stream_backstop_sweep) that
   re-runs the same eligibility selectors against the full backlog.
2. Micro-batching - _select_micro_batch builds a bounded card-id
   list per dispatch (seed card first, filled from the Stage C backlog),
   sized by settings.STAGE_E_MICRO_BATCH_SIZE (default 25, a
   placeholder inside the brief's own 10-100 sanity range - the real
   number is a measured output of the phase-3 Bug-A shakedown, per
   §10(c), not invented here).
3. Rate control - load-governed via the envelope primitive: every
   dispatch samples live host load/RSS/fetch-failure-rate and refuses
   to proceed if operating_envelope.current_trip() is open or a fresh
   sample breaches a bar. No artificial cards/hour cap.
4. Consensus recompute - satisfied for free: the existing Stage D
   calculator entry points (run_join_key_calculator/
   run_fallback_calculator/run_slow_path_calculator) already call
   resolve_and_persist_printing per touched card internally - scoping
   them to the micro-batch via a new optional card_ids parameter
   (additive, None = unchanged BULK-mode behaviour) scopes their
   consensus recompute to the same set, with no separate consensus step
   in the dispatcher.
5. Pipeline stages - the conveyor (cardpicker/stage_e_dispatch.py)
   calls the existing Stage C (compute_card_evidence/persist_evidence)
   and Stage D entry points AS-IS, sequentially per micro-batch. No
   extraction/calculator/consensus logic touched.
6. Observability - one PilotRunLedger row per micro-batch
   (command="stage_e_streaming_dispatch"), counters incl.
   peak_rss_mb, stage_c_completed, stage_d_*_votes, trigger_reason,
   lockout_trip_id. Documented in docs/features/stage-e-operations.md's
   new "Phase 2" section.

Two binding Tron-gate notes from Phase 1's review are honored: the
fetch-failure rolling window is sized to FETCH_FAILURE_WINDOW (500)
exactly (_FetchOutcomeWindow), and the loop refuses to dispatch
while current_trip() is non-None, full stop - no self-resume anywhere
in this module.

BULK-mode commands (run_image_evidence_cohort, local_calculate_verdicts,
reparse_collector_evidence, consensus_recompute) are untouched -
none of their call sites pass the new card_ids parameter, so their
behaviour is byte-identical to before this change.

Also fixes a pre-existing mypy type-arg gap in operating_envelope.py
(Phase 1) that was blocking this PR's own pre-commit hook - a type
annotation only, no behaviour change.

Checklist

• I have installed pre-commit and installed the hooks with pre-commit install before creating any commits.
• I have updated any related tests for code I modified or added new tests where appropriate.
• I have manually tested my changes as follows:
  • Full new test suite (cardpicker/tests/test_stage_e_dispatch.py, 20 tests) covering: default-off gate, envelope-trip halt + no-self-resume + resume-after-acknowledge, the 500-card fetch-failure window sizing (including a boundary check tied to check_envelope's own ratified math), micro-batch selection, an end-to-end event->batch->Stage C+Stage D->ledger test, a Google-lockout-mid-batch trip test, a kill-safety/resume-contract test (mid-batch crash -> truthful FAILED ledger -> idempotent re-invocation), and backstop-sweep idempotence.
  • Ran that suite plus test_operating_envelope.py, test_local_calculate_verdicts.py, test_run_image_evidence_cohort.py, test_reparse_collector_evidence.py, and test_image_evidence.py together (336 tests) against the pre-provisioned host venv (docs/troubleshooting.md's documented safe pattern - throwaway testcontainers Postgres/ES, never the live prod stack) - all green.
  • black/isort/ruff/mypy clean on every changed file (via the actual pre-commit hook on this commit).
  • docs_lint.py clean.
  • Deferred: test_vote_consensus.py/test_views.py/test_import_sources.py hit confirmed testcontainers port contention from a concurrent session on this shared box (docs/troubleshooting.md's documented failure mode, not a regression) - not re-run before this PR; none of those files touch code this PR modifies (I never edit vote_consensus.py/printing_consensus.py/PROTECTED CORE).
• I have updated any relevant documentation or created new documentation where appropriate.
  • docs/features/stage-e-operations.md - new "Phase 2" section (trigger, batching, observability, resume contract) plus a "Phase 3 (not yet built)" section replacing the old "What Phase 2/3 will add" stub.
  • docs/proposals/stage-e-streaming.md - a "Phase 2 built" pointer paragraph alongside the existing Phase 1 one.
  • docs/README.md - updated the stage-e-operations.md index entry to mention Phase 2.

Closes #153 (Phase 2 of the brief - Phase 3, the live shakedown/actual
enablement, remains open and untouched by this PR).

🤖 Generated with Claude Code