Skip to content

feat(stage-e): add host-load soft brake, throttle on approach to the 7.0 ceiling - #717

Merged
WilfordGrimley merged 2 commits into
masterfrom
stage-e-host-load-soft-brake
Aug 6, 2026
Merged

feat(stage-e): add host-load soft brake, throttle on approach to the 7.0 ceiling#717
WilfordGrimley merged 2 commits into
masterfrom
stage-e-host-load-soft-brake

Conversation

@WilfordGrimley

Copy link
Copy Markdown

Description

operating_envelope.HOST_LOAD_CEILING (7.0) is a binary cliff: the instant
_bar_breach reads load_avg > 7.0, check_envelope persists an
EnvelopeTrip, dispatch_micro_batch returns halted-new-trip, and the
driver exits 3 with no self-resume — a fresh human resolve_envelope_trip
action is required to continue. Two full-catalogue passes tripped on narrow
overshoots a day apart (7.0796 on 2026-08-04, then 7.17236328125 on
2026-08-05 — 1.1% and 2.5% over) despite the box otherwise running well
under load, each costing a stopped pipeline and a human interaction. There
was no gradation between "full speed" and "hard stop."

This PR adds cardpicker.stage_e_load_brake: a soft brake inserted in
dispatch_micro_batch, between the existing no-self-resume gate and the
fresh envelope sample. It re-samples os.getloadavg() on its own account
and, while the reading sits between STAGE_E_HOST_LOAD_SOFT_CEILING
(default 6.0) and the hard ceiling, sleeps with jitter and re-samples
rather than proceeding straight to the trip check. A load already above the
hard ceiling stops braking at once and falls through to the envelope's own
fresh sample, which trips exactly as before — the brake never itself
decides "trip" and never suppresses a genuine breach.
Cumulative wait
past STAGE_E_LOAD_BRAKE_MAX_WAIT_S (default 240s, ~4 time constants of
the 1-minute load average's own decay) proceeds anyway — best-effort, never
a deadlock. All three settings default to values that make the brake
active out of the box, no operator opt-in required.

Every dispatcher pausing independently at its own next batch boundary is
what makes this reduce load rather than just delay it: the pass
self-throttles its own effective concurrency down to whatever fits under
the ceiling, continuously, instead of running at full concurrency until it
hits the wall and halts.

DispatchOutcome gains load_brake_waits/load_brake_seconds, merged
into PilotRunLedger.counters on every completed micro-batch, so the
brake's effect is queryable (zero on a quiet box, non-zero under
contention) rather than invisible.

Envelope semantics are unchanged. operating_envelope.py is untouched
— the hard ceiling, the trip row, the no-self-resume gate, and
current_trip's query all stay byte-identical. stage_e_batch_sizing.py
is untouched — its duration term still derives from the ceiling constant,
not a live sample, for the reproducibility reason stated in its own
docstring.

Docs: docs/features/stage-e-operations.md's "Ordering, every dispatch
call" list gains the brake as a step (renumbering the rest), a new "The
host-load soft brake" subsection covers the mechanism/settings/
observability/failure posture, and the "Observability" section's counters
list gains the two new fields.

Checklist

  • I have installed pre-commit and installed the hooks with
    pre-commit install before creating any commits.
  • I have updated any related tests for code I modified or added new
    tests where appropriate.
  • I have manually tested my changes as follows:
    • New file cardpicker/tests/test_stage_e_load_brake.py (19 tests):
      brake_decision's band boundaries (including that the hard boundary
      itself still waits rather than tripping, and that an unreadable load
      proceeds rather than blocking); run_load_brake's injected-loop
      behaviour with exact wait counts/durations asserted (below-soft = zero
      waits, in-band = the specific value, above-hard = zero waits/no sleep
      call at all, cumulative-wait-past-bound = proceeds anyway, re-sampling
      happens every iteration, jitter is applied to every sleep);
      apply_load_brake's settings-integration behaviour against the real
      settings.py defaults (not hand-picked test values), including a
      malformed setting and an os.getloadavg failure both degrading to
      unbraked rather than raising.
    • Ran the new file alone (19/19 passed) and together with the other five
      Stage E test files per this repo's own issue Test suite is order-dependent: leaked fetch-failure window trips the envelope across files (8 failures on master) #679 order-dependency
      caution — 199/199 passed combined, no cross-file interaction found.
    • Ran test_stage_e_dispatch.py alone (88/88 passed) to confirm the
      wiring change didn't regress the existing dispatch suite.
    • Mutation-checked five of the new tests by hand: disabled the brake
      entirely (11 tests correctly failed), widened the hard-ceiling
      boundary to >= (the boundary test correctly failed), dropped the
      jitter multiplier (the jitter test correctly failed), removed the
      try/except in apply_load_brake (the malformed-setting test
      correctly raised instead of degrading), and effectively disabled the
      max-wait bound (both tests exercising it correctly failed with
      the wrong wait/second counts). Reverted each mutation and re-ran the
      full new-file suite clean (19/19) after every revert.
    • mypy --config-file mypy.ini clean on all four touched/added Python
      files. black, isort --profile black, ruff, and docs_lint.py
      all clean. Full pre-commit hook chain (ruff, isort, black, mypy,
      prettier, eslint, readme parity) passed on the actual commit.
  • I have updated any relevant documentation or created new
    documentation where appropriate.

…7.0 ceiling

operating_envelope.HOST_LOAD_CEILING is a binary cliff: the instant a fresh
sample reads above 7.0, dispatch_micro_batch halts and requires a fresh
owner action to resume. Two passes tripped a day apart on narrow overshoots
(7.0796, then 7.17236328125 - 1.1% and 2.5% over) despite the box otherwise
running well under load, each costing a stopped pass and a human
interaction.

Adds cardpicker.stage_e_load_brake: between the no-self-resume gate and the
fresh envelope sample, re-samples os.getloadavg() independently and, while
the reading sits between STAGE_E_HOST_LOAD_SOFT_CEILING (default 6.0) and
the hard ceiling, sleeps with jitter and re-samples rather than proceeding
straight to the trip check. A load above the hard ceiling stops braking at
once and lets the envelope's own fresh sample trip honestly - the brake
never itself decides trip and never suppresses a genuine breach. Cumulative
wait past STAGE_E_LOAD_BRAKE_MAX_WAIT_S (default 240s) proceeds anyway,
matching the ~4 time-constant decay window of the 1-minute load average it
re-samples. All three settings default to values that make the brake active
out of the box.

DispatchOutcome gains load_brake_waits/load_brake_seconds, merged into
PilotRunLedger.counters on every completed micro-batch, so the brake's
effect is queryable rather than invisible.

Tests (test_stage_e_load_brake.py) assert the specific wait counts and
durations for each band, not just direction - including that the shipped
defaults produce a reachable band end to end, and that a load above the
ceiling never sleeps. Every new test's assertion was confirmed to fail when
the behaviour it checks was manually removed (see the PR body's
verification section).
@WilfordGrimley
WilfordGrimley merged commit f46b8b9 into master Aug 6, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant