feat(stage-e): add host-load soft brake, throttle on approach to the 7.0 ceiling - #717
Merged
Merged
Conversation
…7.0 ceiling operating_envelope.HOST_LOAD_CEILING is a binary cliff: the instant a fresh sample reads above 7.0, dispatch_micro_batch halts and requires a fresh owner action to resume. Two passes tripped a day apart on narrow overshoots (7.0796, then 7.17236328125 - 1.1% and 2.5% over) despite the box otherwise running well under load, each costing a stopped pass and a human interaction. Adds cardpicker.stage_e_load_brake: between the no-self-resume gate and the fresh envelope sample, re-samples os.getloadavg() independently and, while the reading sits between STAGE_E_HOST_LOAD_SOFT_CEILING (default 6.0) and the hard ceiling, sleeps with jitter and re-samples rather than proceeding straight to the trip check. A load above the hard ceiling stops braking at once and lets the envelope's own fresh sample trip honestly - the brake never itself decides trip and never suppresses a genuine breach. Cumulative wait past STAGE_E_LOAD_BRAKE_MAX_WAIT_S (default 240s) proceeds anyway, matching the ~4 time-constant decay window of the 1-minute load average it re-samples. All three settings default to values that make the brake active out of the box. DispatchOutcome gains load_brake_waits/load_brake_seconds, merged into PilotRunLedger.counters on every completed micro-batch, so the brake's effect is queryable rather than invisible. Tests (test_stage_e_load_brake.py) assert the specific wait counts and durations for each band, not just direction - including that the shipped defaults produce a reachable band end to end, and that a load above the ceiling never sleeps. Every new test's assertion was confirmed to fail when the behaviour it checks was manually removed (see the PR body's verification section).
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
operating_envelope.HOST_LOAD_CEILING(7.0) is a binary cliff: the instant_bar_breachreadsload_avg > 7.0,check_envelopepersists anEnvelopeTrip,dispatch_micro_batchreturnshalted-new-trip, and thedriver exits 3 with no self-resume — a fresh human
resolve_envelope_tripaction is required to continue. Two full-catalogue passes tripped on narrow
overshoots a day apart (7.0796 on 2026-08-04, then 7.17236328125 on
2026-08-05 — 1.1% and 2.5% over) despite the box otherwise running well
under load, each costing a stopped pipeline and a human interaction. There
was no gradation between "full speed" and "hard stop."
This PR adds
cardpicker.stage_e_load_brake: a soft brake inserted indispatch_micro_batch, between the existing no-self-resume gate and thefresh envelope sample. It re-samples
os.getloadavg()on its own accountand, while the reading sits between
STAGE_E_HOST_LOAD_SOFT_CEILING(default 6.0) and the hard ceiling, sleeps with jitter and re-samples
rather than proceeding straight to the trip check. A load already above the
hard ceiling stops braking at once and falls through to the envelope's own
fresh sample, which trips exactly as before — the brake never itself
decides "trip" and never suppresses a genuine breach. Cumulative wait
past
STAGE_E_LOAD_BRAKE_MAX_WAIT_S(default 240s, ~4 time constants ofthe 1-minute load average's own decay) proceeds anyway — best-effort, never
a deadlock. All three settings default to values that make the brake
active out of the box, no operator opt-in required.
Every dispatcher pausing independently at its own next batch boundary is
what makes this reduce load rather than just delay it: the pass
self-throttles its own effective concurrency down to whatever fits under
the ceiling, continuously, instead of running at full concurrency until it
hits the wall and halts.
DispatchOutcomegainsload_brake_waits/load_brake_seconds, mergedinto
PilotRunLedger.counterson every completed micro-batch, so thebrake's effect is queryable (zero on a quiet box, non-zero under
contention) rather than invisible.
Envelope semantics are unchanged.
operating_envelope.pyis untouched— the hard ceiling, the trip row, the no-self-resume gate, and
current_trip's query all stay byte-identical.stage_e_batch_sizing.pyis untouched — its duration term still derives from the ceiling constant,
not a live sample, for the reproducibility reason stated in its own
docstring.
Docs:
docs/features/stage-e-operations.md's "Ordering, every dispatchcall" list gains the brake as a step (renumbering the rest), a new "The
host-load soft brake" subsection covers the mechanism/settings/
observability/failure posture, and the "Observability" section's counters
list gains the two new fields.
Checklist
pre-commitand installed the hooks withpre-commit installbefore creating any commits.tests where appropriate.
cardpicker/tests/test_stage_e_load_brake.py(19 tests):brake_decision's band boundaries (including that the hard boundaryitself still waits rather than tripping, and that an unreadable load
proceeds rather than blocking);
run_load_brake's injected-loopbehaviour with exact wait counts/durations asserted (below-soft = zero
waits, in-band = the specific value, above-hard = zero waits/no sleep
call at all, cumulative-wait-past-bound = proceeds anyway, re-sampling
happens every iteration, jitter is applied to every sleep);
apply_load_brake's settings-integration behaviour against the realsettings.pydefaults (not hand-picked test values), including amalformed setting and an
os.getloadavgfailure both degrading tounbraked rather than raising.
Stage E test files per this repo's own issue Test suite is order-dependent: leaked fetch-failure window trips the envelope across files (8 failures on master) #679 order-dependency
caution — 199/199 passed combined, no cross-file interaction found.
test_stage_e_dispatch.pyalone (88/88 passed) to confirm thewiring change didn't regress the existing dispatch suite.
entirely (11 tests correctly failed), widened the hard-ceiling
boundary to
>=(the boundary test correctly failed), dropped thejitter multiplier (the jitter test correctly failed), removed the
try/exceptinapply_load_brake(the malformed-setting testcorrectly raised instead of degrading), and effectively disabled the
max-wait bound (both tests exercising it correctly failed with
the wrong wait/second counts). Reverted each mutation and re-ran the
full new-file suite clean (19/19) after every revert.
mypy --config-file mypy.iniclean on all four touched/added Pythonfiles.
black,isort --profile black,ruff, anddocs_lint.pyall clean. Full
pre-commithook chain (ruff, isort, black, mypy,prettier, eslint, readme parity) passed on the actual commit.
documentation where appropriate.