Hold the background warmup ladder while a request is in flight

[Blakeolson21]

Branch: q6-lane/warmup-inflight-foreground-yield (3 commits, base main @ 90d8c4b)
Files: mtplx/server/openai.py, tests/test_background_warmup.py

What is wrong

_BackgroundWarmup._foreground_quiet_for_s decides whether a warm rung may be
admitted, and it reads only state.last_request_at. That field is stamped when a
request completes (five assignment sites, all on completion or cancellation
paths). A request that has arrived and is still generating therefore leaves it
untouched, and on a daemon that has not completed a request yet it is still
0.0, which the function reads as "no request has ever landed" and reports as
infinite quiet:

def _foreground_quiet_for_s(self) -> float:
    last = float(getattr(self.state, "last_request_at", 0.0) or 0.0)
    if last <= 0.0:
        # No request has ever landed (fresh boot): warm immediately.
        return float("inf")
    return max(0.0, time.time() - last)

The comment says "landed". The field means "finished". Everything follows from
that gap.

Why this window is the worst one

The failure is worst exactly where it is most visible. A control UI that
restarts the engine on a configuration change is typed into immediately
afterwards, so the very first request of a serve is both the one that has not
completed yet and the one most likely to be warmed over. Under the turbo
profile's eight-rung ladder (512,1024,2048,2560,4096,8192,16384,32768) a
single rung is tens of seconds of background prefill on a 27B, so the operator's
first request shares the GPU with a rung that the 90 second idle grace was
supposed to have held back.

Observed on a 27B (Qwen3.8-27B, 6-bit, M3 Max) from the server's own request
log, real non-warmup rows recorded while the ladder was walking:

{'prompt_tokens': 2622,  'completion_tokens': 28, 'decode_tok_s': 0.75}
{'prompt_tokens': 2622,  'completion_tokens': 0,  'decode_tok_s': 0.0}
{'prompt_tokens': 11459, 'completion_tokens': 0,  'decode_tok_s': 0.0}

The per-chunk _ForegroundYield abort still works and still bounds the damage
once a rung is running. It cannot help with a rung that should never have been
admitted.

The change

Model work that is in flight or queued holds the ladder as its own admission
condition, whatever the grace is. Only when nothing is running does the
completion stamp decide.

Four details worth reviewing:

• The busy check (_foreground_busy) runs at step admission, in
  _run_step_inner, before the warming generation begins. At that moment
  has_foreground() counts only real requests, which is why this code can use
  it where the _ForegroundYield shim documents that it deliberately cannot
  (the shim runs inside the warming generation, which has incremented the
  counter itself via _run_generation_dispatched).
• The queued half goes through the same _foreground_model_work_pending helper
  the yield shim uses, so foreground work that is queued but not yet executing
  also holds the ladder.
• Busy is its own branch, not a zero folded into the grace comparison.
  MTPLX_WARMUP_IDLE_GRACE_S=0 is how an operator says "do not wait between
  turns", and expressing busy as zero quiet would collapse the hold there
  (0.0 < 0.0 is False) and hand a warm rung a request that is still
  generating.
• The completion stamp itself is now gated on the same
  request_observability["warmup"] filter _dashboard_record_completion
  already applies, so a warming generation no longer writes last_request_at
  or requests_completed. Before this, a rung's own completion made the next
  rung read quiet as roughly zero and park for the full 90 second grace, and
  /health reported seconds_since_last_request as if a user had just been
  served on an untouched daemon. last_request_at means "a user request
  completed"; warmup was writing it. The counter moves with the clock so the
  two /health fields cannot disagree.

last_request_started_at is deliberately not used as the anchor even though
_smart_fan_activity_probe pairs it with last_request_at: warming generations
go through begin_foreground too, so that field is set by the warmup itself and
using it would make every rung after the first defer forever.

Verification

Four added to tests/test_background_warmup.py:

• test_background_warmup_defers_while_a_request_is_in_flight (fails before)
• test_background_warmup_holds_a_live_request_even_at_zero_grace (fails
  before)
• test_background_warmup_defers_while_foreground_is_queued (fails before)
• test_background_warmup_still_warms_a_genuinely_idle_fresh_daemon (passes
  before and after, so the guard is shown not to cost the case it exists for)

make_state gains the foreground_active counter and the has_foreground
accessor a real ServerState carries, and the shared _deferral_probe
helper replaces a verbatim copy of the same setup in the existing
foreground-recent test.

The warmup-stamp half of the change has no direct test: every test in this
file fakes _run_generation (the real one needs a model runtime), so no
harness can observe the stamp. The closest pinned evidence is the existing
test_dashboard_record_completion_skips_warmup_rows, which asserts the
identical warmup filter on the sibling call in the same function.

$ python -m pytest tests/test_background_warmup.py -q
................                                                         [100%]
16 passed

$ git checkout main -- mtplx/server/openai.py && python -m pytest tests/test_background_warmup.py -q
FAILED tests/test_background_warmup.py::test_background_warmup_defers_while_a_request_is_in_flight
FAILED tests/test_background_warmup.py::test_background_warmup_holds_a_live_request_even_at_zero_grace
FAILED tests/test_background_warmup.py::test_background_warmup_defers_while_foreground_is_queued
3 failed, 13 passed

Residual, stated plainly

A request is only counted from begin_foreground, so the 100 to 200 ms of
handler Python between arrival and dispatch (tokenize, chat-template render) is
still a window in which a rung can be admitted. Closing that needs an arrival
stamp that warming cannot set, which is a larger change; the per-chunk yield
already bounds that case to one warming chunk, which is what it was designed
for. This change closes the unbounded case.

If a code path ever leaked begin_foreground without its matching
end_foreground, warming would stop rather than misfire. Every site pairs them
in a finally, so this is a degradation mode, not a new failure mode.