Skip to content

Restore the usage-scan budget on the server that enforces it - #189

Merged
DavidBakerEffendi merged 1 commit into
mainfrom
dave/restore-scan-budget
Sep 2, 2026
Merged

Restore the usage-scan budget on the server that enforces it#189
DavidBakerEffendi merged 1 commit into
mainfrom
dave/restore-scan-budget

Conversation

@DavidBakerEffendi

Copy link
Copy Markdown
Collaborator

v0.3.2 is an invalid measurement. It published 27 failures where v0.3.0 had
17, and the regression is ours, not Bifrost's. 23 of those cases carry
scan_usages_incomplete: reason=time_budget with a partial usage set.

Root cause

max_duration_secs was never only a request argument — it was how the
frontend raised Bifrost's per-request analyzer budget. Removing the argument
did not remove the budget; it removed our ability to raise it. Every scan then
fell back to Bifrost's cold-workspace default:

// v0.10.8 crates/bifrost-mcp/src/mcp_common.rs:15
const COLD_WORKSPACE_REQUEST_BUDGET: Duration = Duration::from_millis(4_500);

configured_budget.or(cold_workspace.then_some(COLD_WORKSPACE_REQUEST_BUDGET))

4.5 seconds is a budget for interactive use. This is the "five-second
interactive budget" the README used to warn about — still there, one layer
above ScanUsagesExecutionContext, which is why it was missed when the
argument was dropped.

The fixture corpus never noticed: warm, small, ~1.5s a case. A cold clone of a
real project cannot finish a scan in 4.5s and answers with a partial usage set,
which scores as a failure rather than as the timeout it is.

The v0.3.2 timings confirm it was never our clock:

v0.3.0 v0.3.2
failed 17 27
reason=time_budget 23
workspace readiness 131 ms 130 ms
analyzer query, all 36 cases 52.8 s 55.4 s

55 seconds of query time across the whole run — the 300s client deadline never
came close to firing.

The fix

BIFROST_MCP_REQUEST_BUDGET_SECS, read when the server starts, is the
replacement. Deadline policy is still the frontend's; it is chosen once per
server instead of once per call. The runner sets it from the requested scan
budget, so the flag, its recorded invocation metadata, and the 300-second
default keep the meaning they had before v0.10.8.

Two details worth review:

  • Zero means "leave Bifrost's default alone." Bifrost panics on a zero
    budget, so it must not travel as a literal zero.
  • The runner's deadline is now the budget plus a minute, not equal to it.
    Equal deadlines race, and the client winning is the worse outcome: Bifrost's
    budget ends a long scan with structured incomplete evidence, where the client
    deadline can only abandon the call.

Verified on the case it is for

real-project-v2-rust-01-1 failed in v0.3.2 with reason=time_budget. Against
a cold clone under v0.10.8 with this change:

1 passed, 0 failed, 0 errors
9 TP, 0 FP, 0 FN, 9 exact, 1/1 exact-set across 2 queries
analyzer_query=3226ms   build=1062319ms (cache_hit=false)

3.2 seconds — far inside 300, far outside 4.5.

Follow-up

  • v0.3.2 should be treated as superseded; releases are immutable by design, so
    it stays published and a corrected freeze supersedes it.
  • The docs site was never published from v0.3.2, so nothing public carries the
    bad number.
  • Worth considering separately: the freeze approval gate showed no numbers, so
    the bad result was approved on expectation rather than evidence. Printing
    pass/fail counts and a diff against the previous release into the job summary
    would make that gate able to catch this.

Verification

  • cargo test --locked — 253 passed (3 new), 0 failed
  • python3 -m unittest discover tests — 42 tests, OK
  • validate-reproduction-contract.sh — exit 0

🤖 Generated with Claude Code

The v0.3.2 evaluation freeze published 27 failures where v0.3.0 had 17. The
regression was ours, not Bifrost's: 23 of those cases carry
`scan_usages_incomplete: reason=time_budget` and a `partial` usage set.

`max_duration_secs` was never only a request argument; it was how the frontend
raised Bifrost's per-request analyzer budget. Removing the argument did not
remove the budget, it removed our ability to raise it, so every scan fell back
to Bifrost's cold-workspace default:

    // crates/bifrost-mcp/src/mcp_common.rs
    const COLD_WORKSPACE_REQUEST_BUDGET: Duration = Duration::from_millis(4_500);

4.5 seconds is a budget meant for interactive use. The fixture corpus never
noticed -- it is warm and small, about 1.5 seconds a case -- but a cold clone of
a real project cannot finish a scan inside it, and answers with a partial usage
set that scores as a failure rather than as the timeout it is.

The replacement is `BIFROST_MCP_REQUEST_BUDGET_SECS`, read when the server
starts. Deadline policy is still the frontend's; it is chosen once per server
rather than once per call. The runner now sets it from the requested scan
budget, so the flag, its recorded invocation metadata, and the 300-second
default all keep the meaning they had before v0.10.8.

Bifrost panics on a zero budget, so zero means "leave Bifrost's default alone"
rather than reaching the server as a literal zero.

The runner's own deadline becomes the budget plus a minute instead of matching
it exactly. Equal deadlines race, and the client winning that race is the worse
outcome: Bifrost's budget ends a long scan by returning structured incomplete
evidence, where the client deadline can only abandon the call.

Verified on the case this fix is for. `real-project-v2-rust-01-1` failed in
v0.3.2 with `reason=time_budget`; against a cold clone under v0.10.8 with this
change it passes with 9 true positives, no false positives or negatives, and
both queries exact. Analyzer query time was 3.2 seconds -- far inside 300, far
outside 4.5.

Refs #185

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@DavidBakerEffendi
DavidBakerEffendi merged commit 4d1bc86 into main Sep 2, 2026
5 checks passed
@DavidBakerEffendi
DavidBakerEffendi deleted the dave/restore-scan-budget branch September 2, 2026 09:21
DavidBakerEffendi added a commit that referenced this pull request Sep 2, 2026
Prerequisite for the corrected real-project-v2 freeze. `freeze.yml:94`
hard-asserts the citation version matches the snapshot version, so
preflight
fails until this lands.

v0.3.3 supersedes v0.3.2, which measured the slice under Bifrost's
4.5-second
cold-workspace budget and recorded 27 failures against v0.3.0's 17. #189
fixed
the cause; this is the version metadata for re-running the freeze under
it.

Still a **patch**: the corpus, its 36 cases, and their reviewed ground
truth are
unchanged across all three freezes. Only harness behaviour moved.

v0.3.2 stays published — releases are immutable by design, so a
corrected
measurement supersedes it rather than replacing it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant