Choose the scan-budget mechanism from the server's own descriptors - #191
Merged
Conversation
The corrected v0.3.3 freeze failed the same way v0.3.2 did, 28 cases carrying `reason=time_budget`, and the reason is structural: the budget mechanism is version-split, and freezes pin candidates on both sides of the split. Through Bifrost v0.10.7 the scan tools accept a per-call `max_duration_secs` argument, and that argument is the only way to raise the scan's own deadline. From v0.10.8 the argument is rejected and the budget is read from `BIFROST_MCP_REQUEST_BUDGET_SECS` at server launch. The evaluation registry pins v0.10.2 (`d1a7c0cc`), the active registry pins v0.10.8, and the runner must measure both correctly. Sending the argument errors every scan on one side; withholding it re-measures the other under the 4.5-second interactive default, truncating large scans into failures. No fixed behaviour serves both, so the runner now asks the server which side it is on. `McpSession` gains a `tools/list` probe, issued once per session, and the scan sends `max_duration_secs` exactly when the server's `scan_usages_by_location` descriptor advertises it. The signal is authoritative on both sides: v0.10.2 authors the property in its schema, v0.10.8's own test asserts it absent. A server that does not advertise the scan tool at all is refused loudly rather than guessed at, because either guess mismeasures. The environment variable stays set at launch, the client backstop deadline stays budget-plus-grace, and the flag and its recorded invocation metadata are unchanged. Dry-run before any freeze, all three legs: - v0.10.2 (`d1a7c0cc`), 300s: `real-project-v2-rust-01-1` reproduces the frozen v0.3.0 record byte for byte -- `failed`, the same four unexpected usages in src/config.rs, zero missing -- with the scan running 204 seconds uncut where v0.3.2/v0.3.3 cut it at 4.5. - v0.10.2, 1-second budget: bites immediately, proving the mechanism live rather than vacuously accepted. - v0.10.8, 300s: the case still passes, 9 true positives, both queries exact. The readiness-continuation test's scripted server now answers `tools/list`, which real servers serve statically before workspace readiness. A consequence worth stating: a v2 re-freeze with this fix reproduces v0.3.0's 17 failures, because it measures v0.10.2 by construction. Publishing v0.10.8's improvement on that slice means advancing the evaluation candidate registry, which is a reviewed change, not a re-freeze. Refs #185 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DavidBakerEffendi
added a commit
that referenced
this pull request
Sep 2, 2026
Closes #185, recast: the published v2 result understates Bifrost not because a re-freeze is missing, but because the slice's frozen candidate registry still pins v0.10.2 — every v2 freeze re-measures last month's analyzer by construction (v0.3.2/v0.3.3 demonstrated exactly this). The slice's own README states the doctrine: > Any later Bifrost run is evidence for a subsequent snapshot or release, never > a replacement for v0.3.0 evidence. ## The change **`candidates-v0.3.4.json`** — a new frozen identity set, not an edit: - Bifrost advances to **v0.10.8 @ `8ddf1362`**, the same identity the active development registry pins. - Everything else is byte-identical to the v0.3.0 registry — reference-server identities, profiles, hashes. The corpus, protocol, selection, and reviewed ground truth are untouched. - `candidates-v0.3.0.json` stays byte-identical as the immutable record of what v0.3.0 measured. Consumers: the freeze workflow selects the new registry for evaluation snapshots; shard identities hash both registries and read candidate identity from the current one; the reproduction contract asserts the superseding registry's Bifrost matches the active pin **and** that no non-Bifrost identity moved, alongside its existing guard that the historical registry is unchanged. Tests cover the "advances only the Bifrost pin" property directly, in both the Python and Rust suites. `CITATION.cff` moves to 0.3.4 for freeze preflight. ## Expected effect on the next v2 freeze From the repository-tree measurement under v0.10.8: **12 of v0.3.0's 17 failures resolve, 5 remain** (#181 ×2, #182 ×2, #183), **0 new**. The freeze is the real acceptance test — please check the aggregate's counts against this at the release approval gate rather than taking the prediction on trust; in particular, any `reason=time_budget` diagnostic means the scan-budget fix regressed and the run should not be approved. Depends on #191 (merged): without the capability fix, a v0.10.8 evaluation freeze errors every usage scan. ## Verification - `cargo test --locked` — 257 passed (1 new), 0 failed - Python suites — 44 tests OK (2 new) - `validate-reproduction-contract.sh` — exit 0 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The corrected v0.3.3 freeze failed identically to v0.3.2 — 28 cases with
reason=time_budget— and the reason is structural, not a bug in eitherearlier attempt alone: the budget mechanism is version-split, and freezes pin
candidates on both sides of the split.
max_duration_secsargumentBIFROST_MCP_REQUEST_BUDGET_SECSSending the argument errors every scan on one side (v0.3.2's failure mode);
withholding it re-measures the other under the 4.5-second interactive default
(v0.3.3's). No fixed behaviour serves both.
The fix
The runner asks the server which side it is on.
McpSessiongains a standardtools/listprobe, issued once per session, and the scan sendsmax_duration_secsexactly when the server'sscan_usages_by_locationdescriptor advertises it in its
inputSchema.The signal is authoritative on both sides — v0.10.2 authors the property in its
schema; v0.10.8's own test asserts it absent — so this is self-describing
rather than a version table the runner would have to keep true. A server that
does not advertise the scan tool at all is refused loudly rather than
guessed at, because either guess silently mismeasures.
Unchanged: the env var still set at launch, the budget-plus-grace client
backstop, the flag and its recorded invocation metadata.
Dry-run evidence (all three legs, before any freeze)
v0.10.2 (
d1a7c0cc), 300s — the acceptance bar was not "the case passes"but reproduce the frozen v0.3.0 measurement, and it does, byte for byte:
The scan ran 204 seconds uncut where v0.3.2/v0.3.3 cut it at 4.5.
v0.10.2, 1-second budget — bites immediately, proving the mechanism live
rather than vacuously accepted.
v0.10.8, 300s — still passes: 9 TP, 0 FP, 0 FN, both queries exact.
Consequence worth agreeing on before the next freeze
A v2 re-freeze with this fix reproduces v0.3.0's 17 failures, because it
measures v0.10.2 by construction — that is the registry the preregistered slice
binds. Publishing v0.10.8's improvement on that slice means advancing the
evaluation candidate registry, a reviewed change; #185 should be recast
accordingly. v0.3.2 and v0.3.3 both stand superseded; the site never published
either.
Verification
cargo test --locked— 256 passed (4 new), 0 failedpython3 -m unittest discover tests— 42 tests, OKvalidate-reproduction-contract.sh— exit 0tools/list, whichreal servers serve statically before workspace readiness
🤖 Generated with Claude Code