Give Roslyn the runtime its build asks for - #171
Merged
Conversation
The roslyn shard failed the v0.3.1 freeze with ten errored C# cases. The language server started and every initialize request failed in milliseconds. Its stderr, which the workflow log does not carry, says: You must install or update .NET to run this application. Framework: 'Microsoft.NETCore.App', version '10.0.0' The following frameworks were found: 8.0.30 vscode-csharp 2.140.9 targets net10.0. This step asserted net8.0 and pinned the Homebrew .NET 8 keg to guarantee it, so it provisioned the one runtime the server cannot use. Its runtimeconfig sets rollForward: Major, which only rolls upward, so 8.x can never satisfy a 10.0.0 request. That premise came from a comment rather than from the server, and the previous two changes here inherited it: the original gate rejected the runner for offering .NET 10 -- exactly what Roslyn needs -- and the follow-up made the provisioning succeed, which moved the failure from the gate to the launch and made it look like a runtime that was merely absent. Read the requirement instead of restating it. Stage the VSIX first, take the framework version from the server's own runtimeconfig, select the matching Homebrew formula, and assert that major is present. A future VSIX bump now moves the runtime with it rather than failing at launch against a version pinned in a comment. Verified against the pinned VSIX and both runtimes on macOS arm64. Under .NET 8.0.30 the C# baseline errors on all six cases with the stderr above; under .NET 10.0.11 it is six passed, zero errors. The full ten-case C# legacy slice satisfies the freeze gate: authoredCases 10, errors 0, three case files, seven passed and three reference disagreements that do not gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The freeze-shard test asserted the exact dotnet@8 provisioning lines, and the toolchain probe rehearsed the same sequence to catch regressions cheaply. Both encoded net8.0 as a fact rather than reading it from the server, so neither could have caught the premise being wrong -- the test pinned it in place and the probe would have reported a green contract the freeze no longer depends on. Assert the derivation instead of the value. The test now requires that the framework is read from the runtimeconfig and that the runtime gate is written against that major, and forbids a restated dotnet@8 prefix or a literal NETCore.App 8 grep surviving anywhere in the workflow. The probe resolves the framework the same way the freeze does: download the pinned VSIX, read its runtimeconfig, choose the Homebrew formula, and pass both values to the later steps as step outputs. A future VSIX bump moves the probe with the freeze rather than leaving it rehearsing an old contract. Testing: unittest discover runs 39 tests green, and both workflows' shell blocks pass bash -n. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DavidBakerEffendi
added a commit
that referenced
this pull request
Aug 26, 2026
The v0.3.1 legacy freeze ([32954887242](https://github.com/BrokkAi/usagebench/actions/runs/32954887242)) ran **all eleven candidate shards green** — roslyn included, so #171 worked — and then failed in aggregation: ``` Error: canonicalize promotion artifact docs/legacy-promotion-selection-policy.md Caused by: No such file or directory (os error 2) ``` `freeze-manifest` resolves a promotion manifest's bound artifacts relative to the staged corpus. The legacy manifest binds its eligibility policy under `docs/`, and `stage-release-bundle.sh` copies `benchmarks fixtures adapters schema src containers scripts` plus a few root files — never `docs/`. So the bundle could not canonicalize an artifact its own manifest hash-binds, which is exactly the self-verifying property the tier exists to provide. It is the only such artifact: sweeping every `file` reference in `benchmarks/promotion/**` yields one path outside the staged prefixes, bound by both `manifest.json` and `cohort.json`. ## The fix Copy exactly the bound files, **derived by reading the promotion manifests** rather than naming `docs/` here, so a newly bound artifact is staged without editing this script. Not all of `docs/` — that carries `node_modules` and build output, which would bloat the bundle and hand the analyzer a large unrelated tree to index. ## Verification, against the failed run's own evidence - Staging emits the policy at the manifest's expected path, sha256 `1ac2968892c97af5…` — **matching the binding** — and `docs/` contributes that one file and nothing else. - Downloaded all eleven shard reports from that run and re-ran the exact aggregation: **succeeds**, satisfies the workflow's `snapshotKind == "legacy_promoted" and balancedCoreCaseCount == 110 and (.corpus | length == 30)` gate, and `generate-results` produces both pages. That run's Bifrost report also confirms the expected set from #160 exactly: ``` 110 cases: 106 passed, 1 improved, 3 failed, 0 errors (bifrost 6624e883) improved cpp-parity-function-like-macro-expanded-call failed rust-struct-construction (#170) failed rust-parity-module-declaration-definition (#170) failed scala-parity-case-class-generated-construction-and-copy ``` ## Regression cover Adds a test asserting the derivation is present and that every artifact bound outside the staged prefixes actually exists. `unittest discover` runs 40 tests green; reproduction contract exits 0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The roslyn shard failed the v0.3.1 freeze
(32933786955)
with ten errored C# cases — every
initializefailing in milliseconds. Theserver's stderr, which the workflow log does not carry, says:
vscode-csharp 2.140.9 targets net10.0. Its runtimeconfig:
{ "tfm": "net10.0", "framework": { "name": "Microsoft.NETCore.App", "version": "10.0.0" }, "rollForward": "Major" }rollForward: Majoronly rolls upward, so 8.x can never satisfy 10.0.0. Thisstep asserted net8.0 and pinned the .NET 8 keg to guarantee it — provisioning
the one runtime the server cannot use.
How this survived two prior changes
The net8.0 premise lived in a comment, not in the server, and both earlier
changes inherited it:
exactly what Roslyn needs. It was failing a runner that would have worked.
failure from the gate to the launch and disguised it as an absent runtime.
The fix
Read the requirement rather than restate it: stage the VSIX first, take the
framework version from the server's own
runtimeconfig.json, select thematching Homebrew formula (
dotnetfor 10,dotnet@Notherwise), and assertthat major is present. A future VSIX bump now moves the runtime with it.
The muxer-by-absolute-path check, the runtime-not-SDK check, and the PATH
resolution check are all kept — they were right, just aimed at the wrong major.
Verification
On macOS arm64 against the pinned VSIX:
csharp-baseline.yamlFull ten-case C# legacy slice against the freeze gate:
The three failures are reference-server disagreement and do not gate.
Also exercised the new selection logic directly against the extracted VSIX:
required_framework=10.0.0→required_major=10→ formuladotnet→ runtimeassertion passes.
Closes the last blocker on #155.
🤖 Generated with Claude Code