Skip to content

Mark current Bifrost regressions as expected - #194

Merged
DavidBakerEffendi merged 1 commit into
mainfrom
dave/mark-run-33842914977-expected-failures
Sep 4, 2026
Merged

Mark current Bifrost regressions as expected#194
DavidBakerEffendi merged 1 commit into
mainfrom
dave/mark-run-33842914977-expected-failures

Conversation

@DavidBakerEffendi

Copy link
Copy Markdown
Collaborator

Description

Scheduled benchmark run 33842914977 reported 10 new Bifrost semantic failures. This change marks those cases as current expected failures through a runner overlay, so the scheduled benchmark reports them as expected_failure while the content-addressed legacy and preregistered evaluation case documents remain byte-for-byte unchanged.

Key Changes:

  • Add a strict 10-case Bifrost expected-failure overlay with reasons taken from the run artifact.
  • Add --expected-failures support and apply the overlay in the scheduled benchmark before scoring.
  • Reject duplicate, missing, already classified, or expected-pass case entries, and test the checked-in overlay against all referenced cases.

Touch Points:

  • benchmarks/expectations/bifrost-expected-failures.yaml
  • src/runners/bifrost.rs
  • src/main.rs
  • .github/workflows/benchmark.yml
  • benchmarks/README.md

Validation:

  • cargo test --locked expected_failure_overlay (3 passed)
  • cargo test --locked (256 passed; 3 existing local Git LFS archive-hash failures remain because the source archives are not materialized correctly in this worktree)
  • Workflow and expectation YAML parsed successfully
  • git diff --check

@DavidBakerEffendi
DavidBakerEffendi merged commit b003373 into main Sep 4, 2026
5 checks passed
@DavidBakerEffendi
DavidBakerEffendi deleted the dave/mark-run-33842914977-expected-failures branch September 4, 2026 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant