Skip to content

Bug: Allow the canonical benchmark workflow to finish its declared suite #169

Description

@smiggleworth

Type

Bug

Affected subsystem

Canonical benchmark evidence workflow and Tier 1-6 artifact retention.

User/operator impact

Operators cannot retain a complete same-commit Tier 1-6 manifest because the workflow timeout is shorter than the declared suite's observed lead-in plus its mandatory endurance floor. Canonical evidence issue #7 therefore cannot complete even when every benchmark owner is healthy.

Current behavior

.github/workflows/benchmarks.yml sets timeout-minutes: 180. The performance contract requires exactly two Tier 6 scenarios with at least 3,600 measured seconds each, so Tier 6 alone requires at least 120 minutes. Canonical run 32764029001 spent more than 80 minutes reaching and passing the repaired Tier 3 startup boundary; Tier 4 and all Tier 5 owners still precede Tier 6. The declared suite therefore cannot fit within 180 minutes.

Expected behavior/outcome

Give the canonical workflow the maximum bounded GitHub Actions job budget of 360 minutes and enforce that budget with a focused workflow contract test. This changes only orchestration time allowance; it must not shorten Tier 6, filter owners, weaken manifest gates, or change benchmark semantics.

Deterministic evidence

  • Workflow source: .github/workflows/benchmarks.yml:26 declares timeout-minutes: 180.
  • Contract source: docs/performance-contracts.md:202-204 rejects Tier 6 durations below 3,600 seconds and requires two one-hour scenarios.
  • Live run: https://github.com/cntryl/cassie/actions/runs/32764029001 crossed 80 minutes while still inside the unfiltered owner-suite step, before Tier 4/5 and Tier 6 completion.
  • Baseline commit: 00bd0e1e4710716da2fd17e30aabbb253eac5ce2.

Confidence and support status

High confidence. This is deterministic arithmetic over the workflow timeout, observed lead-in, benchmark ordering, and mandatory Tier 6 duration. It is an evidence-orchestration bug, not benchmark variance and not a product runtime regression.

Acceptance criteria

  • Add a focused red/green contract proving the canonical workflow allows 360 minutes.
  • Raise only the benchmark-evidence job timeout to 360 minutes.
  • Preserve the unfiltered wildcard command, 3,600-second Tier 6 input, complete-manifest validator, and artifact upload.
  • Pass the repository validation ladder and exact-head backend CI.
  • Squash-merge the repair before dispatching the next final-main canonical run.

Related: #7.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is not working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions