This page is the entry point for the testsuite's quality signals. It links to live sources rather than copying point-in-time values that quickly become stale.
| Signal | Quality expectation | Live source |
|---|---|---|
| Repository CI | The latest main run and every pull request pass. |
CI workflow runs |
| Repository policy | Required governance, CI, documentation, and test assets satisfy the versioned policy on every pull request. | policies/repository.json |
| Nightly compliance | Repository policy, Ruff correctness checks, Python compilation, feature discovery, and CodeQL pass each night; scheduled failures are tracked as bug issues. | Nightly workflow runs |
| Runtime suites | Behave suites pass against published images; failures are investigated in the lab that exercised the image. | frostyard/lab workflow runs |
| Coverage | Project coverage does not fall by more than 2%; changed Python code reaches 80%. | Codecov dashboard and gate configuration |
| Pull request acceptance | Accepted and closed-unmerged pull requests are reported monthly with their counts and acceptance percentage. | Metric definition, merged PRs, and closed, unmerged PRs |
| Open defects | Regressions and test defects are triaged rather than hidden by broad allowlists. | Open bug reports |
| Advisory AI review | Eligible same-repository pull requests receive least-privilege Claude feedback; comments remain untrusted until human verification. | Workflow and trust/setup contract |
A red repository CI run blocks its commit or pull request. A runtime-suite
failure may indicate either a testsuite regression or a defect in the image
under test; use the failing lab run's pinned image reference and Behave output
to distinguish them. Coverage status follows codecov.yml, which is the source
of truth for thresholds.
Reviewers should apply the pull request review rubric before
approval. Claude review is advisory evidence, not approval and not a substitute
for repository CI or live image validation. Non-gating scenarios must carry
@wip, and expected nested-container
failures must remain narrowly scoped and documented. These exceptions are
visible in test output and are not equivalent to passing coverage.
- Per pull request: inspect repository CI, affected runtime-suite evidence, and patch coverage.
- After each published image: inspect the corresponding lab run and triage unexpected Behave failures.
- Monthly: report pull request acceptance using the UTC-period definition in
public metrics, and review open defects and stale
@wipscenarios.