Skip to content

ci: add coverage measurement and gate to backend-test - #149

Closed
lucastononro wants to merge 2 commits into
mainfrom
fix/116-coverage-gate
Closed

ci: add coverage measurement and gate to backend-test#149
lucastononro wants to merge 2 commits into
mainfrom
fix/116-coverage-gate

Conversation

@lucastononro

@lucastononro lucastononro commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Summary

  • backend-test had no coverage visibility at all — pytest tests/ -v --tb=short ran with no --cov, and pytest-cov wasn't even a dependency.
  • This adds coverage measurement, a terminal report, an uploaded XML artifact, and a fail-under floor so future coverage regressions in untested modules (e.g. runner, sandbox) can't merge silently.

Changes

  • backend/requirements.txt: add pytest-cov>=6.0.0.
  • .github/workflows/ci.yml (backend-test): run pytest tests/ -v --tb=short --cov=. --cov-report=xml --cov-report=term-missing --cov-fail-under=60, then upload backend/coverage.xml via actions/upload-artifact@v4 (if: always() so failed runs still publish the report).
  • Floor is set at 60%, a few points below the measured baseline (65.81% total, 296 passed / 8 skipped) — enough margin to avoid flaking on incidental drift while still catching real regressions.

Test plan

Existing

  • Ran the full backend suite locally in a fresh venv (Python 3.12, deps from requirements.txt): 296 passed, 8 skipped, unchanged from current main.

New

  • Ran the exact new CI command locally: pytest tests/ -v --tb=short --cov=. --cov-report=xml --cov-report=term-missing --cov-fail-under=60 → passes with Required test coverage of 60% reached. Total coverage: 65.81%.

Closes #116

🤖 Generated with Claude Code

Greptile Summary

This PR adds pytest-cov coverage measurement and a fail-under gate to the backend-test CI job, with test files excluded from the metric via a new .coveragerc so the floor reflects untested production code only.

  • .github/workflows/ci.yml: Expands the pytest invocation with --cov=., XML + terminal reports, and a --cov-fail-under=50 gate; uploads backend/coverage.xml as an artifact on every run (including failures) via actions/upload-artifact@v4.
  • backend/.coveragerc: New config that omits tests/*, keeping the coverage percentage focused on application code (~51% baseline vs ~66% when tests were included).
  • backend/requirements.txt: Adds pytest-cov>=6.0.0 as a test dependency.

Confidence Score: 5/5

This is a safe, additive CI change with no modifications to application logic.

The change only adds coverage instrumentation to the test job. The .coveragerc omit pattern is appropriate for the flat tests/ structure, the artifact upload path is correct relative to the workspace root, and the 50% floor gives a few points of headroom above the measured ~51% production-only baseline so the gate won't flake on minor drift.

No files require special attention.

Important Files Changed

Filename Overview
.github/workflows/ci.yml Adds --cov flags, coverage fail-under=50 gate, and artifact upload to backend-test job; artifact path backend/coverage.xml is correct relative to workspace root
backend/.coveragerc New file; omits tests/* from coverage measurement — pattern is correct since all test files are flat under backend/tests/
backend/requirements.txt Adds pytest-cov>=6.0.0 as a test dependency; straightforward addition

Reviews (2): Last reviewed commit: "fix(review): address Greptile findings o..." | Re-trigger Greptile

Adds pytest-cov to backend/requirements.txt and runs the suite with
--cov=. --cov-report=xml --cov-report=term-missing, uploading the
coverage.xml as a build artifact. Sets --cov-fail-under=60 as a floor
(measured baseline is ~66%), catching future regressions in untested
modules while leaving headroom.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wp66uSq9saSpRq9Bbmjxj4
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml
Comment on lines +49 to +53
- uses: actions/upload-artifact@v4
if: always()
with:
name: backend-coverage
path: backend/coverage.xml

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Artifact name not unique across concurrent runs

name: backend-coverage is a fixed string. If two PRs or branches trigger backend-test simultaneously (e.g. a push to main and an open PR), both jobs will try to upload an artifact with the same name into the same Actions scope. actions/upload-artifact@v4 will reject the second upload with a conflict error. Consider parameterising with the run id, e.g. name: backend-coverage-${{ github.run_id }}.

Fix in Claude Code

- Scope coverage to production code: add backend/.coveragerc omitting
  tests/ so test modules (always ~100% covered) no longer inflate the
  total. Adjust the gate 60 -> 50 to match real production coverage
  (51.27% measured; with tests included it was 65.81%).
- Dismissed the artifact-name finding: Actions artifacts are scoped per
  workflow run, and this job uploads backend-coverage once per run, so
  concurrent runs cannot conflict.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wp66uSq9saSpRq9Bbmjxj4
@lucastononro

Copy link
Copy Markdown
Owner Author

Greptile review findings addressed (commit cd58511)

Fixed — coverage source includes test files (ci.yml:48): Valid. Measured it: with --cov=. the total was 65.81%, inflated by test modules at ~100%. Added backend/.coveragerc with [run] omit = tests/* so the metric reflects production code only, and adjusted the gate --cov-fail-under=6050 to match real production coverage (51.27% measured locally; 60 would have failed every build once tests were excluded). The gate stays a meaningful ratchet just below current coverage.

Dismissed — artifact name not unique across concurrent runs (ci.yml:53): False positive. GitHub Actions artifacts are scoped per workflow run; the upload-artifact@v4 name-conflict error only occurs when the same run uploads the same artifact name twice. This workflow uploads backend-coverage exactly once per run, so a concurrent main push and PR run (distinct run IDs, distinct artifact namespaces) cannot collide. A stable name is also easier to fetch programmatically.

Verification (local, Python 3.12, fresh venv from backend/requirements.txt):

  • Exact CI command pytest tests/ -v --tb=short --cov=. --cov-report=xml --cov-report=term-missing --cov-fail-under=50: 296 passed, 8 skipped, coverage 51.27%, gate reached
  • coverage.xml verified to contain zero tests/ entries after the omit
  • ruff check . and ruff format --check .: clean (142 files)
  • ci.yml parses as valid YAML

lucastononro added a commit that referenced this pull request Jul 29, 2026
 #116)

Stack wave 5, PR 1/4. Greptile findings already addressed on the PR
branch by author fixup cd58511 (.coveragerc omits tests/, gate 50).
@lucastononro

Copy link
Copy Markdown
Owner Author

Merged into integration branch staging-v0.0.5 (see the STAGING-v0.0.5.md ledger on that branch for merge order, Greptile follow-ups and test results). These changes will land on main via the staging merge — closing to clear the queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add coverage measurement and a coverage gate to CI

1 participant