Skip to content

Add opt-in line and branch coverage reporting - #276

Open
DicksonWu654 wants to merge 2 commits into
stx-labs:masterfrom
DicksonWu654:feat/coverage-report
Open

DicksonWu654 wants to merge 2 commits into
stx-labs:masterfrom
DicksonWu654:feat/coverage-report

Conversation

@DicksonWu654

@DicksonWu654 DicksonWu654 commented Oct 5, 2026 •

Copy link
Copy Markdown

Context

When fuzzing Clarity contracts, I wanted to know how much of a contract a run
actually reached before trusting a clean result, and to compare seeds, run
counts and test designs. #85 asks for the same thing, for example to check
coverage in CI.

Adds opt-in coverage reporting for #85.

rv example counter invariant --runs=100 --seed=42 --coverage \
  --coverage-json=coverage.json --coverage-lcov=coverage.lcov
Coverage: 19/831 lines, 3/257 branches in 0.833 s.
  contracts/counter.clar: 19/34 lines, 3/8 branches
  • --coverage enables the Clarinet SDK's coverage tracking for the run and
    prints line and branch totals. It then lists each file the run reached.
    Totals cover the project's contracts (including their #[env(simnet)]
    test functions) and called contracts, but not boot contracts.
  • --coverage-json=<file> saves overall and per-file totals and a curve of
    the points where coverage increased. It collects after every run, so it
    costs more than --coverage alone.
  • --coverage-lcov=<file> saves merged line and branch LCOV for existing
    tooling.
  • Config keys: coverage, coverage_json, coverage_lcov.

Without --coverage, Rendezvous creates the simnet exactly as before, so
the calls, output and exit codes are unchanged.

How it works. collectReport drains the SDK's coverage. Rendezvous
collects before each session reset and at the end, and merges the deltas.
A file/line or file/line/block/branch counts as covered only when its hit
count is positive.

Why #1638 does not block this. clarinet#1638 is about coverage being
empty for contracts deployed after initEmptySession. Rendezvous now
deploys the contracts and their #[env(simnet)] tests from Clarinet.toml
through initSimnet, so that situation does not arise. The tests check for
positive hits on the example contracts, including across a session reset.

Tests.

  • LCOV aggregation edge cases: zero and - branches, repeated records,
    invalid rows, and an LCOV round trip.
  • The evaluation observer, with and without listeners.
  • SDK report draining across a reset.
  • The CLI on the reverse (test) and counter (invariant) examples: seeded
    calls are identical with and without --coverage, and JSON and LCOV
    agree.
  • CLI and config validation, for example that output paths require
    --coverage.

🤖 Generated with Claude Code

Add --coverage to print line and branch totals at the end of a run,
followed by the counts for each file the run reached, with optional
--coverage-json and --coverage-lcov outputs. Coverage tracking stays
disabled unless requested, so default runs keep the same calls and
output. Reports are drained from the SDK and merged across session
resets, counting only positive hits.

Refs stx-labs#85.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@moodmosaic moodmosaic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this. The overall design makes sense to me. 👍

Comment thread app.ts Outdated
radio,
eligibleAccounts,
allAddresses,
} finally {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if there is an exception thrown inside the finally block? Is the primary runner error preserved?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, it wasn't. A throw from the report (for example an unwritable --coverage-json path) replaced the runner's error. Fixed in d7532e6: coverage is now reported through reportAfterRun(), which logs a report failure and rethrows the run's own error. Tests cover a failed run whose report also fails, a failed run with a working report, and a report failure after a successful run.

Comment thread app.ts
);

const resetSession = async () => {
coverage?.collect(simnet);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could collect() before a reset discover new hits that the next sample() then won’t record in the curve? Perhaps you should record increases in collect() too, and add a test interleaving the two.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it could. Hits drained by collect() before a reset were counted in the totals but never added to the curve. Fixed in d7532e6: collect() now records a curve point whenever it finds new hits, and a new test interleaves sample() and collect() (it fails without the change). Default runs are unchanged (the CI example commands give identical output).

Record a curve point whenever collect() finds new hits, so coverage
drained before a session reset is not missed by the next sample. Add a
test that interleaves sample() and collect().

Report coverage after the run through reportAfterRun(): if the run
failed and writing the report also fails, the report error is logged
and the run's own error is rethrown instead of being replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants