Repository navigation
Add opt-in line and branch coverage reporting - #276
DicksonWu654 wants to merge 2 commits into
Conversation
Add --coverage to print line and branch totals at the end of a run, followed by the counts for each file the run reached, with optional --coverage-json and --coverage-lcov outputs. Coverage tracking stays disabled unless requested, so default runs keep the same calls and output. Reports are drained from the SDK and merged across session resets, counting only positive hits. Refs stx-labs#85. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moodmosaic
left a comment
There was a problem hiding this comment.
Thank you for working on this. The overall design makes sense to me. 👍
| radio, | ||
| eligibleAccounts, | ||
| allAddresses, | ||
| } finally { |
There was a problem hiding this comment.
What happens if there is an exception thrown inside the finally block? Is the primary runner error preserved?
There was a problem hiding this comment.
Good catch, it wasn't. A throw from the report (for example an unwritable --coverage-json path) replaced the runner's error. Fixed in d7532e6: coverage is now reported through reportAfterRun(), which logs a report failure and rethrows the run's own error. Tests cover a failed run whose report also fails, a failed run with a working report, and a report failure after a successful run.
| ); | ||
|
|
||
| const resetSession = async () => { | ||
| coverage?.collect(simnet); |
There was a problem hiding this comment.
Could collect() before a reset discover new hits that the next sample() then won’t record in the curve? Perhaps you should record increases in collect() too, and add a test interleaving the two.
There was a problem hiding this comment.
Yes, it could. Hits drained by collect() before a reset were counted in the totals but never added to the curve. Fixed in d7532e6: collect() now records a curve point whenever it finds new hits, and a new test interleaves sample() and collect() (it fails without the change). Default runs are unchanged (the CI example commands give identical output).
Record a curve point whenever collect() finds new hits, so coverage drained before a session reset is not missed by the next sample. Add a test that interleaves sample() and collect(). Report coverage after the run through reportAfterRun(): if the run failed and writing the report also fails, the report error is logged and the run's own error is rethrown instead of being replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Context
When fuzzing Clarity contracts, I wanted to know how much of a contract a run
actually reached before trusting a clean result, and to compare seeds, run
counts and test designs. #85 asks for the same thing, for example to check
coverage in CI.
Adds opt-in coverage reporting for #85.
--coverageenables the Clarinet SDK's coverage tracking for the run andprints line and branch totals. It then lists each file the run reached.
Totals cover the project's contracts (including their
#[env(simnet)]test functions) and called contracts, but not boot contracts.
--coverage-json=<file>saves overall and per-file totals and a curve ofthe points where coverage increased. It collects after every run, so it
costs more than
--coveragealone.--coverage-lcov=<file>saves merged line and branch LCOV for existingtooling.
coverage,coverage_json,coverage_lcov.Without
--coverage, Rendezvous creates the simnet exactly as before, sothe calls, output and exit codes are unchanged.
How it works.
collectReportdrains the SDK's coverage. Rendezvouscollects before each session reset and at the end, and merges the deltas.
A file/line or file/line/block/branch counts as covered only when its hit
count is positive.
Why #1638 does not block this. clarinet#1638 is about coverage being
empty for contracts deployed after
initEmptySession. Rendezvous nowdeploys the contracts and their
#[env(simnet)]tests fromClarinet.tomlthrough
initSimnet, so that situation does not arise. The tests check forpositive hits on the example contracts, including across a session reset.
Tests.
-branches, repeated records,invalid rows, and an LCOV round trip.
reverse(test) andcounter(invariant) examples: seededcalls are identical with and without
--coverage, and JSON and LCOVagree.
--coverage.🤖 Generated with Claude Code