Report unchecked tests and count runtime errors

[DicksonWu654]

Context

I've been using Rendezvous to fuzz Clarity contracts, both on real projects'
existing Rendezvous tests and on contracts with known planted bugs that I use to
evaluate the fuzzer. Several properties have preconditions, and some runs ended
with "OK, properties passed" even though every case had been discarded, so
nothing was actually checked. If a green run is treated as evidence that a
contract was tested, a pass that tested nothing is the worst outcome. I also saw
runs that logged a runtime error but reported FAILED x0.

What

• When a property test has no passing case by the end of a run (every case
  that selected it was discarded, or it was never selected), Rendezvous no
  longer prints OK, properties passed after N runs.. It prints a warning
  that names those tests. Invariant testing does the same for an invariant
  that was never selected. Exit codes are unchanged.
• Runtime errors are now counted in the execution statistics. Runtime errors
  in property test functions, discard functions and invariants count as
  FAILED. Runtime errors in public function calls during invariant testing
  count as IGNORED. Before this change they were logged, but the
  statistics could show FAILED x0 for a run that failed on a runtime error.
  A runtime error in a discard function is now also logged as a [FAIL] line.

Why

A discarded case does not check anything, so a test whose cases were all
discarded was not tested at all. For example, on master,
rv . slice test --seed=3 --runs=40 in example/ discards all 40 cases
(PASSED x0 for all six tests) and still prints
OK, properties passed after 40 runs.. With this change, it prints:

Warning: not checked after 40 runs: test-slice-ascii, test-slice-buff, test-slice-list-bool, test-slice-list-int, test-slice-list-uint, test-slice-string.
They were never selected or all of their cases were discarded.

The run statistics are the only summary of a run, so they should count
every failure that the log shows.

Opt-in failure

Exit codes are unchanged by default, because the CI example rv . slice test
(100 runs) would fail otherwise: with seeds 1 to 8, 4 to 6 of the 6 slice tests
had no passing case. With --fail-on-unchecked (config key
fail_on_unchecked), a test or invariant that was never checked fails the run
with exit code 1. The reporter emits an unchecked event with the names, and
the CLI turns it into a failure only when the flag is set.

Tests

• outcome.tests.ts runs checkProperties and checkInvariants on a
  temporary Clarinet project that has no requirements. It covers in-place and
  discard-function discards, a passing test next to an always-discarded one,
  a normal pass, and an error result, a runtime error in the test and a
  runtime error in the discard function, each counted once with bail. It
  also covers false and runtime-error invariants counted once, and public
  function runtime errors counted as ignored.
• heatstroke.tests.ts covers the reporter warning for tests and
  invariants, and the OK message when everything was checked.
• app.tests.ts runs rv . slice test --seed=3 --runs=40 with and without
  --fail-on-unchecked and checks the exit code. cli.tests.ts and
  config.tests.ts cover the flag, the config key and its validation.
• npm test: 12 files, 177 tests pass. Format and lint are clean.

🤖 Generated with Claude Code

[moodmosaic]

This makes sense. 👍I’d keep exit codes unchanged here and consider an opt-in --fail-on-unchecked.

[DicksonWu654]

Thanks! Agreed, exit codes stay unchanged here. I'll follow up with an opt-in --fail-on-unchecked once this lands.

[DicksonWu654]

Added the opt-in --fail-on-unchecked (config key fail_on_unchecked) to this PR in 7ddeae9, as you suggested. Without it, exit codes are unchanged.

@hugo-stacks, Radu mentioned you'd know who maintains Rendezvous at Stacks Labs now. Could you point me to the right reviewer for #275, #276 and #277? Thanks!

[hugo-stacks]

Hey @DicksonWu654
I'll take a look today, thanks

[hugo-stacks]

@DicksonWu654 For each of the PRs, can you briefly explain how your needs emerged, what's your use case, and give some context around what made you open this PRs?

It'll help us assess the priority for each

[DicksonWu654]

Thanks @hugo-stacks! I've added a short Context section at the top of #275, #276 and #277 explaining where each need came from and the use case. None of them are urgent, so please take your time. No rush at all.