Skip to content

Tests: planted bugs as a script with a record, and clearer input labels - #4

Merged
ahrazzle merged 2 commits into
mainfrom
claude/semver-first-slice
Oct 28, 2026
Merged

ahrazzle merged 2 commits into
mainfrom
claude/semver-first-slice

Conversation

@zero-aska

@zero-aska zero-aska commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

This is a follow-up to #3, which merged today. It does the two things I promised in the review thread there, and fixes the author list in bar.toml. Nothing outside tests/, README.md and bar.toml changes.

Planted bugs, now in the repo

The body of #3 gave counts for two planted bugs that I had run by hand, so nobody could trace them. tests/plant_bugs.py now runs them, with eight more. It writes tests/PLANTED-BUGS.md, in the same shape as hostname and punycode.

The script plants each bug as a one-line change to SemVer/Basic.lean in a copy of the project, then runs tests/run_tests.py --count 2000:

Planted bug Failures
none 0
leading zeros are accepted in numbers 874
identifiers are compared by length only, not in ASCII order 338
a hyphen is not an identifier character 945
an empty build identifier is accepted 257
a numeric identifier comes after an alphanumeric one 309
a version with a pre-release part comes after the release 163
pre-release identifiers are compared the wrong way round 1471
build metadata takes part in precedence 47
numeric identifiers are compared as text 17
a hyphen in the build part starts the pre-release part 1418

All 10 are caught. The first two counts match the ones #3 stated, 874 and 338.

A run counts as a catch only if it finishes. A bug that does not build, a run that dies before it writes a record, and a run that gives up all count as missed. The rule about dying runs fixes a flaw that a review found in the same script in bech32: a run that died could reuse the old record. The fix also went into hostname and punycode. Each bug also breaks a proof. The tests build only the executable, so they still run.

Clearer labels in the record

As promised on #3, the input kind valid is now random version, and random is now random string. Before this, the body's "random versions" meant the record's valid. The counts are unchanged: 3,021,721 inputs, 0 failures.

The spec authors

bar.toml listed the spec authors as claude and fdn-0. The commits and pull requests come from the GitHub account zero-aska. So bar item 1's check would not have refused a spec approval from the account that wrote the spec. spec_authors now names zero-aska too. A self-review of bech32#2 found this, and the same fix went into hostname#2, punycode#2 and bech32#2.

Checks

  • python3 tests/run_tests.py: 3,021,721 of 3,021,721 requests agree, 0 failures.
  • python3 tests/plant_bugs.py: 10 of 10 caught.
  • bar/check.py --offline --no-lean at the pinned mission commit: items 2, 4, 5 and 6 pass. Item 1 waits on a spec reviewer.
  • bar.toml now lists tests/plant_bugs.py among the test files.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AovXVgJBHGiN44TxQM4dcd

Follow-up to PR 3, as promised in its review thread. The two planted
bugs whose counts the PR body gave by hand now run from
tests/plant_bugs.py, with eight more, and tests/PLANTED-BUGS.md records
the result: 10 of 10 caught, including the two original counts, 874
and 338. A bug that does not build, or a run that does not finish,
is not counted as caught.

The record's input kinds 'valid' and 'random' become 'random version'
and 'random string', so the two read apart. The counts are unchanged:
3,021,721 inputs, 0 failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AovXVgJBHGiN44TxQM4dcd
bar.toml listed the spec authors as claude and fdn-0. The commits and
pull requests come from the GitHub account zero-aska, so bar item 1's
independence check would not have refused an approval from it. The
same fix went into hostname#2, punycode#2 and bech32#2. plant_bugs.py
also counts any run that gives up as missed, as in those repos.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AovXVgJBHGiN44TxQM4dcd
@ahrazzle
ahrazzle merged commit c843546 into main Oct 28, 2026
3 checks passed
@ahrazzle
ahrazzle deleted the claude/semver-first-slice branch October 28, 2026 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants