Skip to content

test: pin the match threshold from both sides - #23

Merged
ijuinryukichi merged 1 commit into
mainfrom
test/pin-the-match-threshold
Jul 25, 2026
Merged

ijuinryukichi merged 1 commit into
mainfrom
test/pin-the-match-threshold

Conversation

@ijuinryukichi

Copy link
Copy Markdown
Owner

The tests looked comprehensive. Mutation testing says they were not covering the one contract this project is built on.

What was open

test_unmatched_line_is_honest passes window=1, so it withholds the segment entirely — it exercises the forward window, not the score. Deleting the threshold check that implements the honest-gap contract left it green.

Two mutations against main, 57 tests:

mutation tests killed the test named for the contract
CJK_THRESHOLD 0.25 → 0.60 (2.4x) 1 (test_latin_default_rejects_chance_similarity) passed
drop the best_score > threshold gate 1 (test_interpolate_fills_gaps) passed

Both survivors were indirect. Nothing was holding the decision boundary on purpose.

What this adds

Two tests, pinning that boundary from opposite sides.

test_a_reachable_segment_below_the_threshold_stays_unmatched — puts a candidate squarely in reach and lets the score alone reject it, so the gate is guarded rather than the window standing in for it.

test_the_cjk_floor_still_accepts_a_badly_heard_true_match — real ASR output for a real line, and the reason the floor sits at 0.25: 九月十五朝霧開戦 / 黒田細川矢刃交わし合戦 was heard as 9月15 朝霧開戦 来るだ 遅かえば 可視化せん and scores 0.256. The model got the moment right and still clears the floor by a hair. Raise the floor and this true match is lost — which is the evidence the README's 0.25 rests on, now enforced.

Verification

Each new test fails under its corresponding mutation — that is the point of adding them, and it was checked rather than assumed:

CJK floor 0.25 -> 0.60          2 failed (incl. cjk_floor_still_accepts_a_badly_heard_true_match)
threshold gate removed          2 failed (incl. reachable_segment_below_the_threshold_stays_unmatched)
clean                          59 passed

No source changes. Clear __pycache__ between mutation runs — an unchecked hash-based .pyc will keep a reverted constant alive and make git status look clean while the suite still fails.

🤖 Generated with Claude Code

`test_unmatched_line_is_honest` withholds the segment (window=1), so it
exercises the forward window, not the score. Mutation testing found what
that leaves open: raising the CJK floor 0.25 -> 0.60 killed 1 of 57 tests,
and deleting the gate that implements the honest-gap contract killed 1 —
neither of them the test named for that contract.

Two tests now hold the decision boundary from opposite sides:

- a candidate in reach but below the floor stays unmatched, so the gate
  itself is guarded rather than the window standing in for it
- a real mishearing of a real line ("九月十五朝霧開戦" heard as "9月15 朝霧
  開戦 来るだ...") scores 0.256 and must still match, which is the evidence
  the floor sits at 0.25 and cannot be raised

Each fails under its corresponding mutation; that is the whole point of
adding them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ijuinryukichi
ijuinryukichi merged commit f81b881 into main Jul 25, 2026
3 checks passed
@ijuinryukichi
ijuinryukichi deleted the test/pin-the-match-threshold branch July 25, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant