fix: K5 skips percentage columns, and the skill now knows k_checker.py exists - #103
Merged
Merged
Conversation
…y exists
Two follow-ups from the same-day review of 2.14.1.
K5 walking every header (2.14.1's fix for the known-concept-only miss) opened a
new false positive: a real layout sometimes writes the *average* of a
percentage column (e.g. „Клас %“) into the totals row instead of its sum, which
a generic sum check cannot tell from a hand-typed total. Reproduced on a
two-row fixture. K5 now excludes the same day/percentage/hour columns K6
already excluded, at the cost of giving up K5 on a day-column total (which
would still sum validly) to remove the confirmed false positive.
Fixing that exposed a second bug in the exclusion regex itself: a bare "час"
substring match also fired inside "**час**т" ("лична част" - one of
generate_wide.py's own three K5 target columns), which silently dropped K5
coverage for it. Word-bounded now.
Separately: nothing in SKILL.md or proverki.md told the model tools/k_checker.py
exists, so an audit still computed K5/K6 in prose - exactly where the original
evaluation measured the model weakest. Both now point to it, in the same shape
as the existing tools/preflight.py pointers: when its report is available, take
the findings from there instead of recomputing them.
Verified against test/generate_wide.py at 300 seeds: K5 129/129, K6 130/130,
zero false positives, including the new percent-average shape. Confirmed the
regression test has teeth by reverting the K5 filter and watching it fail
before restoring it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XvbzEbQwxwxxHAtXfPMrno
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two follow-ups from the same-day review of 2.14.1.
K5 false positive on averaged percentage totals. Walking every header for K5 (2.14.1's fix for the known-concept-only miss) opened a new gap: a real layout sometimes writes the average of a percentage column (e.g. „Клас %“) into the totals row instead of its sum, which a generic sum check can't distinguish from a hand-typed total. Reproduced on a minimal two-row fixture. K5 now excludes the same day/percentage/hour columns K6 already excluded — trading K5 coverage on day-column totals (which would still sum validly) for removing the confirmed false positive.
Fixing that surfaced a second, adjacent bug: the exclusion regex used a bare
часsubstring match, which also fired inside "част" ("лична част" — one ofgenerate_wide.py's own three K5 target columns), silently dropping K5 coverage for it. Now word-bounded.The skill didn't know the tool exists. Nothing in
SKILL.mdorproverki.mdtold the modeltools/k_checker.pyexists, so an audit still computed K5/K6 in prose — exactly where the original evaluation measured the model weakest. Both now point to it, in the same shape as the existingtools/preflight.pypointers: when its report is available, take the findings from there instead of recomputing them.Test plan
python test/k_checker_test.py --seeds 300— K5 129/129, K6 130/130, zero false positives, including the new percent-average shapepython test/skill_test.py— still green, SKILL.md/proverki.md additions don't trip any length or reference checkspython test/run_tests.py,preflight_test.py,komplekt_test.py,lifecycle_test.py— all green🤖 Generated with Claude Code
https://claude.ai/code/session_01XvbzEbQwxwxxHAtXfPMrno