feat(security): screen for a requirement identifier PAIRED WITH A VERDICT, not for identifiers (BACKLOG #1337) - #588
Open
wshallwshall wants to merge 8 commits into
Open
Conversation
…DICT, not for identifiers (BACKLOG #1337)
The leak gate screens routable IPs, worktree slugs, home paths, customer and vendor names and site
codes. It had no pattern for security-record content at all. This adds one, and the shape of the
pattern is the entire finding.
WHAT IS AND IS NOT RECORD CONTENT:
a BARE CITATION is a forward reference -- "this code was written with that requirement in mind".
It asserts no coverage, no result and no gap, and is legitimately public: the backlog's own item
titles read `#1107 ASVS 1.2.2 -- apiclient path encoding`. NOT a hit.
an IDENTIFIER BESIDE A VERDICT is the assessment itself, and the assessment is vaulted. That pair
is what this matches.
*** I COULD NOT REPRODUCE THE ROW'S HEADLINE MEASUREMENT AND THE FIX IS SHAPED BY MINE. ***
The row specifies "identifier NEAR a verdict word" and reports it scoring 1 file / tens of hits,
every hit real. Implemented directly over 2045 tracked files it scores 1028 hits across 119 files.
Adding an ASVS context marker still leaves 548 across 50, dominated by docs/BACKLOG.md's own
legitimate citations. A gate firing 548 times is switched off within a day -- which is the failure
the row itself is written to prevent. So the row's DIAGNOSIS is right and its SPECIFICATION is not
tight enough, and I built to the measurement rather than to the sentence. The dispatcher had already
flagged that it did not reproduce the row's 4582/2248/tens figures either.
THE RULE THAT WORKS is the TIGHT PAIR: identifier and verdict adjacent on one line, separated only
by punctuation that binds a label to a value. A space alone is deliberately not enough -- `see 1.2.2
later, this will pass` is prose, and admitting it is what took the looser attempts to 548.
MEASURED over the same 2045 tracked files: ZERO hits. Silent on uv.lock (934 bare dotted-triples),
ide/package-lock.json (1233), constraints.lock (90), docs/BACKLOG.md (661), and on the two tracked
PUBLIC documents specifically flagged as the likeliest false positive --
docs/ASVS-ASSESSMENT-METHOD.md (23 triples) and docs/research/asvs-16-2-2-*.md (3). 8018 bare
triples across 649 files, none of them a pair. THE NAMED BUILD RISK DOES NOT MATERIALISE, measured
rather than argued.
A ZERO OVER A CORPUS IS ONLY EVIDENCE BESIDE A CONTROL THAT FIRES. Planted `1.2.2: pass` in a
scratch file: the shipped scanner reports it and exits non-zero. Both halves are in the suite.
THE DETECTOR'S OWN FIXTURES ARE THE SHAPE IT HUNTS, and that is a trap with three possible answers.
I built two and kept the third:
ALLOWLIST ENTRY -- built, then withdrawn. The narrow anchor I could write covered only fixtures
beginning with a digit; five of eight started otherwise, and widening it far enough to catch
them made it broad enough to veto ordinary quoted strings. My own explanatory COMMENT in the
allowlist also tripped the detector it was explaining, which is the detector-counting-itself
shape this repo has recorded before.
RESERVED PLACEHOLDER IDS -- built, measured discriminating on all sixteen probes, then withdrawn.
It works, and it costs a standing bypass: four identifier values that can never be reported, in
any file, forever.
KEPT: the fixtures store the identifier and the verdict as SEPARATE literals and join them at
runtime, so the source carries no pair, the scanner is silent on this file, and there is no
exemption and no allowlist entry to erode. Both rejected designs are written into the source
beside the rule so the next reader does not re-derive them.
TESTS -- 14 new cases, and BOTH CONTROLS ARE REQUIRED because either alone is worthless:
CONTROL 1 a prose citation carrying a verdict MUST fire (8 cases, both argument orders)
CONTROL 2 a lockfile version and a bare citation MUST NOT (7 cases)
plus: the reason must not echo the identifier or the verdict -- the pair IS the disclosure, so a
reason carrying it would publish into a public CI log exactly what the hit reports.
MUTATION-PROVED, and the two control sets DISCRIMINATE rather than being one test wearing two names:
drop the pairing, match the bare shape -> 18 failures, ALL in the must-stay-quiet set
disable the detector -> 8 failures, ALL in the must-fire set
Restored from a byte copy both times, hash-verified identical.
VERIFIED, scope named:
pytest 137 passed, 2 skipped over test_scan_forbidden plus every module importing the scanner
(test_anon_core, test_anon_parity, test_control_char_check, test_csv_formula_consistency) and
test_security_posture
the shipped scanner over the tracked tree: exit 0
ruff format --check, ruff check, mypy -- run separately, each with its own exit code, all clean
NOT a full-suite run
ONE CLAIM IN THE ROW I COULD NOT CONFIRM: "the identifier corpus is vendored and lawfully
redistributable". No such corpus is tracked in this repository -- scripts/asvs/scorecard.py is
deliberately data-free and takes it as a path. It turned out not to matter: the tight pair needs no
membership filter, which is why this builds without one. Reporting it because the row rests part of
its design on a file that is not here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wshallwshall
enabled auto-merge (squash)
August 25, 2026 18:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a leak-gate detector for security-record content: a requirement identifier paired with a verdict word. A bare citation is a legitimate public forward reference (the backlog's own item titles are full of them); an identifier sitting beside a verdict is the assessment itself, which is vaulted. docs/BACKLOG.md untouched.
Zero hits across all 2045 tracked files -- silent on uv.lock (934 bare dotted-triples), ide/package-lock.json (1233), constraints.lock (90), docs/BACKLOG.md (661), and the two tracked public docs flagged as the likeliest false positive. 8018 bare triples in the tree, none of them a pair. A zero is only evidence beside a control that fires: a planted
<id>: passin a scratch file is caught and exits non-zero. Both halves pinned in the suite.Deliberately does not implement the row's literal specification. The row describes "identifier NEAR a verdict word" and reports it scoring 1 file / tens of hits. Measured directly: that pattern scores 1028 hits across 119 files (548 across 50 even with an ASVS context marker). A gate that fires 548 times gets switched off -- which is the exact failure the row exists to prevent. The row's diagnosis is right; its specification wasn't tight enough. Built to the measurement instead, and the commit says so.
Verification, explicitly scoped: 137 passed / 2 skipped over test_scan_forbidden plus every module importing the scanner plus test_security_posture. ruff/ruff format/mypy clean, each run separately. Mutation-proved in both directions, hash-verified restores. Full suite not run.