fix(validator): suppress low-confidence ASF-coupling warnings for org-scoped skills#644
fix(validator): suppress low-confidence ASF-coupling warnings for org-scoped skills#644justinmclean wants to merge 2 commits into
Conversation
…-scoped skills
Review and classify all 85 low-confidence asf-coupling SOFT warnings.
Outcome: all are intentionally advisory — none warrant a placeholder or
capability-flag conversion — classified as one of three cases:
1. organization: ASF skills (83 warnings across committer-onboarding,
contributor-nomination, contributor-to-committer, release-audit-report,
release-promote, release-vote-tally): bare PMC, ICLA, and incubator
terms are intentional ASF-default prose in skills that are explicitly
scoped to ASF. validate_asf_coupling now parses frontmatter and skips
low-confidence patterns for organization: ASF skills; high-confidence
patterns (svn commands, hardcoded apache.org lists) still fire so
those remain behind capability flags even in ASF-only skills.
2. skill-reconciler line 158 ("ASF PMC roles"): already names the ASF
context on the same line — add "ASF PMC" to _ASF_COUPLING_ALLOW_MARKERS.
3. security-issue-triage line 171 ("PMC members" in a prompt-injection
example): PMC appears as an example of attacker-crafted social-
engineering text, not as a process instruction — add "prompt-injection"
to _ASF_COUPLING_ALLOW_MARKERS.
Result: asf-coupling warning count drops from 85 to 0. The remaining
4 SOFT warnings (action-inventory, security-pattern-9, privacy-llm-gate,
gh-list-no-limit) are tracked in the mechanical-soft-warning-cleanup branch.
Adds 7 new tests covering the new suppression behaviours; all 271 pass.
Spec note for plan/update beat: update specs/project-agnosticism.md Known
Gaps to document the classification and the new warning count (0 asf-coupling
hits in the live catalogue; 4 remaining SOFT hits in other categories).
Generated-by: Claude (Opus 4.7)
|
Changes due to self-review skill. High-confidence patterns no longer masked by markers. Split ASF PMC and prompt-injection out of _ASF_COUPLING_ALLOW_MARKERS into a new _ASF_COUPLING_LOW_CONF_ALLOW_MARKERS. These no longer short-circuit the whole line, so a same-line high-confidence pattern (e.g. svn commit) still fires while the soft governance mention stays suppressed. organization: ASF opt-out documented. The docstring now states it's intentional and silent by design (keeps legitimately ASF-scoped release/contributor skills quiet on terms they're supposed to use), gated on the explicit, validated organization: frontmatter key, and only ever silences the advisory low-confidence tier. Tests. Added boundary cases asserting each marker suppresses the soft mention but lets a same-line high-confidence pattern through; renamed the marker tests to reflect the low-confidence-marker mechanism. |
CorrectnessNo findings. High-confidence patterns now fire through both low-confidence markers and the organization: ASF opt-out; line_skips_low is computed once per line; parse_frontmatter is called once per file. Re-ran the boundary scenarios against the real validator with the committed test wording — all pass, and the full validator run is clean (4 pre-existing unrelated warnings, no asf-coupling noise). SecurityNo findings. The change is strictly more conservative than the prior revision: markers that used to silence an entire line now gate only the advisory tier, so operational patterns (svn, apache.org lists, dist paths) always surface. The organization: ASF opt-out is scoped to the validated frontmatter key and documented as intentional. ConventionsNo findings. The previous advisory (stale *_allow_marker_suppresses test names) is resolved — both are now *_low_conf_marker_suppresses_soft_mention, matching the mechanism. Files are modified (no SPDX requirement); new tests follow the existing class style. |
…nce markers Address self-review findings on the low-confidence ASF-coupling pass: - Split 'ASF PMC' / 'prompt-injection' out of _ASF_COUPLING_ALLOW_MARKERS into _ASF_COUPLING_LOW_CONF_ALLOW_MARKERS. They no longer short-circuit the whole line, so a same-line high-confidence pattern (e.g. svn commit) still fires while the soft governance mention stays suppressed. - Document the organization:ASF opt-out trade-off in the docstring: it is intentional and silent by design (keeps legitimately ASF-scoped release and contributor skills quiet on terms they are supposed to use), gated on the explicit, validated organization: frontmatter key, and only ever silences the advisory low-confidence tier. - Tests: assert the marker/high-confidence boundary for both markers; name the marker-suppression tests after the low-confidence-marker mechanism.
04b49b4 to
9711700
Compare
Summary
Review and classify all 85 low-confidence asf-coupling SOFT warnings. Outcome: all are intentionally advisory — none warrant a placeholder or capability-flag conversion — classified as one of three cases:
organization: ASF skills (83 warnings across committer-onboarding, contributor-nomination, contributor-to-committer, release-audit-report, release-promote, release-vote-tally): bare PMC, ICLA, and incubator terms are intentional ASF-default prose in skills that are explicitly scoped to ASF. validate_asf_coupling now parses frontmatter and skips low-confidence patterns for organization: ASF skills; high-confidence patterns (svn commands, hardcoded apache.org lists) still fire so those remain behind capability flags even in ASF-only skills.
skill-reconciler line 158 ("ASF PMC roles"): already names the ASF context on the same line — add "ASF PMC" to _ASF_COUPLING_ALLOW_MARKERS.
security-issue-triage line 171 ("PMC members" in a prompt-injection example): PMC appears as an example of attacker-crafted social- engineering text, not as a process instruction — add "prompt-injection" to _ASF_COUPLING_ALLOW_MARKERS.
Result: asf-coupling warning count drops from 85 to 0. The remaining 4 SOFT warnings (action-inventory, security-pattern-9, privacy-llm-gate, gh-list-no-limit) are tracked in the mechanical-soft-warning-cleanup branch. Adds 7 new tests covering the new suppression behaviours; all 271 pass.
Spec note for plan/update beat: update specs/project-agnosticism.md Known Gaps to document the classification and the new warning count (0 asf-coupling hits in the live catalogue; 4 remaining SOFT hits in other categories).
Generated-by: Claude (Opus 4.7)
Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)