Skip to content

fix(validator): suppress low-confidence ASF-coupling warnings for org-scoped skills#644

Open
justinmclean wants to merge 2 commits into
apache:mainfrom
justinmclean:low-confidence-asf-coupling-pass
Open

fix(validator): suppress low-confidence ASF-coupling warnings for org-scoped skills#644
justinmclean wants to merge 2 commits into
apache:mainfrom
justinmclean:low-confidence-asf-coupling-pass

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

Review and classify all 85 low-confidence asf-coupling SOFT warnings. Outcome: all are intentionally advisory — none warrant a placeholder or capability-flag conversion — classified as one of three cases:

  1. organization: ASF skills (83 warnings across committer-onboarding, contributor-nomination, contributor-to-committer, release-audit-report, release-promote, release-vote-tally): bare PMC, ICLA, and incubator terms are intentional ASF-default prose in skills that are explicitly scoped to ASF. validate_asf_coupling now parses frontmatter and skips low-confidence patterns for organization: ASF skills; high-confidence patterns (svn commands, hardcoded apache.org lists) still fire so those remain behind capability flags even in ASF-only skills.

  2. skill-reconciler line 158 ("ASF PMC roles"): already names the ASF context on the same line — add "ASF PMC" to _ASF_COUPLING_ALLOW_MARKERS.

  3. security-issue-triage line 171 ("PMC members" in a prompt-injection example): PMC appears as an example of attacker-crafted social- engineering text, not as a process instruction — add "prompt-injection" to _ASF_COUPLING_ALLOW_MARKERS.

Result: asf-coupling warning count drops from 85 to 0. The remaining 4 SOFT warnings (action-inventory, security-pattern-9, privacy-llm-gate, gh-list-no-limit) are tracked in the mechanical-soft-warning-cleanup branch. Adds 7 new tests covering the new suppression behaviours; all 271 pass.

Spec note for plan/update beat: update specs/project-agnosticism.md Known Gaps to document the classification and the new warning count (0 asf-coupling hits in the live catalogue; 4 remaining SOFT hits in other categories).

Generated-by: Claude (Opus 4.7)

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other: ran validator

…-scoped skills

Review and classify all 85 low-confidence asf-coupling SOFT warnings.
Outcome: all are intentionally advisory — none warrant a placeholder or
capability-flag conversion — classified as one of three cases:

1. organization: ASF skills (83 warnings across committer-onboarding,
   contributor-nomination, contributor-to-committer, release-audit-report,
   release-promote, release-vote-tally): bare PMC, ICLA, and incubator
   terms are intentional ASF-default prose in skills that are explicitly
   scoped to ASF.  validate_asf_coupling now parses frontmatter and skips
   low-confidence patterns for organization: ASF skills; high-confidence
   patterns (svn commands, hardcoded apache.org lists) still fire so
   those remain behind capability flags even in ASF-only skills.

2. skill-reconciler line 158 ("ASF PMC roles"): already names the ASF
   context on the same line — add "ASF PMC" to _ASF_COUPLING_ALLOW_MARKERS.

3. security-issue-triage line 171 ("PMC members" in a prompt-injection
   example): PMC appears as an example of attacker-crafted social-
   engineering text, not as a process instruction — add "prompt-injection"
   to _ASF_COUPLING_ALLOW_MARKERS.

Result: asf-coupling warning count drops from 85 to 0.  The remaining
4 SOFT warnings (action-inventory, security-pattern-9, privacy-llm-gate,
gh-list-no-limit) are tracked in the mechanical-soft-warning-cleanup branch.
Adds 7 new tests covering the new suppression behaviours; all 271 pass.

Spec note for plan/update beat: update specs/project-agnosticism.md Known
Gaps to document the classification and the new warning count (0 asf-coupling
hits in the live catalogue; 4 remaining SOFT hits in other categories).

Generated-by: Claude (Opus 4.7)
@justinmclean

Copy link
Copy Markdown
Member Author

Changes due to self-review skill.

High-confidence patterns no longer masked by markers. Split ASF PMC and prompt-injection out of _ASF_COUPLING_ALLOW_MARKERS into a new _ASF_COUPLING_LOW_CONF_ALLOW_MARKERS. These no longer short-circuit the whole line, so a same-line high-confidence pattern (e.g. svn commit) still fires while the soft governance mention stays suppressed.

organization: ASF opt-out documented. The docstring now states it's intentional and silent by design (keeps legitimately ASF-scoped release/contributor skills quiet on terms they're supposed to use), gated on the explicit, validated organization: frontmatter key, and only ever silences the advisory low-confidence tier.

Tests. Added boundary cases asserting each marker suppresses the soft mention but lets a same-line high-confidence pattern through; renamed the marker tests to reflect the low-confidence-marker mechanism.

@justinmclean

Copy link
Copy Markdown
Member Author

Correctness

No findings. High-confidence patterns now fire through both low-confidence markers and the organization: ASF opt-out; line_skips_low is computed once per line; parse_frontmatter is called once per file. Re-ran the boundary scenarios against the real validator with the committed test wording — all pass, and the full validator run is clean (4 pre-existing unrelated warnings, no asf-coupling noise).

Security

No findings. The change is strictly more conservative than the prior revision: markers that used to silence an entire line now gate only the advisory tier, so operational patterns (svn, apache.org lists, dist paths) always surface. The organization: ASF opt-out is scoped to the validated frontmatter key and documented as intentional.

Conventions

No findings. The previous advisory (stale *_allow_marker_suppresses test names) is resolved — both are now *_low_conf_marker_suppresses_soft_mention, matching the mechanism. Files are modified (no SPDX requirement); new tests follow the existing class style.

…nce markers

Address self-review findings on the low-confidence ASF-coupling pass:

- Split 'ASF PMC' / 'prompt-injection' out of _ASF_COUPLING_ALLOW_MARKERS
  into _ASF_COUPLING_LOW_CONF_ALLOW_MARKERS. They no longer short-circuit
  the whole line, so a same-line high-confidence pattern (e.g. svn commit)
  still fires while the soft governance mention stays suppressed.
- Document the organization:ASF opt-out trade-off in the docstring: it is
  intentional and silent by design (keeps legitimately ASF-scoped release
  and contributor skills quiet on terms they are supposed to use), gated on
  the explicit, validated organization: frontmatter key, and only ever
  silences the advisory low-confidence tier.
- Tests: assert the marker/high-confidence boundary for both markers; name
  the marker-suppression tests after the low-confidence-marker mechanism.
@justinmclean justinmclean force-pushed the low-confidence-asf-coupling-pass branch from 04b49b4 to 9711700 Compare June 29, 2026 09:25
@justinmclean justinmclean self-assigned this Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant