Skip to content

Fix content_scanner.py: only require SKILL.md frontmatter on SKILL.md itself - #47

Merged
ankitsingh015 merged 1 commit into
mainfrom
claude/content-scanner-frontmatter-fix
Aug 26, 2026
Merged

Fix content_scanner.py: only require SKILL.md frontmatter on SKILL.md itself#47
ankitsingh015 merged 1 commit into
mainfrom
claude/content-scanner-frontmatter-fix

Conversation

@ankitsingh015

Copy link
Copy Markdown
Owner

Summary

Directly implements the review I flagged earlier ("Unreviewed external content" — content_scanner.py exists exactly for this, so 150+ external skill files shouldn't be trusted without actually scanning them). Cloned uphiago/recon-skills and ran content_scanner.py against its full 145-SKILL.md tree.

Found a real bug in the scanner itself: scan_skill_file() required YAML frontmatter on every .md file it was pointed at, not just files literally named SKILL.md. That produced 27 false positives (all references/*.md and top-level docs like README.md/AGENTS.md, which legitimately have no frontmatter — only SKILL.md is the frontmatter contract, matching this repo's own .claude/skills/<name>/{SKILL.md, references/} layout).

Fixed by scoping the frontmatter-specific checks to os.path.basename(path) == "SKILL.md" — the content-safety checks (prompt-injection phrasing, hidden Unicode, base64 blobs) still apply to every .md file regardless of filename.

Scan result after the fix: 5 HIGH + 1 MEDIUM, all manually verified as legitimate — every "prompt-injection" hit is inside a skill that specifically hunts for prompt injection (hunt-llm-ai, llm-prompt-injection, hunt-mcp-security, etc.), so it necessarily documents the injection phrasing as an example test payload — the exact same pattern as this repo's own known emerging-surfaces false positive. The one base64 blob is a synthetic ASP.NET __VIEWSTATE example. No genuine malicious content found.

Test plan

  • 2 new regression tests: a references/*.md without frontmatter is no longer flagged; the same file still gets flagged for genuine content issues (prompt-injection phrasing)
  • All 8 existing content_scanner tests still pass (all use SKILL.md as the filename, so the fix doesn't change their behavior)
  • Full 205-test suite passes
  • Re-ran the real external-repo scan before/after to confirm the false-positive count dropped from 27→0 and the 6 remaining findings are genuine (verified by hand, cited above)

… itself

Found while actually running content_scanner.py against all 145 SKILL.md
files of an external skill repo (uphiago/recon-skills), per the "don't
trust unreviewed external content" gap flagged earlier -- scan_skill_file()
required YAML frontmatter on every *.md file it was pointed at, not just
files literally named SKILL.md. That repo's references/*.md and top-level
docs legitimately have none (only SKILL.md is the frontmatter contract,
per its own AGENTS.md convention, matching this repo's own
.claude/skills/<name>/{SKILL.md, references/} layout). Produced 27 false
positives out of 33 findings on the real scan -- prompt-injection-pattern
and hidden-Unicode checks still apply to non-SKILL.md files; only the
frontmatter-specific checks are now scoped to SKILL.md.

Full external-repo scan result after the fix: 5 HIGH + 1 MEDIUM, all
manually verified as legitimate (skills that hunt FOR prompt injection
necessarily document injection phrasing as example payloads -- same
pattern as this repo's own emerging-surfaces false positive; the one
base64 blob is a synthetic ASP.NET ViewState example). No genuine
malicious content found.
@ankitsingh015
ankitsingh015 merged commit 612c850 into main Aug 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant