diff --git a/README.md b/README.md index be4ea30..1ea6ac4 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Agent Skills compatible

-**SkillConform is the open-source CI and test suite for Agent Skills: conformance, capability inventory, deterministic security review, and regression testing.** It catches broken metadata, unsafe instructions, missing resources, embedded secrets, broad tool grants, and policy regressions before a skill reaches an agent. +**SkillConform is the open-source CI and test suite for Agent Skills: conformance, capability inventory, cross-agent compatibility evidence, deterministic security review, and regression testing.** It catches broken metadata, unsafe instructions, missing resources, embedded secrets, broad tool grants, and policy regressions before a skill reaches an agent. Agent Skills are portable folders built around `SKILL.md`. The reference validator checks the core format; SkillConform adds a complementary quality layer for security review, workspace scanning, policy tests, CI, JSON, and SARIF. @@ -18,7 +18,7 @@ Agent Skills are portable folders built around `SKILL.md`. The reference validat SkillConform scanning seven Agent Skills and returning a clean pass

-> Status: `v0.1.1` developer preview. The npm CLI, library API, and GitHub Action are ready for early adopters. Security findings are evidence for review, not a certification. +> Status: `v0.3.0` development line on `main`. The CLI, library API, and GitHub Action are ready for early adopters; published registry releases may lag `main`. Static compatibility and security results are evidence for review, not runtime certification. ## Quick start @@ -45,6 +45,15 @@ npx skillconform inventory ./skills --format json -o capabilities.json The versioned `skillconform.capabilities/v1` document records declared tools, local references, bundled scripts, detected network hosts, shell/network signals, and portability notes. It is evidence for review and CI policy—not a claim that a skill is safe or compatible with every agent. +Compare the same skill against documented host constraints: + +```bash +npx skillconform matrix ./skills +npx skillconform matrix ./skills --format json -o compatibility.json +``` + +The versioned `skillconform.compatibility/v1` matrix starts with Agent Skills, Claude Code, Claude API, OpenAI Skills, and Gemini CLI profiles. `PASS` means no static blocker was found, `REVIEW` means runtime/host evidence is still needed, and `UNSUPPORTED` means a documented constraint is contradicted. See [cross-agent compatibility profiles](docs/compatibility.md). + For a guided walkthrough, follow the [five-minute demo](docs/quick-demo.md). To run from source: @@ -64,6 +73,7 @@ node dist/src/cli.js audit ./skills --fail-on warning | `check [path]` | Validate Agent Skills structure, metadata, naming, size, and references. | | `audit [path]` | Run conformance checks plus deterministic security rules. | | `inventory [path]` | Produce a deterministic capability/portability inventory. | +| `matrix [path]` | Compare static evidence with versioned host compatibility profiles. | | `test [file]` | Execute regression expectations from `skillconform.yaml`. | | `init [file]` | Create a starter policy-test configuration. | @@ -167,7 +177,7 @@ process.stdout.write(renderReport(result, "json")); - **CI-native:** reports work in terminals, scripts, and SARIF consumers. - **Least privilege:** security rules favor explicit tools, paths, and network requirements. -Read [the architecture](docs/architecture.md), [SkillConform 2.0 direction](docs/skillconform-2.md), [roadmap](ROADMAP.md), and [contribution guide](CONTRIBUTING.md) before proposing larger changes. +Read [the architecture](docs/architecture.md), [cross-agent compatibility profiles](docs/compatibility.md), [SkillConform 2.0 direction](docs/skillconform-2.md), [roadmap](ROADMAP.md), and [contribution guide](CONTRIBUTING.md) before proposing larger changes. ## Community and launch resources diff --git a/ROADMAP.md b/ROADMAP.md index 61a6b96..31e16ee 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -8,17 +8,18 @@ SkillConform is evolving toward an open-source CI and test suite for Agent Skill - [x] CLI and TypeScript API for capability inventory. - [x] Deterministic inventory of declared tools, references, bundled scripts, network hosts, and portability notes. - [ ] JSON Schema for capability inventory and `skillconform.yaml`. -- [ ] Explicit client compatibility profiles. +- [x] Explicit client compatibility profiles. - [ ] Configurable rule severities and documented suppressions. - [ ] Incremental scanning for large repositories. ## v0.3 — Cross-agent compatibility -- Compatibility adapters for major Agent Skills hosts. -- `matrix` command with tested / untested / unsupported evidence per profile. -- Community-maintained compatibility fixtures. -- Shell and PowerShell command-aware analysis. -- Reusable CI examples for multi-agent skill repositories. +- [x] Versioned compatibility profiles for Agent Skills, Claude Code, Claude API, OpenAI Skills, and Gemini CLI. +- [x] `skillconform.compatibility/v1` matrix with pass / review / unsupported evidence. +- [x] `matrix` CLI, TypeScript API, GitHub Action mode, and provider-specific fixtures. +- [ ] Community-maintained compatibility fixtures. +- [ ] Shell and PowerShell command-aware analysis. +- [ ] Reusable CI examples for multi-agent skill repositories. ## v0.4 — Behavioral regression diff --git a/action.yml b/action.yml index 1b3cf1b..7c2c8f4 100644 --- a/action.yml +++ b/action.yml @@ -1,5 +1,5 @@ name: SkillConform -description: Validate, inventory, and security-audit Agent Skills in GitHub Actions. +description: Validate, inventory, compare host compatibility, and security-audit Agent Skills in GitHub Actions. author: Ramzi Sultan branding: icon: check-circle @@ -10,7 +10,7 @@ inputs: required: false default: . mode: - description: Mode: check, audit, or inventory. + description: Mode: check, audit, inventory, or matrix. required: false default: audit fail-on: @@ -45,10 +45,10 @@ runs: SKILLCONFORM_FORMAT: ${{ inputs.format }} SKILLCONFORM_OUTPUT: ${{ inputs.output }} run: | - case "$SKILLCONFORM_MODE" in check|audit|inventory) ;; *) echo "mode must be check, audit, or inventory" >&2; exit 2 ;; esac + case "$SKILLCONFORM_MODE" in check|audit|inventory|matrix) ;; *) echo "mode must be check, audit, inventory, or matrix" >&2; exit 2 ;; esac case "$SKILLCONFORM_FORMAT" in pretty|json|sarif) ;; *) echo "format must be pretty, json, or sarif" >&2; exit 2 ;; esac - if [[ "$SKILLCONFORM_MODE" == "inventory" && "$SKILLCONFORM_FORMAT" == "sarif" ]]; then - echo "inventory mode supports pretty or json format" >&2 + if [[ ( "$SKILLCONFORM_MODE" == "inventory" || "$SKILLCONFORM_MODE" == "matrix" ) && "$SKILLCONFORM_FORMAT" == "sarif" ]]; then + echo "$SKILLCONFORM_MODE mode supports pretty or json format" >&2 exit 2 fi case "$SKILLCONFORM_FAIL_ON" in error|warning|note|none) ;; *) echo "fail-on has an invalid value" >&2; exit 2 ;; esac diff --git a/docs/compatibility.md b/docs/compatibility.md new file mode 100644 index 0000000..df3f943 --- /dev/null +++ b/docs/compatibility.md @@ -0,0 +1,53 @@ +# Cross-agent compatibility profiles + +SkillConform compatibility profiles turn documented host constraints into conservative, machine-readable checks. + +The command is: + +```bash +npx skillconform matrix ./skills +npx skillconform matrix ./skills --format json -o compatibility.json +``` + +The JSON schema identifier is `skillconform.compatibility/v1`. The initial profile set is versioned `2026-09-23`. + +## Status semantics + +- `pass`: no blocker is visible from deterministic static evidence for the checks SkillConform currently implements. +- `review`: the skill may work, but host-specific runtime, permissions, discovery placement, or dependencies still need verification. +- `unsupported`: the skill contradicts a documented constraint for that profile. + +A `pass` is **not** a claim that the skill was executed successfully on that agent. Runtime claims belong in the future behavioral evaluation layer. + +## Initial profiles + +| Profile | Evidence encoded | +|---|---| +| Agent Skills standard | Core SkillConform conformance baseline; runtime is host-defined. | +| Claude Code | Claude-specific name constraints, local network model, and project discovery under `.claude/skills//SKILL.md`. | +| Claude API | Claude-specific name constraints plus the documented no-network and no-runtime-package-install environment. | +| OpenAI Skills | Open Agent Skills compatibility; host-specific permissions are surfaced for review rather than guessed. | +| Gemini CLI | Documented project discovery under `.agents/skills//SKILL.md`; host-specific permission behavior remains reviewable. | + +## Sources + +Profiles intentionally cite first-party documentation: + +- Agent Skills specification: https://agentskills.io/specification +- Anthropic Agent Skills: https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview +- OpenAI Skills: https://developers.openai.com/api/docs/guides/tools-skills +- Google Gemini CLI Agent Skills codelab: https://codelabs.developers.google.com/gemini-cli/how-to-create-agent-skills-for-gemini-cli + +When provider behavior changes, update the profile set version, add or change fixtures, and explain the change in the pull request. + +## Non-goals + +The matrix does not: + +- claim that every runtime feature is statically knowable; +- execute untrusted skill scripts; +- infer unpublished provider behavior; +- convert a host-specific permission declaration into a universal permission model; +- replace real runtime evaluations. + +The behavioral regression layer will add executable evidence separately so static portability and runtime behavior remain auditable concepts. diff --git a/package-lock.json b/package-lock.json index d92af8c..ee979ee 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "skillconform", - "version": "0.2.0", + "version": "0.3.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "skillconform", - "version": "0.2.0", + "version": "0.3.0", "license": "Apache-2.0", "dependencies": { "yaml": "^2.8.1" diff --git a/package.json b/package.json index 5151c69..1fc2e6f 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "skillconform", - "version": "0.2.0", - "description": "CI-native conformance, capability inventory, security review, and regression testing for Agent Skills.", + "version": "0.3.0", + "description": "CI-native conformance, capability inventory, cross-agent compatibility, security review, and regression testing for Agent Skills.", "type": "module", "bin": { "skillconform": "dist/src/cli.js" @@ -34,6 +34,8 @@ "skill-linter", "claude-code", "codex", + "gemini-cli", + "cross-agent", "security", "sarif" ], diff --git a/src/cli.ts b/src/cli.ts index 210f7d6..9412e0b 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1,12 +1,13 @@ #!/usr/bin/env node import { resolve } from "node:path"; +import { buildCompatibilityMatrix, renderCompatibilityMatrix } from "./compatibility.js"; import { buildCapabilityInventory, renderCapabilityInventory } from "./inventory.js"; import { createPolicyConfig, runPolicyTests } from "./policy-tests.js"; import { renderReport, shouldFail, writeReport } from "./reporters.js"; import { scanTarget } from "./scanner.js"; import type { CliOptions, OutputFormat, Severity } from "./types.js"; -const VERSION = "0.2.0"; +const VERSION = "0.3.0"; interface ParsedArguments { command: string; @@ -48,6 +49,13 @@ async function main(): Promise { return; } + if (parsed.command === "matrix") { + if (parsed.options.format === "sarif") throw new Error("The matrix command supports pretty or json output."); + const matrix = buildCompatibilityMatrix(parsed.target); + writeReport(renderCompatibilityMatrix(matrix, parsed.options.format), parsed.options.output); + return; + } + const mode = parsed.command === "audit" ? "audit" : "check"; const result = scanTarget(parsed.target, mode); if (!parsed.options.quiet || parsed.options.output) { @@ -66,7 +74,7 @@ function parseArguments(args: string[]): ParsedArguments { if (args.includes("--version") || args.includes("-v")) return { command: "version", target: ".", options: defaults() }; const command = args[0] ?? "help"; - if (!["check", "audit", "inventory", "test", "init"].includes(command)) throw new Error(`Unknown command '${command}'. Run skillconform --help.`); + if (!["check", "audit", "inventory", "matrix", "test", "init"].includes(command)) throw new Error(`Unknown command '${command}'. Run skillconform --help.`); let target = command === "test" ? "skillconform.yaml" : "."; const options = defaults(); @@ -76,7 +84,7 @@ function parseArguments(args: string[]): ParsedArguments { if (arg === "--format") { const value = args[++index]; if (value !== "pretty" && value !== "json" && value !== "sarif") throw new Error("--format must be pretty, json, or sarif."); - if ((command === "test" || command === "inventory") && value === "sarif") throw new Error(`The ${command} command supports pretty or json output.`); + if ((command === "test" || command === "inventory" || command === "matrix") && value === "sarif") throw new Error(`The ${command} command supports pretty or json output.`); options.format = value as OutputFormat; } else if (arg === "--output" || arg === "-o") { const value = args[++index]; diff --git a/src/compatibility.ts b/src/compatibility.ts new file mode 100644 index 0000000..f81c9a9 --- /dev/null +++ b/src/compatibility.ts @@ -0,0 +1,221 @@ +import { relative, resolve, sep } from "node:path"; +import { buildCapabilityInventory } from "./inventory.js"; +import { scanTarget } from "./scanner.js"; +import type { + CompatibilityCheck, + CompatibilityMatrix, + CompatibilityProfile, + CompatibilityResult, + CompatibilityStatus, + SkillCapabilityProfile, +} from "./types.js"; + +export const COMPATIBILITY_PROFILE_SET_VERSION = "2026-09-23"; + +const PROFILES: readonly CompatibilityProfile[] = [ + { + id: "agent-skills", + label: "Agent Skills standard", + source: "https://agentskills.io/specification", + networkPolicy: "host-defined", + runtimePackageInstall: "host-defined", + discoveryPattern: null, + notes: "Portable baseline. Runtime permissions and execution environment are host-defined.", + }, + { + id: "claude-code", + label: "Claude Code", + source: "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview", + networkPolicy: "full", + runtimePackageInstall: "discouraged", + discoveryPattern: ".claude/skills//SKILL.md", + notes: "Project skills are auto-discovered from .claude/skills. Skills share the local Claude Code execution environment.", + }, + { + id: "claude-api", + label: "Claude API", + source: "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview", + networkPolicy: "none", + runtimePackageInstall: "none", + discoveryPattern: null, + notes: "Custom skills run in the code-execution container with no network access and no runtime package installation.", + }, + { + id: "openai-skills", + label: "OpenAI Skills", + source: "https://developers.openai.com/api/docs/guides/tools-skills", + networkPolicy: "host-defined", + runtimePackageInstall: "host-defined", + discoveryPattern: null, + notes: "OpenAI supports the open Agent Skills format in local and hosted execution surfaces; host behavior can differ.", + }, + { + id: "gemini-cli", + label: "Gemini CLI", + source: "https://codelabs.developers.google.com/gemini-cli/how-to-create-agent-skills-for-gemini-cli", + networkPolicy: "host-defined", + runtimePackageInstall: "host-defined", + discoveryPattern: ".agents/skills//SKILL.md", + notes: "Project skills are discovered from .agents/skills in the documented Gemini CLI workflow.", + }, +]; + +export function getCompatibilityProfiles(): CompatibilityProfile[] { + return PROFILES.map((profile) => ({ ...profile })); +} + +export function buildCompatibilityMatrix(targetInput: string): CompatibilityMatrix { + const target = resolve(targetInput); + const inventory = buildCapabilityInventory(target); + const conformance = scanTarget(target, "check"); + const errorsByFile = new Map(); + + for (const finding of conformance.findings) { + if (finding.severity !== "error") continue; + errorsByFile.set(finding.file, (errorsByFile.get(finding.file) ?? 0) + 1); + } + + return { + schemaVersion: "skillconform.compatibility/v1", + profileSetVersion: COMPATIBILITY_PROFILE_SET_VERSION, + target: inventory.target, + profiles: getCompatibilityProfiles(), + skills: inventory.skills.map((skill) => ({ + skillFile: skill.skillFile, + ...(skill.name ? { name: skill.name } : {}), + results: PROFILES.map((profile) => evaluateProfile( + skill, + profile, + target, + errorsByFile.get(skill.skillFile) ?? 0, + )), + })), + }; +} + +export function renderCompatibilityMatrix(matrix: CompatibilityMatrix, format: "pretty" | "json"): string { + if (format === "json") return `${JSON.stringify(matrix, null, 2)}\n`; + + const lines = [ + "SkillConform compatibility matrix", + `Profile set: ${matrix.profileSetVersion}`, + `Scanned ${matrix.skills.length} skill${matrix.skills.length === 1 ? "" : "s"}.`, + ]; + + for (const skill of matrix.skills) { + lines.push("", `Skill: ${skill.name ?? skill.skillFile}`); + for (const result of skill.results) { + const profile = matrix.profiles.find((item) => item.id === result.profileId); + lines.push(` ${statusLabel(result.status).padEnd(11)} ${profile?.label ?? result.profileId}`); + for (const check of result.checks) { + if (check.status !== "pass") lines.push(` - ${check.id}: ${check.reason}`); + } + } + } + + lines.push("", "PASS = no static portability blocker found; REVIEW = host/runtime evidence is still required; UNSUPPORTED = documented constraint is contradicted."); + return `${lines.join("\n")}\n`; +} + +function evaluateProfile( + skill: SkillCapabilityProfile, + profile: CompatibilityProfile, + targetRoot: string, + conformanceErrors: number, +): CompatibilityResult { + const checks: CompatibilityCheck[] = []; + + checks.push({ + id: "conformance", + status: conformanceErrors > 0 ? "unsupported" : "pass", + reason: conformanceErrors > 0 + ? `${conformanceErrors} Agent Skills conformance error(s) must be fixed first.` + : "No core conformance error was found.", + }); + + if (profile.id === "claude-code" || profile.id === "claude-api") { + const name = skill.name?.toLowerCase() ?? ""; + const reserved = /(?:^|-)(?:anthropic|claude)(?:-|$)/.test(name); + checks.push({ + id: "claude-name", + status: reserved ? "unsupported" : "pass", + reason: reserved + ? "Claude skill names cannot contain the reserved words 'anthropic' or 'claude'." + : "No Claude-reserved word was detected in the skill name.", + }); + } + + if (profile.networkPolicy === "none") { + checks.push({ + id: "network", + status: skill.capabilities.network ? "unsupported" : "pass", + reason: skill.capabilities.network + ? `Network use was detected${skill.networkHosts.length ? ` for ${skill.networkHosts.join(", ")}` : ""}, but this profile documents no network access.` + : "No network requirement was detected.", + }); + } else if (profile.networkPolicy === "host-defined" && skill.capabilities.network) { + checks.push({ + id: "network", + status: "review", + reason: "Network use was detected; availability and allowlists depend on the selected host/runtime.", + }); + } else { + checks.push({ + id: "network", + status: "pass", + reason: skill.capabilities.network ? "Network use is compatible with the documented runtime model." : "No network requirement was detected.", + }); + } + + if (profile.id === "claude-api" && skill.capabilities.bundledScripts) { + checks.push({ + id: "runtime-dependencies", + status: "review", + reason: "Bundled scripts are supported, but the Claude API runtime cannot install packages at execution time; required dependencies must already exist.", + }); + } + + if ((profile.id === "openai-skills" || profile.id === "gemini-cli") && skill.allowedTools.length > 0) { + checks.push({ + id: "tool-permissions", + status: "review", + reason: "The skill declares allowed-tools; permission syntax and enforcement are host-specific and require runtime verification.", + }); + } + + if (profile.discoveryPattern) { + const relativeSkill = normalizePath(relative(targetRoot, resolve(process.cwd(), skill.skillFile))); + const discovered = profile.id === "claude-code" + ? /^\.claude\/skills\/[^/]+\/SKILL\.md$/.test(relativeSkill) + : /^\.agents\/skills\/[^/]+\/SKILL\.md$/.test(relativeSkill); + checks.push({ + id: "project-discovery", + status: discovered ? "pass" : "review", + reason: discovered + ? `The skill is located at the documented project discovery path (${profile.discoveryPattern}).` + : `Automatic project discovery expects ${profile.discoveryPattern}; direct, plugin, or user-level installation may use a different path.`, + }); + } + + return { + profileId: profile.id, + status: summarizeStatus(checks), + checks, + }; +} + +function summarizeStatus(checks: CompatibilityCheck[]): CompatibilityStatus { + if (checks.some((check) => check.status === "unsupported")) return "unsupported"; + if (checks.some((check) => check.status === "review")) return "review"; + return "pass"; +} + +function statusLabel(status: CompatibilityStatus): string { + if (status === "unsupported") return "UNSUPPORTED"; + if (status === "review") return "REVIEW"; + return "PASS"; +} + +function normalizePath(value: string): string { + return value.split(sep).join("/"); +} diff --git a/src/index.ts b/src/index.ts index 5cbd177..e3852d9 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,3 +1,4 @@ +export { buildCompatibilityMatrix, getCompatibilityProfiles, renderCompatibilityMatrix } from "./compatibility.js"; export { parseSkill } from "./frontmatter.js"; export { buildCapabilityInventory, renderCapabilityInventory } from "./inventory.js"; export { runPolicyTests } from "./policy-tests.js"; @@ -5,6 +6,11 @@ export { renderReport, shouldFail } from "./reporters.js"; export { discoverSkillFiles, scanTarget } from "./scanner.js"; export type { CapabilityInventory, + CompatibilityCheck, + CompatibilityMatrix, + CompatibilityProfile, + CompatibilityResult, + CompatibilityStatus, Finding, OutputFormat, ParsedSkill, @@ -14,4 +20,5 @@ export type { ScanResult, Severity, SkillCapabilityProfile, + SkillCompatibilityResult, } from "./types.js"; diff --git a/src/types.ts b/src/types.ts index e94bb66..0841851 100644 --- a/src/types.ts +++ b/src/types.ts @@ -60,6 +60,46 @@ export interface CapabilityInventory { skills: SkillCapabilityProfile[]; } +export type CompatibilityStatus = "pass" | "review" | "unsupported"; +export type NetworkPolicy = "full" | "none" | "host-defined"; +export type RuntimePackageInstall = "allowed" | "none" | "discouraged" | "host-defined"; + +export interface CompatibilityProfile { + id: "agent-skills" | "claude-code" | "claude-api" | "openai-skills" | "gemini-cli"; + label: string; + source: string; + networkPolicy: NetworkPolicy; + runtimePackageInstall: RuntimePackageInstall; + discoveryPattern: string | null; + notes: string; +} + +export interface CompatibilityCheck { + id: string; + status: CompatibilityStatus; + reason: string; +} + +export interface CompatibilityResult { + profileId: CompatibilityProfile["id"]; + status: CompatibilityStatus; + checks: CompatibilityCheck[]; +} + +export interface SkillCompatibilityResult { + skillFile: string; + name?: string; + results: CompatibilityResult[]; +} + +export interface CompatibilityMatrix { + schemaVersion: "skillconform.compatibility/v1"; + profileSetVersion: string; + target: string; + profiles: CompatibilityProfile[]; + skills: SkillCompatibilityResult[]; +} + export interface CliOptions { format: OutputFormat; output?: string; diff --git a/test/compatibility.test.ts b/test/compatibility.test.ts new file mode 100644 index 0000000..e51253e --- /dev/null +++ b/test/compatibility.test.ts @@ -0,0 +1,42 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { join } from "node:path"; +import { buildCompatibilityMatrix, getCompatibilityProfiles, renderCompatibilityMatrix } from "../src/compatibility.js"; + +const fixtures = join(process.cwd(), "test", "fixtures"); + +test("publishes versioned compatibility profiles", () => { + const profiles = getCompatibilityProfiles(); + assert.deepEqual( + profiles.map((profile) => profile.id), + ["agent-skills", "claude-code", "claude-api", "openai-skills", "gemini-cli"], + ); + assert.ok(profiles.every((profile) => profile.source.startsWith("https://"))); +}); + +test("builds a compatibility matrix for a portable skill", () => { + const result = buildCompatibilityMatrix(join(fixtures, "valid-skill")); + assert.equal(result.schemaVersion, "skillconform.compatibility/v1"); + assert.equal(result.skills.length, 1); + assert.equal(result.skills[0]?.results.length, 5); + assert.equal(result.skills[0]?.results.find((item) => item.profileId === "agent-skills")?.status, "pass"); +}); + +test("flags detected network use as unsupported for Claude API", () => { + const result = buildCompatibilityMatrix(join(fixtures, "risky-skill")); + const claudeApi = result.skills[0]?.results.find((item) => item.profileId === "claude-api"); + assert.equal(claudeApi?.status, "unsupported"); + assert.ok(claudeApi?.checks.some((check) => check.id === "network" && check.status === "unsupported")); +}); + +test("enforces documented Claude reserved words without applying them to the open standard", () => { + const result = buildCompatibilityMatrix(join(fixtures, "claude-named-skill")); + assert.equal(result.skills[0]?.results.find((item) => item.profileId === "agent-skills")?.status, "pass"); + assert.equal(result.skills[0]?.results.find((item) => item.profileId === "claude-code")?.status, "unsupported"); +}); + +test("renders the matrix as JSON", () => { + const result = buildCompatibilityMatrix(join(fixtures, "valid-skill")); + const parsed = JSON.parse(renderCompatibilityMatrix(result, "json")); + assert.equal(parsed.profileSetVersion, "2026-09-23"); +}); diff --git a/test/fixtures/claude-named-skill/SKILL.md b/test/fixtures/claude-named-skill/SKILL.md new file mode 100644 index 0000000..85b2003 --- /dev/null +++ b/test/fixtures/claude-named-skill/SKILL.md @@ -0,0 +1,8 @@ +--- +name: claude-named-skill +description: Demonstrates a structurally valid Agent Skill name that is reserved by Claude-specific authoring requirements. Use only as a compatibility test fixture. +--- + +# Compatibility fixture + +Return a short deterministic message.