Add deterministic baseline-to-candidate regression gate

.github/workflows/release.yml

@@ -26,12 +26,19 @@ jobs:
       - uses: actions/checkout@v7
         with:
           fetch-depth: 0
+          persist-credentials: false
 
       - uses: actions/setup-node@v6
         with:
-          node-version: 20
+          node-version: 24
           registry-url: https://registry.npmjs.org
-          cache: npm
+          package-manager-cache: false
+
+      - name: Use npm with trusted-publishing support
+        run: |
+          npm install --global npm@^11.15.0
+          node --version
+          npm --version
 
       - name: Install dependencies
         run: npm ci
@@ -65,11 +72,9 @@ jobs:
             echo "published=false" >> "$GITHUB_OUTPUT"
           fi
 
-      - name: Publish to npm
+      - name: Publish to npm with OIDC
         if: steps.npm.outputs.published != 'true'
-        run: npm publish --access public --provenance
-        env:
-          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+        run: npm publish --access public
 
       - name: Check GitHub release
         id: release

CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes follow the principles of Keep a Changelog and Semantic Versi
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-09-23
+
+### Added
+
+- Baseline-to-candidate regression reports with the `skillconform.regression/v1` contract.
+- Deterministic detection of new findings, network/shell capability expansion, bundled scripts, tool permission expansion, and new network hosts.
+- `regress` CLI and TypeScript API plus GitHub Action support for pull-request gates.
+
+### Fixed
+
+- Normalize detected URL evidence to hostnames instead of storing URL paths in `networkHosts`.
+
+## [0.3.0] - 2026-09-23
+
+### Added
+
+- Versioned cross-agent compatibility profiles and `skillconform.compatibility/v1`.
+- `matrix` CLI/API support for Agent Skills, Claude Code, Claude API, OpenAI Skills, and Gemini CLI.
+- Conservative PASS / REVIEW / UNSUPPORTED evidence semantics and provider-specific fixtures.
+
+## [0.2.0] - 2026-09-23
+
+### Added
+
+- Versioned capability inventory with `skillconform.capabilities/v1`.
+- `inventory` CLI/API and GitHub Action mode for tools, references, scripts, network hosts, and portability notes.
+
+
 ## [0.1.1] - 2026-08-04
 
 ### Fixed

README.md

@@ -18,7 +18,7 @@ Agent Skills are portable folders built around `SKILL.md`. The reference validat
   <img src="docs/demo.svg" width="900" alt="SkillConform scanning seven Agent Skills and returning a clean pass">
 </p>
 
-> Status: `v0.3.0` development line on `main`. The CLI, library API, and GitHub Action are ready for early adopters; published registry releases may lag `main`. Static compatibility and security results are evidence for review, not runtime certification.
+> Status: `v0.4.0` development line on `main`. The CLI, library API, and GitHub Action are ready for early adopters; published registry releases may lag `main`. Static compatibility and security results are evidence for review, not runtime certification.
 
 ## Quick start
 
@@ -54,6 +54,15 @@ npx skillconform matrix ./skills --format json -o compatibility.json
 
 The versioned `skillconform.compatibility/v1` matrix starts with Agent Skills, Claude Code, Claude API, OpenAI Skills, and Gemini CLI profiles. `PASS` means no static blocker was found, `REVIEW` means runtime/host evidence is still needed, and `UNSUPPORTED` means a documented constraint is contradicted. See [cross-agent compatibility profiles](docs/compatibility.md).
 
+Gate a pull request against a baseline without executing target scripts:
+
+```bash
+npx skillconform regress ./candidate/skills --baseline ./baseline/skills
+npx skillconform regress ./candidate/skills --baseline ./baseline/skills --fail-on warning
+```
+
+The versioned `skillconform.regression/v1` report detects newly introduced findings and trust-surface expansion such as network, shell, bundled scripts, tool permissions, and hosts. See [baseline-to-candidate regression](docs/regression.md).
+
 For a guided walkthrough, follow the [five-minute demo](docs/quick-demo.md).
 
 To run from source:
@@ -74,6 +83,7 @@ node dist/src/cli.js audit ./skills --fail-on warning
 | `audit [path]` | Run conformance checks plus deterministic security rules. |
 | `inventory [path]` | Produce a deterministic capability/portability inventory. |
 | `matrix [path]` | Compare static evidence with versioned host compatibility profiles. |
+| `regress [candidate] --baseline <path>` | Compare a candidate tree against a baseline and gate trust-surface regressions. |
 | `test [file]` | Execute regression expectations from `skillconform.yaml`. |
 | `init [file]` | Create a starter policy-test configuration. |
 
@@ -158,7 +168,7 @@ jobs:
           fail-on: warning
 ```
 
-Pin the action to a version tag or immutable commit SHA in maintained repositories.
+Pin the action to a version tag or immutable commit SHA in maintained repositories. Release automation uses npm Trusted Publishing; maintainers should review [the release setup](docs/release.md) before publishing a new version.
 
 ## Library API
 
@@ -177,7 +187,7 @@ process.stdout.write(renderReport(result, "json"));
 - **CI-native:** reports work in terminals, scripts, and SARIF consumers.
 - **Least privilege:** security rules favor explicit tools, paths, and network requirements.
 
-Read [the architecture](docs/architecture.md), [cross-agent compatibility profiles](docs/compatibility.md), [SkillConform 2.0 direction](docs/skillconform-2.md), [roadmap](ROADMAP.md), and [contribution guide](CONTRIBUTING.md) before proposing larger changes.
+Read [the architecture](docs/architecture.md), [baseline-to-candidate regression](docs/regression.md), [cross-agent compatibility profiles](docs/compatibility.md), [SkillConform 2.0 direction](docs/skillconform-2.md), [roadmap](ROADMAP.md), and [contribution guide](CONTRIBUTING.md) before proposing larger changes.
 
 ## Community and launch resources
 

ROADMAP.md

@@ -23,11 +23,12 @@ SkillConform is evolving toward an open-source CI and test suite for Agent Skill
 
 ## v0.4 — Behavioral regression
 
-- Versioned evaluation manifest.
-- Baseline-versus-candidate skill comparisons.
-- Deterministic assertions before optional model-based judges.
-- Reproducible agent CLI adapters with explicit sandbox and permission contracts.
-- Machine-readable evaluation evidence suitable for pull requests.
+- [x] `skillconform.regression/v1` baseline-versus-candidate evidence contract.
+- [x] Deterministic PR gate for new findings and capability expansion.
+- [x] `regress` CLI, TypeScript API, GitHub Action mode, and side-by-side checkout guidance.
+- [ ] Versioned live-evaluation manifest for task-level assertions.
+- [ ] Reproducible agent CLI adapters with explicit sandbox and permission contracts.
+- [ ] Optional runtime evidence that remains separate from deterministic static results.
 
 ## v0.5 — Supply chain and ecosystem
 

action.yml

@@ -1,5 +1,5 @@
 name: SkillConform
-description: Validate, inventory, compare host compatibility, and security-audit Agent Skills in GitHub Actions.
+description: Validate, inventory, compare compatibility, regression-gate, and security-audit Agent Skills in GitHub Actions.
 author: Ramzi Sultan
 branding:
   icon: check-circle
@@ -10,9 +10,13 @@ inputs:
     required: false
     default: .
   mode:
-    description: Mode: check, audit, inventory, or matrix.
+    description: Mode: check, audit, inventory, matrix, or regress.
     required: false
     default: audit
+  baseline:
+    description: Baseline path relative to the workspace. Required when mode is regress.
+    required: false
+    default: ""
   fail-on:
     description: Failure threshold, one of error, warning, note, or none.
     required: false
@@ -41,19 +45,27 @@ runs:
       env:
         SKILLCONFORM_TARGET: ${{ inputs.path }}
         SKILLCONFORM_MODE: ${{ inputs.mode }}
+        SKILLCONFORM_BASELINE: ${{ inputs.baseline }}
         SKILLCONFORM_FAIL_ON: ${{ inputs.fail-on }}
         SKILLCONFORM_FORMAT: ${{ inputs.format }}
         SKILLCONFORM_OUTPUT: ${{ inputs.output }}
       run: |
-        case "$SKILLCONFORM_MODE" in check|audit|inventory|matrix) ;; *) echo "mode must be check, audit, inventory, or matrix" >&2; exit 2 ;; esac
+        case "$SKILLCONFORM_MODE" in check|audit|inventory|matrix|regress) ;; *) echo "mode must be check, audit, inventory, matrix, or regress" >&2; exit 2 ;; esac
         case "$SKILLCONFORM_FORMAT" in pretty|json|sarif) ;; *) echo "format must be pretty, json, or sarif" >&2; exit 2 ;; esac
-        if [[ ( "$SKILLCONFORM_MODE" == "inventory" || "$SKILLCONFORM_MODE" == "matrix" ) && "$SKILLCONFORM_FORMAT" == "sarif" ]]; then
+        if [[ ( "$SKILLCONFORM_MODE" == "inventory" || "$SKILLCONFORM_MODE" == "matrix" || "$SKILLCONFORM_MODE" == "regress" ) && "$SKILLCONFORM_FORMAT" == "sarif" ]]; then
           echo "$SKILLCONFORM_MODE mode supports pretty or json format" >&2
           exit 2
         fi
         case "$SKILLCONFORM_FAIL_ON" in error|warning|note|none) ;; *) echo "fail-on has an invalid value" >&2; exit 2 ;; esac
 
         args=("$SKILLCONFORM_MODE" "$GITHUB_WORKSPACE/$SKILLCONFORM_TARGET" --format "$SKILLCONFORM_FORMAT" --fail-on "$SKILLCONFORM_FAIL_ON")
+        if [[ "$SKILLCONFORM_MODE" == "regress" ]]; then
+          if [[ -z "$SKILLCONFORM_BASELINE" ]]; then
+            echo "baseline input is required for regress mode" >&2
+            exit 2
+          fi
+          args+=(--baseline "$GITHUB_WORKSPACE/$SKILLCONFORM_BASELINE")
+        fi
         if [[ -n "$SKILLCONFORM_OUTPUT" ]]; then
           args+=(--output "$GITHUB_WORKSPACE/$SKILLCONFORM_OUTPUT")
         fi

docs/regression.md

@@ -0,0 +1,76 @@
+# Baseline-to-candidate regression gate
+
+SkillConform can compare a candidate skill tree against a known baseline without executing target scripts.
+
+```bash
+npx skillconform regress ./candidate/skills --baseline ./baseline/skills
+npx skillconform regress ./candidate/skills --baseline ./baseline/skills --fail-on warning
+npx skillconform regress ./candidate/skills --baseline ./baseline/skills --format json -o regression.json
+```
+
+The output contract is `skillconform.regression/v1`.
+
+## What the deterministic gate detects
+
+The first version reports:
+
+- newly introduced conformance or security findings;
+- new network capability;
+- new shell usage;
+- newly bundled executable/source scripts;
+- expansion of `allowed-tools`;
+- newly observed network hosts;
+- added or removed skills;
+- the count of findings resolved by the candidate.
+
+A capability expansion is evidence for review, not proof of malicious behavior. Default `--fail-on error` blocks only new error-level regressions. Teams that want a stricter policy can use `--fail-on warning`.
+
+## Pull-request workflow
+
+A safe workflow can check out the base and head revisions side-by-side and run SkillConform without executing code from either tree:
+
+```yaml
+name: Skill regression
+
+on:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  skill-regression:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout baseline
+        uses: actions/checkout@v7
+        with:
+          ref: ${{ github.event.pull_request.base.sha }}
+          path: baseline
+          persist-credentials: false
+
+      - name: Checkout candidate
+        uses: actions/checkout@v7
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+          path: candidate
+          persist-credentials: false
+
+      - name: Compare skills
+        uses: GenRamzi/skillconform@v0.4.0
+        with:
+          mode: regress
+          baseline: baseline/skills
+          path: candidate/skills
+          fail-on: warning
+```
+
+For fork pull requests, keep the workflow on the unprivileged `pull_request` event and do not execute scripts from the candidate tree. SkillConform's static regression gate reads files only.
+
+## Why this is separate from live evaluation
+
+Static regression answers: **what changed in the skill's trust and portability surface?**
+
+Live evaluation answers: **did an agent perform the task better or worse with this skill?**
+
+Keeping these evidence classes separate makes pull-request decisions easier to audit. Future evaluation adapters can attach runtime evidence without weakening the deterministic gate.

docs/release.md

@@ -0,0 +1,47 @@
+# Release and npm trusted publishing
+
+SkillConform publishes from GitHub Actions using npm Trusted Publishing (OIDC). The release workflow does not require a long-lived npm publish token.
+
+## One-time npm configuration
+
+The npm package owner must configure a Trusted Publisher for the existing `skillconform` package:
+
+1. Open the `skillconform` package on npmjs.com.
+2. Open **Settings → Trusted publishing**.
+3. Add **GitHub Actions** as a trusted publisher.
+4. Configure:
+   - GitHub organization/user: `GenRamzi`
+   - Repository: `skillconform`
+   - Workflow filename: `release.yml`
+   - Allowed action: enable direct `npm publish`
+5. Save the trusted publisher configuration.
+
+The workflow file is `.github/workflows/release.yml`, but npm expects only the filename `release.yml`.
+
+## Workflow security contract
+
+The workflow:
+
+- runs on a GitHub-hosted runner;
+- grants `id-token: write` for OIDC;
+- uses Node.js 24 and npm 11.15+;
+- disables package-manager caching in the release job;
+- validates TypeScript, tests, self-policy tests, self-audit, and package contents before publishing;
+- calls `npm publish` without `NODE_AUTH_TOKEN`;
+- relies on npm to create provenance automatically for trusted publishing;
+- creates the GitHub release only after npm publication succeeds or the version already exists.
+
+## Why token publishing was removed
+
+The previous release workflow used the repository secret `NPM_TOKEN`. It successfully published v0.1.1, but later v0.2.0 and v0.3.0 publish attempts were rejected by npm after all project validation passed. Trusted publishing removes the long-lived write credential and binds publishing to this repository and workflow.
+
+## Recovery
+
+If a release fails with an authentication error:
+
+- verify the npm Trusted Publisher values match `GenRamzi/skillconform` and `release.yml` exactly;
+- verify direct `npm publish` is allowed for that trusted publisher;
+- keep `id-token: write` enabled;
+- do not add a write-capable `NODE_AUTH_TOKEN` back as a workaround.
+
+Once the trusted publisher is configured, re-running the failed release or merging the next version bump will perform a fresh OIDC publish.

package.json

@@ -1,7 +1,7 @@
 {
   "name": "skillconform",
-  "version": "0.3.0",
-  "description": "CI-native conformance, capability inventory, cross-agent compatibility, security review, and regression testing for Agent Skills.",
+  "version": "0.4.0",
+  "description": "CI-native conformance, capability inventory, cross-agent compatibility, security review, and baseline-to-candidate regression gates for Agent Skills.",
   "type": "module",
   "bin": {
     "skillconform": "dist/src/cli.js"
@@ -36,6 +36,8 @@
     "codex",
     "gemini-cli",
     "cross-agent",
+    "regression-testing",
+    "github-actions",
     "security",
     "sarif"
   ],