Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 16 additions & 12 deletions .github/workflows/skillgate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,17 @@ jobs:
- uses: actions/checkout@v7
- uses: actions/setup-python@v6
with:
python-version: "3.11"
python-version-file: ".python-version"
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
enable-cache: true
- name: Install SkillGate
run: python -m pip install -e ".[dev]"
run: uv sync --locked --group dev
- name: Check golden snapshots
id: snapshots
continue-on-error: true
run: python tools/update_snapshots.py --check --artifacts test-outputs/snapshots
run: uv run python tools/update_snapshots.py --check --artifacts test-outputs/snapshots
- name: Upload snapshot review artifacts
if: steps.snapshots.outcome == 'failure'
uses: actions/upload-artifact@v7
Expand All @@ -35,24 +39,24 @@ jobs:
if: steps.snapshots.outcome == 'failure'
run: exit 1
- name: Run tests
run: python -m pytest
run: uv run pytest
- name: Run lint
run: python -m ruff check .
run: uv run ruff check .
- name: Check formatting
run: python -m ruff format --check .
run: uv run ruff format --check .
- name: Scan repository
run: skillgate scan . --format text
run: uv run skillgate scan . --format text
- name: Run deterministic demo sessions
run: |
skillgate demo skill --output test-outputs/reviewable-demo --validate --scan
skillgate demo mcpb --output test-outputs/reviewable-node.mcpb --scan
uv run skillgate demo skill --output test-outputs/reviewable-demo --validate --scan
uv run skillgate demo mcpb --output test-outputs/reviewable-node.mcpb --scan
- name: Run policy smoke check
run: skillgate check fixtures/benchmark/01-safe-documentation-skill --policy skillgate.example.yaml
run: uv run skillgate check fixtures/benchmark/01-safe-documentation-skill --policy skillgate.example.yaml
- name: Verify benchmark fixtures
run: skillgate fixtures summary fixtures/benchmark --format text
run: uv run skillgate fixtures summary fixtures/benchmark --format text
- name: Generate SARIF
if: always()
run: skillgate scan . --format sarif --output skillgate.sarif
run: uv run skillgate scan . --format sarif --output skillgate.sarif
- name: Upload SARIF review artifact
if: always()
uses: actions/upload-artifact@v7
Expand Down
1 change: 1 addition & 0 deletions .python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.12
17 changes: 17 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,3 +101,20 @@ python tools/update_snapshots.py --accept
## Documentation

Documentation should be clear about the threat model. SkillGate detects static risks and capability drift; it does not prove that an agent skill or MCP server is safe.

# Development setup

SkillGate uses Python 3.12 for its reproducible development environment. Install
[`uv`](https://docs.astral.sh/uv/) and run:

```bash
uv sync --locked --group dev
uv run pytest
uv run ruff check .
uv run ruff format --check .
npm test
```

Use `uv run` for repository tools so local execution matches CI. If dependency
metadata changes, regenerate `uv.lock` with `uv lock`, review the diff, and
verify it with `uv sync --locked` before opening a pull request.
48 changes: 37 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,30 @@ skillgate check . --policy skillgate.yaml
```

That is the primary workflow. Baselines, provenance, inventory reports, registry
comparison, SARIF upload, and advanced GitHub Action settings are available
after the first useful scan.
comparison, optional SARIF export, and optional GitHub Action settings are
available after the first useful scan.

## Connection Boundaries

The default path is local and upload-free:

- `scan`, `skills validate`, `review preinstall` with a local path, and `mcpb
scan` read local files and write results to the terminal or local output files.
- These local paths do not call GitHub, upload reports, install packages, start
servers, or execute scanned content.

Connections are explicit opt-ins:

- `github scan` and `review preinstall` with a GitHub URL make bounded requests
to GitHub to fetch the requested source for static review.
- GitHub Actions, SARIF uploads, Actions artifacts, and Code Scanning are
optional integrations used only when a repository owner enables a workflow.

Nothing in a local SkillGate invocation uploads findings automatically.

For a copyable first repository, use the
[pre-install starter](examples/preinstall-starter/README.md) and its
[review-only Action workflow](docs/starter-repository.md).

For concrete examples of how to interpret output, see the
[public scan reports](docs/public-scan-reports/README.md).
Expand All @@ -96,7 +118,7 @@ For a guided walkthrough with a deterministic local input, see the
| Inspect an MCP bundle before installing | [`skillgate mcpb scan bundle.mcpb`](#5-scan-an-mcp-bundle-before-installing) | Static startup, file, endpoint, secret-reference, binary, and nested-archive review |
| Inspect MCP registry metadata without installing | [`skillgate mcp registry scan`](#6-review-mcp-registry-metadata) | MCP tool, transport, package, and registry drift checks |
| Build a review inventory | [`skillgate inventory .`](#7-build-a-capability-inventory) | Trust-boundary summary and filterable JSON |
| Upload findings to GitHub code scanning | [`--format sarif`](#8-export-sarif-for-github-code-scanning) | Stable SARIF alerts with capability tags |
| Optionally export SARIF for GitHub code scanning | [`--format sarif`](#8-optionally-export-sarif-for-github-code-scanning) | Local SARIF output; upload is a separate opt-in |
| Follow a guided review session | [Review sessions](docs/sessions/README.md) | Copy-pasteable pre-install, pre-merge, and approval workflows |

## Install
Expand Down Expand Up @@ -135,7 +157,8 @@ Teams that require maximum reproducibility should pin the full commit SHA in
install commands and GitHub Action references.

GitHub installs require Python 3.11 or newer and `git` on the customer machine.
After PyPI publication, the preferred low-friction install path will be:
PyPI publication is deferred for `0.1.3`. When it is intentionally revisited,
the planned distribution name is `openevalgate-skillgate`:

```bash
pipx install openevalgate-skillgate
Expand All @@ -162,9 +185,9 @@ is marked private until then. Details: [GitHub-first Node wrapper](docs/node-wra
For contributor or source-checkout development:

```bash
python -m pip install -e .
skillgate --version
skillgate --help
uv sync --locked --group dev
uv run skillgate --version
uv run skillgate --help
```

## 1. Scan A Local Repository
Expand Down Expand Up @@ -346,9 +369,10 @@ skillgate inventory . --source-file "scripts/*"

Inventory output groups detected capabilities and findings by source file and summarizes trust boundaries for local execution, remote endpoints, secrets, generated files, MCP servers, prompt controls, and obfuscation.

## 8. Export SARIF For GitHub Code Scanning
## 8. Optionally Export SARIF For GitHub Code Scanning

Any scan that supports SARIF can be uploaded to GitHub code scanning:
Any scan that supports SARIF can write a local SARIF file. This command does not
upload anything:

```bash
skillgate scan . --format sarif --output skillgate.sarif
Expand All @@ -360,7 +384,9 @@ skillgate mcp registry compare . --server io.example.server --format sarif --out

Plain `scan` SARIF reports static findings. Policy-aware `check` SARIF includes policy waiver and suppression metadata while `--dry-run` keeps SARIF generation from becoming a second blocking step. SARIF output includes deterministic alert fingerprints, stable run categories, capability tags, severity tags, and capability taxa. Local scans use `skillgate/local-repository`, remote GitHub scans use `skillgate/remote-github`, MCPB scans use `skillgate/mcp-bundle`, and MCP registry comparisons use `skillgate/mcp-registry-compare`.

The repository includes `.github/workflows/skillgate.yml` as a complete example workflow and a composite action for CI adoption:
If you explicitly choose GitHub CI, the repository includes
`.github/workflows/skillgate.yml` and a composite Action example. Those workflows
are separate from local execution and may upload artifacts or SARIF to GitHub:

```yaml
steps:
Expand All @@ -376,7 +402,7 @@ steps:
json-output: skillgate-review.json
```

Add `policy: skillgate.yaml` when the repository has a policy file and should block unapproved behavior. Without `policy`, the Action runs a nonblocking scan. When both `policy` and `sarif-output` are supplied, the Action uploads policy-aware SARIF with waiver suppressions.
Add `policy: skillgate.yaml` when the repository has a policy file and should block unapproved behavior. Without `policy`, the optional Action runs a nonblocking scan. When both `policy` and `sarif-output` are supplied, that GitHub workflow can upload policy-aware SARIF with waiver suppressions.

For pull-request review, add `step-summary: "true"` to publish a Markdown summary in the GitHub job page. Add `json-output: skillgate-review.json` when you also want a downloadable machine-readable review artifact with introduced capabilities, removed capabilities, changed trust boundaries, high-risk findings, policy violations, active waivers, and artifact locations.

Expand Down
71 changes: 71 additions & 0 deletions docs/benchmark/0.1.3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# SkillGate Benchmark Report

- Scanner version: `0.1.2`
- Fixture root: `fixtures/benchmark`
- Fixtures: 27
- Passed: 27
- Failed: 0

## Rule Coverage

| Rule | Expected fixtures | Actual fixtures | Coverage status |
| --- | ---: | ---: | --- |
| SG001 | 7 | 7 | covered |
| SG002 | 2 | 2 | covered |
| SG003 | 19 | 19 | covered |
| SG004 | 2 | 2 | covered |
| SG005 | 6 | 6 | covered |
| SG006 | 5 | 5 | covered |
| SG007 | 3 | 3 | covered |
| SG008 | 1 | 1 | covered |
| SG009 | 9 | 9 | covered |
| SG010 | 1 | 1 | covered |
| SG011 | 2 | 2 | covered |
| SG012 | 2 | 2 | covered |

## Fixture Results

| Fixture | Status | Expected rules | Actual rules | Attribution |
| --- | --- | --- | --- | --- |
| `01-safe-documentation-skill` | **pass** | `none` | `none` | no |
| `02-shell-execution` | **pass** | `SG001` | `SG001` | no |
| `03-destructive-command` | **pass** | `SG001, SG002` | `SG001, SG002` | no |
| `04-network-egress` | **pass** | `SG003` | `SG003` | no |
| `05-remote-download-execute` | **pass** | `SG001, SG003, SG004` | `SG001, SG003, SG004` | no |
| `06-secret-access` | **pass** | `SG005` | `SG005` | no |
| `07-filesystem-write` | **pass** | `SG006` | `SG006` | no |
| `08-prompt-override` | **pass** | `SG007` | `SG007` | no |
| `09-unicode-obfuscation` | **pass** | `SG008` | `SG008` | no |
| `10-mcp-config` | **pass** | `SG003, SG005, SG009` | `SG003, SG005, SG009` | no |
| `11-mcp-capability-drift-before` | **pass** | `SG003, SG005, SG009` | `SG003, SG005, SG009` | no |
| `12-mcp-capability-drift-after` | **pass** | `SG001, SG003, SG005, SG009, SG010` | `SG001, SG003, SG005, SG009, SG010` | no |
| `13-public-pattern-python-node-extraction` | **pass** | `SG003, SG006` | `SG003, SG006` | yes |
| `14-public-pattern-shell-powershell-extraction` | **pass** | `SG001, SG002, SG003, SG006` | `SG001, SG002, SG003, SG006` | yes |
| `15-public-pattern-mcp-remote-config` | **pass** | `SG003, SG005, SG009` | `SG003, SG005, SG009` | yes |
| `16-public-pattern-mcp-http-remote` | **pass** | `SG003, SG009` | `SG003, SG009` | yes |
| `17-public-pattern-agent-skill-plugin` | **pass** | `SG003, SG009` | `SG003, SG009` | yes |
| `18-public-pattern-mcp-nested-profile` | **pass** | `SG003, SG005, SG009` | `SG003, SG005, SG009` | yes |
| `19-public-pattern-plugin-hooks` | **pass** | `SG001, SG003, SG004, SG006` | `SG001, SG003, SG004, SG006` | yes |
| `20-public-pattern-marketplace-mcp-package` | **pass** | `SG003, SG009` | `SG003, SG009` | yes |
| `21-public-pattern-agent-command-pack` | **pass** | `SG003, SG006` | `SG003, SG006` | yes |
| `22-public-pattern-mcp-local-bridge` | **pass** | `SG003, SG009` | `SG003, SG009` | yes |
| `23-public-pattern-skill-tool-metadata` | **pass** | `SG007` | `SG007` | yes |
| `24-public-pattern-mcp-tool-metadata-risk` | **pass** | `SG003, SG007, SG011` | `SG003, SG007, SG011` | yes |
| `25-public-pattern-mcp-app-web-surface` | **pass** | `SG003, SG011` | `SG003, SG011` | yes |
| `26-public-pattern-mcp-dangerous-transport` | **pass** | `SG001, SG003, SG012` | `SG001, SG003, SG012` | yes |
| `27-public-pattern-mcp-registry-package-metadata` | **pass** | `SG003, SG012` | `SG003, SG012` | yes |

## Reproduce

```bash
skillgate fixtures summary fixtures/benchmark --format markdown
```

## Attribution

Public-pattern fixtures retain source URLs, retrieval dates, and reduction notes in their expected-findings metadata.

## Limitations

This report verifies deterministic fixture expectations and rule coverage. It is not a real-world detection accuracy benchmark, malware verdict, or completeness claim.
Fixtures are intentionally reduced examples and may not represent the behavior, context, or security posture of their source projects.
33 changes: 26 additions & 7 deletions docs/examples/github-action-minimal.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,22 @@

These examples use the stable `charliechenye/SkillGate@v0` Action tag. Teams
that require immutable Action references should pin a full commit SHA instead.
SkillGate generates SARIF when `sarif-output` is supplied; GitHub's upload action
uploads that SARIF file to code scanning.
SkillGate generates SARIF when `sarif-output` is supplied.

For pull requests, retain SARIF as an artifact so intentional demo and test
findings remain reviewable without creating a blocking Code Scanning status.
Publish SARIF to Code Scanning on protected branches or manual runs, or use the
blocking policy example below when the repository is ready to enforce reviewed
behavior.

The copyable [pre-install starter repository](../../examples/preinstall-starter/)
contains this review-only workflow with Markdown, JSON, and SARIF artifacts.
On protected branches and manual runs, GitHub's upload action publishes the
SARIF artifact to Code Scanning and uploads that SARIF file for durable review.

These Action examples are optional GitHub integrations. They do not change the
local-only behavior of SkillGate commands, which write reports locally and do
not upload findings.

## Nonblocking Scan With SARIF

Expand Down Expand Up @@ -41,7 +55,7 @@ jobs:
json-output: skillgate-review.json

- uses: github/codeql-action/upload-sarif@v4
if: always()
if: github.event_name != 'pull_request' && always()
with:
sarif_file: skillgate.sarif

Expand All @@ -52,6 +66,7 @@ jobs:
path: |
skillgate-summary.md
skillgate-review.json
skillgate.sarif
```

## Blocking Policy Check With Policy-Aware SARIF
Expand Down Expand Up @@ -92,7 +107,7 @@ jobs:
json-output: skillgate-review.json

- uses: github/codeql-action/upload-sarif@v4
if: always()
if: github.event_name != 'pull_request' && always()
with:
sarif_file: skillgate.sarif

Expand All @@ -103,6 +118,7 @@ jobs:
path: |
skillgate-summary.md
skillgate-review.json
skillgate.sarif
```

## Baseline Drift Blocking
Expand Down Expand Up @@ -142,7 +158,7 @@ jobs:
fail-on-drift: "true"

- uses: github/codeql-action/upload-sarif@v4
if: always()
if: github.event_name != 'pull_request' && always()
with:
sarif_file: skillgate.sarif

Expand All @@ -153,6 +169,7 @@ jobs:
path: |
skillgate-summary.md
skillgate-review.json
skillgate.sarif
```

## Repository Plus Committed MCPB Bundle
Expand Down Expand Up @@ -195,12 +212,12 @@ jobs:
json-output: skillgate-review.json

- uses: github/codeql-action/upload-sarif@v4
if: always()
if: github.event_name != 'pull_request' && always()
with:
sarif_file: skillgate.sarif

- uses: github/codeql-action/upload-sarif@v4
if: always()
if: github.event_name != 'pull_request' && always()
with:
sarif_file: skillgate-mcpb.sarif

Expand All @@ -211,4 +228,6 @@ jobs:
path: |
skillgate-summary.md
skillgate-review.json
skillgate.sarif
skillgate-mcpb.sarif
```
21 changes: 21 additions & 0 deletions docs/starter-repository.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Pre-install starter repository

Copy [`examples/preinstall-starter/`](../examples/preinstall-starter/) when a
new Agent Skill repository needs a safe first review. It demonstrates the full
adoption path without requiring a real third-party package or executing any
repository content.

The local review command is the default path. It reads the checked-out files and
writes reports locally; it does not contact GitHub or upload findings.

The GitHub workflow is a separate, optional integration with two deliberate
modes:

- Pull requests retain Markdown, JSON, and SARIF as artifacts. Findings remain
reviewable but do not create a blocking Code Scanning status.
- Pushes to `main` and manual runs publish SARIF to Code Scanning for durable
visibility.

After reviewers approve the observed capability surface, add a reviewed policy
to the Action or adopt a baseline with `fail-on-drift: "true"`. Keep that
enforcement decision separate from the advisory pre-install packet.
Loading
Loading