From 6151561809c83605a1a32ae7c4991429466b83c6 Mon Sep 17 00:00:00 2001 From: devkey Date: Sun, 21 Jun 2026 17:18:09 +0900 Subject: [PATCH] docs: remove internal superpowers notes --- .gitignore | 1 + ...6-21-skillspector-action-implementation.md | 376 ------------------ .../2026-06-21-skillspector-action-design.md | 216 ---------- 3 files changed, 1 insertion(+), 592 deletions(-) delete mode 100644 docs/superpowers/plans/2026-06-21-skillspector-action-implementation.md delete mode 100644 docs/superpowers/specs/2026-06-21-skillspector-action-design.md diff --git a/.gitignore b/.gitignore index 46cd947..fc6ffd1 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ __pycache__/ .pytest_cache/ skillspector-results/ skillspector-results-*/ +docs/superpowers/ diff --git a/docs/superpowers/plans/2026-06-21-skillspector-action-implementation.md b/docs/superpowers/plans/2026-06-21-skillspector-action-implementation.md deleted file mode 100644 index 1e698f0..0000000 --- a/docs/superpowers/plans/2026-06-21-skillspector-action-implementation.md +++ /dev/null @@ -1,376 +0,0 @@ -# SkillSpector Action Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build a distributable GitHub Action that runs SkillSpector from a public GHCR image, discovers skill targets, produces JSON/SARIF/Markdown reports, optionally uploads SARIF, and fails only when configured. - -**Architecture:** A composite `action.yml` invokes a runner-side shell script that computes the scanner image and runs it with `docker run`. The image contains a Python wrapper package that performs discovery, changed-only diffing, SkillSpector invocation, suppression, report merging, summary generation, and failure policy. The wrapper normalizes SARIF into a single run and the publish workflow publishes branch, semver, major, and `sha-` image tags. - -**Tech Stack:** GitHub composite actions, Docker/GHCR, Python 3.12, pytest, shell scripts, GitHub CodeQL `upload-sarif@v4`, `actions/upload-artifact@v4`, Docker Buildx. - ---- - -## Review Constraints - -The implementation must address the adversarial review findings before it can be considered complete: - -- Avoid a hardcoded mutable runtime image in `action.yml`. Use `docker run` from a shell script so the default image can be derived from `github.action_ref`. -- Publish and consume `sha-` image tags when the action is pinned by full commit SHA. -- Merge per-target SARIF output into one SARIF run to avoid GitHub Code Scanning multi-run/category upload failures. -- For `changed-only`, fetch the PR base SHA when needed and fall back to full discovery rather than skipping scans when the diff cannot be resolved. -- Do not pass LLM-related environment variables into the container unless `llm` is explicitly true. - -## File Structure - -- Create `action.yml`: public action inputs/outputs and composite steps for running the container, uploading artifacts, and uploading SARIF. -- Create `scripts/run-action.sh`: runner-side image resolution, docker invocation, and output forwarding. -- Create `scripts/entrypoint.sh`: container entrypoint that invokes the Python wrapper. -- Create `Dockerfile`: Python 3.12 runtime with SkillSpector installed from pinned commit `a5092dd9b9521ff57a9b53612bb129ce78019002`. -- Create `pyproject.toml`: local package and pytest config for the wrapper. -- Create `src/skillspector_action/__init__.py`: package marker and version. -- Create `src/skillspector_action/__main__.py`: CLI entrypoint. -- Create `src/skillspector_action/config.py`: typed config parsing. -- Create `src/skillspector_action/discovery.py`: skill target discovery and changed-file mapping. -- Create `src/skillspector_action/baseline.py`: baseline loading and finding suppression keys. -- Create `src/skillspector_action/reports.py`: JSON/SARIF normalization and Markdown summary generation. -- Create `src/skillspector_action/runner.py`: SkillSpector subprocess orchestration and failure decision. -- Create tests under `tests/` for discovery, baseline, reports, policy, and CLI smoke behavior. -- Create `.github/workflows/ci.yml`: run unit tests and action smoke checks. -- Create `.github/workflows/publish-image.yml`: publish the GHCR image on pushes and tags. -- Replace `README.md`: user-facing examples, inputs, outputs, security notes, billing/availability notes, and release process. - -### Task 1: Project Skeleton and Discovery - -**Files:** -- Create: `pyproject.toml` -- Create: `src/skillspector_action/__init__.py` -- Create: `src/skillspector_action/discovery.py` -- Test: `tests/test_discovery.py` - -- [ ] **Step 1: Write failing discovery tests** - -```python -from pathlib import Path - -from skillspector_action.discovery import discover_skill_targets, targets_for_changed_files - - -def test_discovers_skill_directories(tmp_path: Path) -> None: - (tmp_path / "skills" / "alpha").mkdir(parents=True) - (tmp_path / "skills" / "alpha" / "SKILL.md").write_text("# Alpha\n", encoding="utf-8") - (tmp_path / "skills" / "beta").mkdir(parents=True) - (tmp_path / "skills" / "beta" / "SKILL.md").write_text("# Beta\n", encoding="utf-8") - - assert discover_skill_targets(tmp_path) == [ - tmp_path / "skills" / "alpha", - tmp_path / "skills" / "beta", - ] - - -def test_single_skill_file_is_a_target(tmp_path: Path) -> None: - skill_file = tmp_path / "SKILL.md" - skill_file.write_text("# Root\n", encoding="utf-8") - - assert discover_skill_targets(skill_file) == [skill_file] - - -def test_changed_files_map_to_owning_skill_directory(tmp_path: Path) -> None: - skill_dir = tmp_path / "skills" / "alpha" - skill_dir.mkdir(parents=True) - (skill_dir / "SKILL.md").write_text("# Alpha\n", encoding="utf-8") - (skill_dir / "notes.md").write_text("notes\n", encoding="utf-8") - - assert targets_for_changed_files(tmp_path, ["skills/alpha/notes.md"]) == [skill_dir] -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `python -m pytest tests/test_discovery.py -q` - -Expected: import failure for `skillspector_action.discovery`. - -- [ ] **Step 3: Implement minimal discovery** - -Create `pyproject.toml` with pytest path setup, package metadata, and Python 3.12 target. Implement `discover_skill_targets(root, excludes=())` and `targets_for_changed_files(root, changed_files, excludes=())` using `Path.rglob("SKILL.md")`, stable sorting, and parent walking to find the nearest owning `SKILL.md`. - -- [ ] **Step 4: Run discovery tests** - -Run: `python -m pytest tests/test_discovery.py -q` - -Expected: 3 passed. - -### Task 2: Excludes and Changed-Only Diff - -**Files:** -- Modify: `src/skillspector_action/discovery.py` -- Create: `src/skillspector_action/gitdiff.py` -- Test: `tests/test_discovery.py` -- Test: `tests/test_gitdiff.py` - -- [ ] **Step 1: Write failing exclude and diff tests** - -Add tests proving comma/newline exclude patterns skip matching targets and that `changed_files_from_git()` returns changed paths from a temporary git repo. - -- [ ] **Step 2: Run tests to verify failure** - -Run: `python -m pytest tests/test_discovery.py tests/test_gitdiff.py -q` - -Expected: missing `gitdiff` module or exclude behavior failure. - -- [ ] **Step 3: Implement excludes and git diff helper** - -Use `fnmatch.fnmatch` against both POSIX relative target paths and file paths. Implement `changed_files_from_git(repo, base_ref, head_ref="HEAD")` with `git diff --name-only --diff-filter=ACMR`. - -- [ ] **Step 4: Run tests** - -Run: `python -m pytest tests/test_discovery.py tests/test_gitdiff.py -q` - -Expected: all tests pass. - -### Task 3: Baseline Suppression - -**Files:** -- Create: `src/skillspector_action/baseline.py` -- Test: `tests/test_baseline.py` - -- [ ] **Step 1: Write failing baseline tests** - -Test that a baseline entry matching `rule_id`, normalized path, and message fingerprint suppresses a finding, and non-matching findings remain active. - -- [ ] **Step 2: Run test to verify failure** - -Run: `python -m pytest tests/test_baseline.py -q` - -Expected: import failure for `skillspector_action.baseline`. - -- [ ] **Step 3: Implement baseline matching** - -Implement baseline JSON shape: - -```json -{ - "version": 1, - "findings": [ - { - "rule_id": "SS001", - "path": "skills/alpha/SKILL.md", - "message_hash": "sha256-prefix" - } - ] -} -``` - -Normalize paths to POSIX relative paths and hash normalized messages with SHA-256. - -- [ ] **Step 4: Run baseline tests** - -Run: `python -m pytest tests/test_baseline.py -q` - -Expected: all tests pass. - -### Task 4: Report Merging and Summary - -**Files:** -- Create: `src/skillspector_action/reports.py` -- Test: `tests/test_reports.py` - -- [ ] **Step 1: Write failing report tests** - -Test that multiple SkillSpector-like JSON reports are combined, highest severity and score are computed, and multiple SARIF inputs become one SARIF `run` with deduplicated rules and merged results. - -- [ ] **Step 2: Run test to verify failure** - -Run: `python -m pytest tests/test_reports.py -q` - -Expected: import failure for `skillspector_action.reports`. - -- [ ] **Step 3: Implement report utilities** - -Implement: - -- `merge_json_reports(reports, suppressed_findings)` -- `merge_sarif_reports(reports)` -- `write_markdown_summary(summary)` -- severity ordering `none < low < medium < high < critical` - -The SARIF merger must emit SARIF version `2.1.0`, one run, one `tool.driver`, deduplicated `rules`, and all results. - -- [ ] **Step 4: Run report tests** - -Run: `python -m pytest tests/test_reports.py -q` - -Expected: all tests pass. - -### Task 5: Failure Policy and Wrapper CLI - -**Files:** -- Create: `src/skillspector_action/config.py` -- Create: `src/skillspector_action/runner.py` -- Create: `src/skillspector_action/__main__.py` -- Test: `tests/test_policy.py` -- Test: `tests/test_cli.py` - -- [ ] **Step 1: Write failing policy and CLI tests** - -Test `fail-on: none`, `high`, `critical`, `min-score`, empty target success, and CLI output files using a fake scanner callable. - -- [ ] **Step 2: Run test to verify failure** - -Run: `python -m pytest tests/test_policy.py tests/test_cli.py -q` - -Expected: missing modules or missing CLI behavior. - -- [ ] **Step 3: Implement config, policy, and CLI orchestration** - -Parse environment variables from the action: - -- `INPUT_PATH` -- `INPUT_CHANGED_ONLY` -- `INPUT_FAIL_ON` -- `INPUT_MIN_SCORE` -- `INPUT_EXCLUDE` -- `INPUT_BASELINE` -- `INPUT_LLM` -- `INPUT_OUTPUT_DIR` - -Run SkillSpector through `subprocess.run`, write per-target JSON/SARIF under a temporary directory, apply suppressions, merge final reports, write `$GITHUB_OUTPUT` values when present, append Markdown to `$GITHUB_STEP_SUMMARY` when present, and exit non-zero only for configured policy failures or scanner execution errors. - -- [ ] **Step 4: Run policy and CLI tests** - -Run: `python -m pytest tests/test_policy.py tests/test_cli.py -q` - -Expected: all tests pass. - -### Task 6: Action Metadata and Shell Entrypoints - -**Files:** -- Create: `action.yml` -- Create: `scripts/run-action.sh` -- Create: `scripts/entrypoint.sh` -- Test: `tests/test_action_metadata.py` - -- [ ] **Step 1: Write failing metadata tests** - -Test that `action.yml` declares the planned inputs/outputs, uses composite, includes artifact upload, includes SARIF upload with `github/codeql-action/upload-sarif@v4`, and does not hardcode `docker://ghcr.io/npjigak/skillspector-action:v1`. - -- [ ] **Step 2: Run test to verify failure** - -Run: `python -m pytest tests/test_action_metadata.py -q` - -Expected: `action.yml` missing. - -- [ ] **Step 3: Implement action and scripts** - -`scripts/run-action.sh` computes the default image: - -- if `INPUT_IMAGE` is set, use it; -- else if `ACTION_REF` is a 40-character hex SHA, use `ghcr.io/npjigak/skillspector-action:sha-${ACTION_REF}`; -- else use `ghcr.io/npjigak/skillspector-action:${ACTION_REF:-v1}`. - -It runs Docker with the workspace mounted read-write only for report output, passes action inputs, and passes LLM provider environment variables only when `INPUT_LLM=true`. - -`action.yml` uses a composite `run` step for the shell script, `actions/upload-artifact@v4`, and conditional `github/codeql-action/upload-sarif@v4`. - -- [ ] **Step 4: Run metadata tests** - -Run: `python -m pytest tests/test_action_metadata.py -q` - -Expected: tests pass. - -### Task 7: Dockerfile and CI - -**Files:** -- Create: `Dockerfile` -- Create: `.github/workflows/ci.yml` -- Create: `.github/workflows/publish-image.yml` -- Test: `tests/test_workflows.py` - -- [ ] **Step 1: Write failing workflow tests** - -Test that Dockerfile pins `SKILLSPECTOR_REF` to `a5092dd9b9521ff57a9b53612bb129ce78019002`, CI runs pytest, and publish workflow includes `packages: write`, GHCR login, build-push-action, semver tags, major tags, and `sha-{{sha}}` tags. - -- [ ] **Step 2: Run test to verify failure** - -Run: `python -m pytest tests/test_workflows.py -q` - -Expected: files missing. - -- [ ] **Step 3: Implement Dockerfile and workflows** - -Use `python:3.12-slim-bookworm`, install git, install this package, install SkillSpector from pinned Git commit, and set `/entrypoint.sh`. Configure CI for Python tests and a container build smoke test. Configure publish image on branch and tag pushes. - -- [ ] **Step 4: Run workflow tests** - -Run: `python -m pytest tests/test_workflows.py -q` - -Expected: tests pass. - -### Task 8: README and Smoke Verification - -**Files:** -- Modify: `README.md` -- Test: all tests - -- [ ] **Step 1: Write failing README tests** - -Add test assertions that README documents basic usage, permissions, inputs, outputs, `pull_request` guidance, `pull_request_target` warning, GHCR public/free note, and LLM opt-in warning. - -- [ ] **Step 2: Run README test to verify failure** - -Run: `python -m pytest tests/test_readme.py -q` - -Expected: README content missing. - -- [ ] **Step 3: Replace README** - -Write a complete README with: - -- copy-paste workflow; -- report-only default; -- CI failure examples; -- changed-only behavior; -- SARIF upload permissions; -- LLM opt-in and fork PR warnings; -- billing and GHCR notes; -- release and image publishing notes; -- baseline format. - -- [ ] **Step 4: Run full verification** - -Run: - -```bash -python -m pytest -q -docker build -t skillspector-action:test . -``` - -Expected: pytest passes and Docker image builds successfully. - -### Task 9: Final Self-Review and Completion Check - -**Files:** -- Review all modified files. - -- [ ] **Step 1: Run unified adversarial review** - -Review the final diff read-only for ship-blocking risks, focusing on GitHub Actions security, SARIF upload reliability, Docker runtime behavior, and changed-only correctness. - -- [ ] **Step 2: Digest findings** - -Accept only grounded material findings and fix them with TDD when they touch code. - -- [ ] **Step 3: Run final verification** - -Run: - -```bash -python -m pytest -q -docker build -t skillspector-action:test . -git diff --check -``` - -Expected: all commands succeed. - -- [ ] **Step 4: Report completion** - -Summarize changed files, verification evidence, and any remaining release steps such as tagging and pushing the GHCR image. diff --git a/docs/superpowers/specs/2026-06-21-skillspector-action-design.md b/docs/superpowers/specs/2026-06-21-skillspector-action-design.md deleted file mode 100644 index 3612f6a..0000000 --- a/docs/superpowers/specs/2026-06-21-skillspector-action-design.md +++ /dev/null @@ -1,216 +0,0 @@ -# SkillSpector Action Design - -## Goal - -Build a distributable GitHub Action for running NVIDIA SkillSpector in skill distribution repositories and CI pipelines with a normal `uses:` experience: - -```yaml -- uses: NPJigaK/skillspector-action@v1 - with: - path: . - changed-only: true - fail-on: critical - upload-sarif: true -``` - -The action should feel like a standard GitHub Action. Users should not need to understand GHCR, Docker image publishing, or SkillSpector installation details. - -## Positioning - -This is an unofficial, experimental integration around SkillSpector. The value is not only wrapping the CLI, but giving skill authors and registry maintainers a PR security-check workflow with safe defaults: - -- API-key-free scanning by default with SkillSpector `--no-llm`. -- Report-only default so CI does not fail unless explicitly configured. -- SARIF support for GitHub Code Scanning. -- Changed-skill scanning for pull requests. -- JSON, SARIF, Markdown summary, and artifacts for triage. -- Suppression hooks through excludes and baselines to reduce false positives. - -## Architecture - -Use a Composite Action backed by a published GHCR image. - -The repository will publish a public container image at: - -```text -ghcr.io/npjigak/skillspector-action: -``` - -The composite action hides that image from users. It prepares inputs, runs the containerized scanner, optionally uploads SARIF through `github/codeql-action/upload-sarif`, and exposes outputs. The GHCR image contains the pinned SkillSpector runtime and this action's scanner wrapper. - -This gives users a one-step `uses:` interface while keeping runtime dependencies controlled by this repository. - -## User Experience - -The minimal user workflow is: - -```yaml -name: SkillSpector - -on: - pull_request: - push: - branches: [main] - -permissions: - contents: read - security-events: write - -jobs: - scan: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: NPJigaK/skillspector-action@v1 - with: - path: . - changed-only: true - upload-sarif: true -``` - -Users only configure action inputs. They do not pull the GHCR image directly. - -## Inputs - -The MVP action supports: - -- `path`: scan root, default `.`. -- `changed-only`: when true on pull requests, scan only changed skill directories, default `false`. -- `fail-on`: one of `none`, `high`, `critical`, default `none`. -- `min-score`: optional numeric risk score threshold for failure, default unset. -- `upload-sarif`: whether to upload SARIF to GitHub Code Scanning, default `false`. -- `exclude`: newline or comma separated glob patterns to skip, default empty. -- `baseline`: optional path to a baseline JSON file for suppressing already accepted findings, default empty. -- `llm`: enable SkillSpector LLM analysis, default `false`. -- `artifact-name`: artifact name for generated reports, default `skillspector-results`. -- `image`: override scanner image for development, default to this repository's public GHCR image. - -## Outputs - -The action exposes: - -- `sarif`: path to the generated SARIF report. -- `json`: path to the generated JSON report. -- `markdown`: path to the Markdown summary. -- `risk-score`: highest risk score found across scanned skills. -- `risk-severity`: highest severity found across scanned skills. -- `findings-count`: total finding count after suppressions. -- `scanned-count`: number of skill targets scanned. - -## Scan Target Discovery - -The scanner wrapper discovers skill targets by finding `SKILL.md` files under `path`. - -When `changed-only` is false, it scans each discovered skill directory. If `path` itself is a single `SKILL.md`, it scans that file. - -When `changed-only` is true on a pull request, it compares the PR diff against the base ref and scans only skill directories whose files changed. If changed-only discovery finds no changed skill targets, the action exits successfully with an empty report and a clear summary. - -For non-PR events with `changed-only: true`, the action falls back to full discovery unless a base commit is available. - -## Report Generation - -For each target, the wrapper runs SkillSpector in static mode by default: - -```bash -skillspector scan --no-llm --format json --output -skillspector scan --no-llm --format sarif --output -``` - -If `llm: true`, it omits `--no-llm` and relies on caller-provided environment variables or secrets. The README must warn users not to pass LLM credentials to untrusted fork PRs. - -The wrapper merges per-target outputs into: - -- `skillspector-results/results.json` -- `skillspector-results/results.sarif` -- `skillspector-results/summary.md` - -The job summary displays the Markdown summary. - -## Failure Policy - -Default behavior is report-only: - -```yaml -fail-on: none -``` - -Failure is opt-in: - -- `fail-on: high` fails when any finding severity is `high` or `critical`. -- `fail-on: critical` fails when any finding severity is `critical`. -- `min-score` fails when the highest risk score is greater than or equal to the configured threshold. - -If both `fail-on` and `min-score` are configured, either condition can fail the action. - -## Suppression Policy - -The MVP supports two suppression mechanisms: - -- `exclude` prevents target discovery and report inclusion for matching paths. -- `baseline` suppresses findings matching a stable key built from rule id, normalized file path, and message fingerprint. - -Suppressed findings are omitted from failure decisions but counted separately in JSON and summary output. The baseline format is owned by this action so it can remain stable even if SkillSpector report details change. - -## Security Defaults - -The action follows conservative CI defaults: - -- `llm` defaults to `false`. -- `fail-on` defaults to `none`. -- Recommended workflow permissions are `contents: read`; `security-events: write` is only required when SARIF upload is enabled. -- Documentation must recommend `pull_request`, not privileged `pull_request_target`, for scanning untrusted PR code. -- Documentation must recommend pinning third-party actions to a full commit SHA in high-security environments. -- The publish workflow should pin the upstream SkillSpector source by commit SHA or a controlled build argument. -- The published image should be public so users do not need registry credentials. - -## Repository Files - -Expected implementation files: - -- `action.yml`: composite action metadata and public inputs/outputs. -- `Dockerfile`: scanner runtime image with SkillSpector installed. -- `scripts/entrypoint.sh`: container entrypoint that invokes the scanner wrapper. -- `src/skillspector_action/`: Python scanner wrapper package. -- `tests/`: unit tests for discovery, suppression, SARIF/JSON merging, and failure policy. -- `.github/workflows/ci.yml`: test workflow for this action repo. -- `.github/workflows/publish-image.yml`: GHCR publish workflow for tags/releases. -- `README.md`: usage, inputs, outputs, security notes, and examples. - -## Publishing - -This repository publishes the runtime image to GHCR using GitHub Actions. Tags should include: - -- commit SHA for immutable development builds. -- semantic version tags such as `v1.0.0`. -- major version tag such as `v1`. - -The action should default to the major version image for normal users, while advanced users can override `image` for testing. - -## Testing Strategy - -Use test-driven implementation for wrapper behavior: - -- Discovery finds skill directories from `SKILL.md`. -- Changed-only scanning maps changed files to owning skill directories. -- Exclude patterns remove targets. -- Baseline suppresses matching findings. -- Failure policy handles `none`, `high`, `critical`, and score threshold. -- Report merger produces valid JSON and SARIF structures. - -Use CI to run unit tests and a smoke test against a tiny fixture skill. Image publishing should run only after tests pass. - -## Non-Goals for MVP - -- Marketplace publication polish beyond usable metadata. -- Advanced PR annotations outside SARIF. -- Auto-generating baseline files. -- Full LLM provider setup. -- Non-GitHub CI support. -- Reimplementing SkillSpector detection logic. - -## References - -- SkillSpector supports directory, file, URL, zip, JSON, Markdown, SARIF, and `--no-llm` scanning. -- GitHub public packages are free to use, and public container images can be pulled anonymously. -- GitHub Actions usage remains the caller repository's normal Actions usage. -- SARIF upload uses `github/codeql-action/upload-sarif` with `security-events: write`.