hawkscan: scan-planning discovery + post-scan quality gate + branded output (v2.4.0) [ENG-704]

.claude-plugin/marketplace.json

@@ -14,47 +14,47 @@
       "category": "security",
       "source": "./plugins/hawkscan",
       "homepage": "https://github.com/stackhawk/agent-skills/tree/main/plugins/hawkscan",
-      "version": "2.2.0"
+      "version": "2.4.0"
     },
     {
       "name": "stackhawk-api",
       "description": "Query the StackHawk platform API for security posture reporting, findings analysis, and app management.",
       "category": "security",
       "source": "./plugins/api",
       "homepage": "https://github.com/stackhawk/agent-skills/tree/main/plugins/api",
-      "version": "2.2.0"
+      "version": "2.4.0"
     },
     {
       "name": "hawkscan-ci",
       "description": "Configure HawkScan in your CI/CD pipeline. Provider-agnostic: detects GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Bitbucket, Buildkite, and other CI systems from repo files, edits the pipeline file in place, and prompts for HAWK_API_KEY storage (CI-native secrets store or external secrets manager).",
       "category": "security",
       "source": "./plugins/hawkscan-ci",
       "homepage": "https://github.com/stackhawk/agent-skills/tree/main/plugins/hawkscan-ci",
-      "version": "2.2.0"
+      "version": "2.4.0"
     },
     {
       "name": "stackhawk-data-seed",
       "description": "Read a target repo and emit checked-in seed artifacts so HawkScan has data to scan. Generates SQL / HTTP / gRPC / Mongo / shell steps across the target repo and any upstream services it depends on, plus a manifest and a credentials handoff file that hawkscan consumes.",
       "category": "security",
       "source": "./plugins/stackhawk-data-seed",
       "homepage": "https://github.com/stackhawk/agent-skills/tree/main/plugins/stackhawk-data-seed",
-      "version": "2.2.0"
+      "version": "2.4.0"
     },
     {
       "name": "wingman",
       "description": "StackHawk umbrella \u2014 one-command install of the default skill set (hawkscan, api, data-seed, optimize).",
       "category": "security",
       "source": "./plugins/wingman",
       "homepage": "https://github.com/stackhawk/agent-skills/tree/main/plugins/wingman",
-      "version": "2.2.0"
+      "version": "2.4.0"
     },
     {
       "name": "stackhawk-optimize",
       "description": "Analyze a codebase and produce an optimal HawkScan setup (tech flags, scan policy, stackhawk.yml), apply it as a non-destructive trial, run one trial scan, and promote or discard.",
       "category": "security",
       "source": "./plugins/optimize",
       "homepage": "https://github.com/stackhawk/agent-skills/tree/main/plugins/optimize",
-      "version": "2.2.0"
+      "version": "2.4.0"
     }
   ]
 }

.claude/skills/skill-authoring/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: skill-authoring
-version: 2.2.0
+version: 2.4.0
 description: >
   Guides authoring and maintaining agent skills in this repository — enforcing
   Anthropic best practices, bumping versions, regenerating Cursor rules, and
@@ -106,6 +106,32 @@ immediately before the first `##` section. Format:
 
 Shorter files may omit the ToC.
 
+### Examples vs. placeholders (avoid literal-match traps)
+
+Agents copy **runnable code blocks verbatim** far more readily than they adapt them. A
+concrete app-specific value baked into a command is therefore a trap: the agent runs it
+literally against a different codebase and silently does the wrong thing (curls a route that
+doesn't exist, greps a directory that isn't there).
+
+The rule that prevents this:
+
+- **Runnable code blocks use `<placeholder>`s that fail loudly if pasted as-is.** Prefer
+  `curl "$HOST<spec-path>"` over `curl "$HOST/authors"` — a literal paste of the placeholder
+  form breaks visibly, while a literal paste of `/authors` hits the wrong route silently. Put
+  the concrete value in an adjacent comment as a clearly-labeled `e.g.`
+- **Concrete examples live in prose or comments, marked `e.g.`** There they aid comprehension
+  without being executable, so literal-matching them is harmless.
+- **Literal values are fine only when they are genuinely universal** — e.g. probing the real,
+  fixed served-spec paths (`/openapi.json`, `/v3/api-docs`) is correct precisely because those
+  are the same across every app of that framework. If a value changes per codebase, it is a
+  placeholder, not a literal.
+- **Command tables (grep/derivation recipes) are illustrative starting points.** Say so once
+  near the table, and point them at the repo's real source root and filenames rather than a
+  literal `src/` or `openapi.yaml`.
+
+The test: *"if an agent runs this exact text against a completely different repo, does it fail
+loudly or mislead silently?"* Silent-mislead means parameterize it.
+
 ---
 
 ## Creating a New Plugin

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "stackhawk",
-  "version": "2.2.0",
+  "version": "2.4.0",
   "description": "Dynamic application and API security testing (DAST) powered by HawkScan. Configure scans, interpret findings, and query the StackHawk platform API.",
   "author": {
     "name": "StackHawk",

.github/workflows/skill-evals.yml

@@ -17,7 +17,7 @@ on:
         required: true
         default: "all"
         type: choice
-        options: [all, claude-code, codex, agy, cursor]
+        options: [all, claude-code, codex, cursor]  # agy disabled (OAuth-only, can't auth in CI)
       rubric:
         description: "Also run the qualitative rubric grader (extra ANTHROPIC_API_KEY cost)"
         required: false
@@ -256,105 +256,109 @@ jobs:
           retention-days: 30
 
   # ── Antigravity (agy) — replaces Gemini ───────────────────────────────────
-  eval-agy:
-    name: agy / ${{ matrix.skill }} / ${{ matrix.model }}
-    runs-on: ubuntu-latest
-    needs: validate-config
-    if: |
-      inputs.platform == 'all' ||
-      inputs.platform == 'agy'
-    strategy:
-      fail-fast: false
-      matrix:
-        skill: [hawkscan, api, stackhawk-data-seed, hawkscan-ci, optimize]
-        model: [default]
-
-    steps:
-      - uses: actions/checkout@v7
-      - uses: astral-sh/setup-uv@v5
-      - name: Install Claude Code CLI (native, rubric grader)
-        run: |
-          curl -fsSL https://claude.ai/install.sh | bash
-          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
-
-      - name: Install agy CLI
-        run: |
-          # /cli/install.sh is the real bootstrapper; /install-cli returns the
-          # site's HTML landing page (piping that into bash is what broke before).
-          curl -fsSL https://antigravity.google/cli/install.sh | bash
-          echo "$HOME/.local/bin" >> "$GITHUB_PATH"   # installer drops `agy` here
-        continue-on-error: true  # don't abort the job — evals records any launch failure
-
-      - name: Verify agy CLI
-        run: agy --version
-        continue-on-error: true  # if unavailable, the eval run captures it as a per-prompt error
-
-      - name: Install StackHawk plugins
-        env:
-          # NOTE: agy ignores API-key env vars and forces interactive Google OAuth
-          # (verified empirically; upstream antigravity-cli#78 open). This mapping is
-          # kept for if/when key auth lands; headless CI needs the OAuth-token
-          # injection below.
-          ANTIGRAVITY_API_KEY: ${{ secrets.AGY_API_KEY }}
-        run: |
-          echo y | agy plugin install plugins/hawkscan
-          echo y | agy plugin install plugins/api
-          echo y | agy plugin install plugins/stackhawk-data-seed
-          echo y | agy plugin install plugins/hawkscan-ci
-          echo y | agy plugin install plugins/optimize
-        continue-on-error: true  # depends on agy CLI; best-effort so evals still runs
-
-      # HawkScan 6 ships as a single self-contained native binary (no JDK) that
-      # provides BOTH scanning (`hawk`) and the platform/API CLI (`hawk op …`).
-      # Install it from the download manifest (source of truth for version + URL)
-      # into /usr/local/bin (already on PATH). One install covers the hawkscan
-      # skill's CLI path AND the api skill + hawkscan Step-1 dedup checks.
-      - name: Install hawk CLI (combined binary, manifest-driven)
-        run: |
-          set -euo pipefail
-          MANIFEST=https://download.stackhawk.com/hawkdocs/hawk.manifest.json
-          url="$(curl -fsSL "$MANIFEST" | jq -r '.latest.assets[]
-            | select(.asset.group == "linux-x64" and (.url | endswith("/hawk"))) | .url')"
-          [ -n "$url" ] || { echo "ERROR: no linux-x64 hawk binary in manifest"; exit 1; }
-          echo "Installing hawk from $url"
-          curl -fsSL "$url" -o /tmp/hawk
-          chmod +x /tmp/hawk
-          sudo mv /tmp/hawk /usr/local/bin/hawk
-        continue-on-error: true  # absence is captured per-prompt in the eval traces
-      - name: Verify hawk CLI
-        run: |
-          hawk version
-          hawk op --help
-        continue-on-error: true  # absence is captured per-prompt in the eval traces
-
-      - name: Run ${{ matrix.skill }} evals
-        env:
-          # agy ignores API-key env vars and forces OAuth (upstream #78); kept for
-          # if/when key auth lands. Headless CI needs OAuth-token injection (TODO).
-          ANTIGRAVITY_API_KEY: ${{ secrets.AGY_API_KEY }}
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}  # rubric grader (claude)
-          HAWK_API_KEY: ${{ secrets.HAWK_API_KEY }}
-          HAWK_FORMAT: json
-        run: |
-          MODEL_ARGS=()
-          if [ "${{ matrix.model }}" != "default" ]; then MODEL_ARGS=(--model "${{ matrix.model }}"); fi
-          RUBRIC=""; if [ "${{ inputs.rubric }}" = "true" ]; then RUBRIC="--rubric"; fi
-          uv run evals --harness agy --skill ${{ matrix.skill }} "${MODEL_ARGS[@]}" $RUBRIC
-        # No continue-on-error: the CLI now decides job color (exit 1 only on a
-        # PLUMBING FAILURE — 0 prompts ran cleanly). agy can't auth in headless CI
-        # (OAuth-only, upstream #78), so it exits 1 here and the job goes RED, which
-        # is correct: red = "couldn't run, needs work." The digest still renders agy's
-        # 🚫 cell because Upload results + the report job are both `if: always()`.
-
-      - name: Upload results
-        if: always()
-        uses: actions/upload-artifact@v7
-        with:
-          name: eval-agy-${{ matrix.skill }}-${{ matrix.model }}
-          path: evals/harnesses/agy/results/${{ matrix.skill }}/
-          retention-days: 30
-
-  # ── Cursor ────────────────────────────────────────────────────────────────
+  # --- eval-agy DISABLED: agy is OAuth-only and cannot authenticate in headless CI.
+  # Re-enable by uncommenting the job below, re-adding `agy` to the platform options,
+  # and adding `eval-agy` back to the report job's `needs:` list. See git history.
+
+#   eval-agy:
+#     name: agy / ${{ matrix.skill }} / ${{ matrix.model }}
+#     runs-on: ubuntu-latest
+#     needs: validate-config
+#     # agy is OAuth-only and cannot authenticate in headless CI (always red), so it is
+#     # EXCLUDED from `all` — run it explicitly with platform: agy only when needed.
+#     if: inputs.platform == 'agy'
+#     strategy:
+#       fail-fast: false
+#       matrix:
+#         skill: [hawkscan, api, stackhawk-data-seed, hawkscan-ci, optimize]
+#         model: [default]
+# 
+#     steps:
+#       - uses: actions/checkout@v7
+#       - uses: astral-sh/setup-uv@v5
+#       - name: Install Claude Code CLI (native, rubric grader)
+#         run: |
+#           curl -fsSL https://claude.ai/install.sh | bash
+#           echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+# 
+#       - name: Install agy CLI
+#         run: |
+#           # /cli/install.sh is the real bootstrapper; /install-cli returns the
+#           # site's HTML landing page (piping that into bash is what broke before).
+#           curl -fsSL https://antigravity.google/cli/install.sh | bash
+#           echo "$HOME/.local/bin" >> "$GITHUB_PATH"   # installer drops `agy` here
+#         continue-on-error: true  # don't abort the job — evals records any launch failure
+# 
+#       - name: Verify agy CLI
+#         run: agy --version
+#         continue-on-error: true  # if unavailable, the eval run captures it as a per-prompt error
+# 
+#       - name: Install StackHawk plugins
+#         env:
+#           # NOTE: agy ignores API-key env vars and forces interactive Google OAuth
+#           # (verified empirically; upstream antigravity-cli#78 open). This mapping is
+#           # kept for if/when key auth lands; headless CI needs the OAuth-token
+#           # injection below.
+#           ANTIGRAVITY_API_KEY: ${{ secrets.AGY_API_KEY }}
+#         run: |
+#           echo y | agy plugin install plugins/hawkscan
+#           echo y | agy plugin install plugins/api
+#           echo y | agy plugin install plugins/stackhawk-data-seed
+#           echo y | agy plugin install plugins/hawkscan-ci
+#           echo y | agy plugin install plugins/optimize
+#         continue-on-error: true  # depends on agy CLI; best-effort so evals still runs
+# 
+#       # HawkScan 6 ships as a single self-contained native binary (no JDK) that
+#       # provides BOTH scanning (`hawk`) and the platform/API CLI (`hawk op …`).
+#       # Install it from the download manifest (source of truth for version + URL)
+#       # into /usr/local/bin (already on PATH). One install covers the hawkscan
+#       # skill's CLI path AND the api skill + hawkscan Step-1 dedup checks.
+#       - name: Install hawk CLI (combined binary, manifest-driven)
+#         run: |
+#           set -euo pipefail
+#           MANIFEST=https://download.stackhawk.com/hawkdocs/hawk.manifest.json
+#           url="$(curl -fsSL "$MANIFEST" | jq -r '.latest.assets[]
+#             | select(.asset.group == "linux-x64" and (.url | endswith("/hawk"))) | .url')"
+#           [ -n "$url" ] || { echo "ERROR: no linux-x64 hawk binary in manifest"; exit 1; }
+#           echo "Installing hawk from $url"
+#           curl -fsSL "$url" -o /tmp/hawk
+#           chmod +x /tmp/hawk
+#           sudo mv /tmp/hawk /usr/local/bin/hawk
+#         continue-on-error: true  # absence is captured per-prompt in the eval traces
+#       - name: Verify hawk CLI
+#         run: |
+#           hawk version
+#           hawk op --help
+#         continue-on-error: true  # absence is captured per-prompt in the eval traces
+# 
+#       - name: Run ${{ matrix.skill }} evals
+#         env:
+#           # agy ignores API-key env vars and forces OAuth (upstream #78); kept for
+#           # if/when key auth lands. Headless CI needs OAuth-token injection (TODO).
+#           ANTIGRAVITY_API_KEY: ${{ secrets.AGY_API_KEY }}
+#           ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}  # rubric grader (claude)
+#           HAWK_API_KEY: ${{ secrets.HAWK_API_KEY }}
+#           HAWK_FORMAT: json
+#         run: |
+#           MODEL_ARGS=()
+#           if [ "${{ matrix.model }}" != "default" ]; then MODEL_ARGS=(--model "${{ matrix.model }}"); fi
+#           RUBRIC=""; if [ "${{ inputs.rubric }}" = "true" ]; then RUBRIC="--rubric"; fi
+#           uv run evals --harness agy --skill ${{ matrix.skill }} "${MODEL_ARGS[@]}" $RUBRIC
+#         # No continue-on-error: the CLI now decides job color (exit 1 only on a
+#         # PLUMBING FAILURE — 0 prompts ran cleanly). agy can't auth in headless CI
+#         # (OAuth-only, upstream #78), so it exits 1 here and the job goes RED, which
+#         # is correct: red = "couldn't run, needs work." The digest still renders agy's
+#         # 🚫 cell because Upload results + the report job are both `if: always()`.
+# 
+#       - name: Upload results
+#         if: always()
+#         uses: actions/upload-artifact@v7
+#         with:
+#           name: eval-agy-${{ matrix.skill }}-${{ matrix.model }}
+#           path: evals/harnesses/agy/results/${{ matrix.skill }}/
+#           retention-days: 30
+# 
+#   # ── Cursor ────────────────────────────────────────────────────────────────
   eval-cursor:
     name: cursor / ${{ matrix.skill }} / ${{ matrix.model }}
     runs-on: ubuntu-latest
@@ -438,7 +442,7 @@ jobs:
   # ── PR comment ────────────────────────────────────────────────────────────
   report:
     name: Eval report (run summary + PR comment)
-    needs: [validate-config, eval-claude-code, eval-codex, eval-agy, eval-cursor]
+    needs: [validate-config, eval-claude-code, eval-codex, eval-cursor]
     if: always()
     runs-on: ubuntu-latest
     permissions:

VERSION

@@ -1 +1 @@
-2.2.0
+2.4.0

cursor/.cursor/rules/stackhawk-api-hawk-op.mdc

@@ -53,8 +53,7 @@ hawk op status         # Confirm auth
 ```
 
 `hawk init` writes credentials to `~/.hawk/hawk.properties`.  
-`hawk op` reads that file plus the `HAWK_API_KEY` env var. The legacy
-standalone-binary config file is no longer used.
+`hawk op` reads that file plus the `HAWK_API_KEY` env var.
 
 **CI/CD only:** For pipeline use, set `HAWK_API_KEY` directly as a secret.
 Org ID and output format can be set via env vars:
@@ -109,8 +108,7 @@ extraction, no token handling.
 ### Latest scan for an app (overview + alerts)
 
 ```bash
-hawk op scan get --app "<APP_NAME>"
-hawk op scan get --app-id <APP_ID>
+hawk op scan get --app "<NAME|UUID>"    # --app takes a name or a UUID
 ```
 
 Wraps: `/scan/{orgId}` + `/scan/{scanId}/alerts`. Returns a scan header, alert
@@ -301,7 +299,7 @@ call — useful when building a reporting pipeline.
 - **Don't forget `--detail full` when you need the remediation/HTTP message payload.**
   The default `hawk op scan get` output is the overview — it won't include per-URI
   evidence.
-- **Don't combine `--app` (name) and `--app-id` (UUID) in the same command.**
-  Pick one. `--app` resolves by name (case-insensitive exact match); `--app-id` is exact by UUID.
-  If a name matches **more than one** app, `hawk op` errors and tells you to use `--app-id` —
-  fall back to `--app-id <UUID>` (grab the UUID from `hawk op app list`).
+- **`--app` takes a name or a UUID.** A UUID resolves exactly; a name resolves by
+  case-insensitive exact match. If a name matches **more than one** app, `hawk op` errors and
+  lists the matches — disambiguate by passing that app's UUID to `--app` (grab it from
+  `hawk op app list`).

cursor/.cursor/rules/stackhawk-api.mdc

@@ -33,8 +33,7 @@ Before making any calls, check what's available:
    - valid org + JWT → proceed.
    - installed but unconfigured → `hawk init --browser` (interactive; the combined binary
      has no `hawk op init` — `hawk init` writes `~/.hawk/hawk.properties`, which
-     `hawk op` reads along with the `HAWK_API_KEY` env var; the legacy
-     standalone-binary config file is no longer used).
+     `hawk op` reads along with the `HAWK_API_KEY` env var).
    - not installed → instruct the user to install the `hawk` CLI (docs:
      [docs.stackhawk.com](https://docs.stackhawk.com)) and stop; do **not** fall back to curl.