hawkscan: scan-planning discovery + post-scan quality gate + branded output (v2.4.0) [ENG-704] by kcberg · Pull Request #76 · stackhawk/agent-skills

kcberg · 2026-07-20T12:32:05Z

What & why

Rewrites the hawkscan skill to get the app + scan configuration solid before the fix loop, then quality-gate every scan against code-derived expectations — plus StackHawk-branded loop progress output and a batch of skill-eval matrix-consistency fixes. Tracking: ENG-704 (skill), ENG-708 (eval harness).

Skill (hawkscan):

Code-first discovery (references/scan-planning.md) — per-surface detection, route-inventory derivation, verified-accurate OpenAPI handling (references/openapi-specs.md, incl. the base-path resolve-check that catches the 404-wall), and a user-confirmed pre-scan summary.
Post-scan quality gate (SKILL.md Step 4.5 + references/scan-quality.md) — five checks (coverage evidence-only, base-path resolve, auth, surface-completeness, health), 8 stable reason identifiers, additive-only iteration (cap 2 interactive / 1 autonomous), feedback-loop-not-governor (never blocks a finding).
Branded progress output — every loop phase narrates a StackHawk | status line (ASCII-only; Cursor/Windows-safe).

Eval harness (matrix consistency):

guard_scope denial classification — a blocked-and-recovered app-start reflex is a warning, not a blocking fail (fixed cross-model flapping on discovery cells).
Skip claude-code-only discovery cells on other harnesses (removed phantom fails); agy disabled in CI (OAuth-only).
data-seed: tier live-only checks extended + add a Phase-2 reasoning fixture (ds-17).
hawkscan-ci: trigger description now has explicit "Do NOT trigger for" exclusions.

Type of change

Minor — additive capability (no breaking changes to triggers or existing commands). bump-version.sh --minor 2.3.4 → 2.4.0 across all manifests + frontmatter.

Tests

Skill-evals (workflow_dispatch, latest run on this branch): hawkscan 🟢 on all Claude models (26/26), data-seed 🟢 across all model/platforms (17/17), api/optimize green; codex/cursor unblocked. Read-only + gate behavior verified; branded output is a live-loop behavior confirmed by dogfood (not CI-observable).
lib unit tests: uv run pytest — 197 passed.
config validation: uv run validate — all suites valid.
cursor rules: regenerated + idempotent; version consistency verified across all manifests.

Adds references/scan-planning.md, the code-first discovery procedure that configures HawkScan for the highest-value DAST coverage: find every API surface (docs-first, then per-framework route detection), recommend code changes for spec/auth gaps, ask the user rather than guess on ambiguous URL space, configure one stackhawk.yml per surface ordered by DAST value, and cross-check config shapes against docs.stackhawk.com / hawk config show. Absorbs and supersedes the docs-first source table from app-discovery.md; the route-inventory derivation guidance here is the contract the future quality-gate reference reuses to diff actual scan coverage against expectation.

Adds the post-scan quality gate reference that runs after every scan, before findings are parsed into fix tasks. Encodes the four settled gate parameters: structural-only gap threshold (route-diff is always computed/reported as evidence but never fails the gate alone), a 2-interactive/1-autonomous iteration cap counted in rescans, config-vs- environment signal classing, and sequential value-ordered multi-config execution with per-config gate/caps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Add auth-validate-failed to the config-class bullet so the signal-class list is actually closed over all non-evidence identifiers - Use a per-surface config placeholder in the auth check command instead of a hardcoded stackhawk.yml, matching the file's multi-config framing - Reconcile the surface-completeness degradation fallback to name both the local per-surface yml files and hawk validate config output Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rewires SKILL.md around the new scan-planning.md (code-first discovery) and scan-quality.md (post-scan quality gate) references: shrinks Step 1a to a pointer, adds a gate-gap bullet to Step 2b, inserts Step 4.5 Quality Gate between the scan run and findings parsing, adds gate guard rails to the Autonomous Security Loop and Common Mistakes, adds hawk op scan uris/config rows to Companion Skills, and updates the frontmatter description. No mode-dispatch section added; app-discovery.md stays linked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

app-discovery's goal-driven framing and discovery procedure were absorbed into references/scan-planning.md. Slim the file to a pointer paragraph plus the docs-first source table, which SKILL.md's Step 1a still links to directly.

…bump minor Adds MAPPINGS rows for the new scan-planning.md (code-first discovery) and scan-quality.md (post-scan quality gate) references so Cursor users get the same guidance as Claude/Codex. Bumps 2.1.1 -> 2.2.0 across all platform manifests and SKILL.md frontmatter.

…iew polish Insert an explicit Step 4.5 quality-gate invocation into the autonomous loop's 7-step detail (before findings triage and again after the rescan), cap autonomous config-fix iterations at 1, reconcile the exit-1 retry guard rail with the gate's environment-class rule, and qualify the clean-report wording so a zero-findings rescan with open gate gaps is never reported as "done and secure". Also: add missing soapConf to scan-planning's spec-wiring list, remove an overclaimed SKILL.md-consumes-identifiers contract from scan-quality.md, qualify SKILL.md's Step 4.5 "any exit code" for the no-scan-started case, restore precision to the Step 1a discovery hand-off line, and retitle app-discovery.md so its H1 matches its actual (docs-first table) content. Regenerated cursor/ .mdc mirrors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

Same applicationId, one environment per surface (dev-rest / dev-graphql), configs named stackhawk-<surface>.yml. Decided with KC at the dogfood checkpoint 2026-07-07. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

… gate signal Dogfood against ecordel-restapi showed a scan can 404 on 71% of its URIs when app.host and the spec disagree on the servlet context path, while login still works — and the old guidance only fired when NO spec was found, trusting any checked-in file. - New references/openapi-specs.md: framework-generic procedure for an ACCURATE spec — prefer a spec the running app serves; suggest the concrete code/build change that makes the framework generate one (12-stack matrix + find-out fallback); base/context-path alignment rules; mandatory resolve-check (curl host+spec-path) before scanning; hand-derived spec as last resort. - scan-planning.md: spec verdict is now verified-accurate | stale/unverified | missing (a stale spec is functionally missing); gap recommendation covers inaccurate specs, not just absent ones. - scan-quality.md + SKILL.md Step 4.5: new base-path-mismatch config-class gate signal for the mixed 404/200 pattern all-4xx misses. - Cursor MAPPINGS + regenerated rules (idempotent); bump 2.2.0 -> 2.3.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

…dPaths demotion Dogfood round 2 (firefly-iii + gitea) showed: references only loaded on friction (gitea hit a spec wall and opened openapi-specs.md -> 618/688 coverage; firefly's seedPaths 'worked' so nothing forced consultation -> 22/254). Five tuning changes: 1. SKILL.md Step 2a: a proactive, imperative REST-spec directive — get an accurate spec BEFORE the first scan, open+follow openapi-specs.md, run the resolve-check; seedPaths is a last resort not a shortcut. Fires without waiting for a failure. 2. openapi-specs.md: new preference-order step for a spec published OUTSIDE the repo (docs site, <project>/api-docs repo, release asset) — Firefly III's exact case — ahead of derive-by-hand. 3. seedPaths demoted below a hand-derived spec (URLs-only: no methods/ bodies/params) in openapi-specs.md + SKILL.md low-path bullet. 4. resolve-check reinforced as a pre-first-scan step via the SKILL directive (gitea ran it reactively, burning a scan). 5. Companion note: select an app by UUID with --app-id on released hawk (--app is name-only there; newer hawk accepts UUID on --app); parse --format json defensively past the skills-version banner. scan-planning.md gap section points at the external-spec + resolve steps. Cursor rules regenerated (idempotent); 2.3.0 -> 2.3.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

Release is coordinated so hawkscan (carrying the hawkop --app name|uuid change) ships before the skills. Skills now present --app as accepting a name or a UUID and drop the --app-id fallback guidance (--app-id remains a working hidden alias). repo link keeps --app-id (that command wasn't unified). Rewrote the authoritative --app/--app-id explanation in api hawk-op-shortcuts.md, the hawkscan companion table + selecting-an-app note, tech-flags.md, platform-model.md, and optimize cli-contract.md. Cursor rules regenerated; 2.3.1 -> 2.3.2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

…skills Adds an early skills-freshness check to the preflight — recommends the user update (surfacing the exact /plugin update command) when the installed agent skills are out of date, before continuing. Pairs with the hawkscan-side removal of the ambient per-command staleness banner: freshness is now reported only on the explicit 'hawk skills status', which the skill drives once up front. Recommendation, not a hard stop; skipped gracefully if the subcommand predates the installed hawk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

Local pr-review caught three text-consistency gaps from commit a05e16a (adding base-path-mismatch) not fully propagated: - autonomous-loop.md Step 5 config-class list was missing base-path-mismatch (the unattended path — the one place a doc gap bites hardest). - 'four checks' naming (SKILL.md, scan-quality.md header/ToC, autonomous-loop.md) didn't reflect the added base-path resolve check — now 'five checks' (coverage evidence-only + base-path/auth/surface/health gating). - scan-quality.md said 'seven reason identifiers' but lists eight. Cursor rules regenerated (idempotent); 2.3.3 -> 2.3.4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

The agent only knows the hawk CLI version (preflight requires >= 6.0.0), not the embedded hawkop-lib version, so "hawk op >= 0.11.7" gates were unknowable and unnecessary — scan uris/config ship with the combined CLI. Reframe to behavior-based degradation ("skip if the subcommand errors"). Also remove struck-design lineage that a fresh user has no context for: "(replace the old autonomy lock)" heading, "no longer treats exit 0 as success", "superseded by", legacy standalone-binary config file, and the deprecated --app-id alias note. Anchor rename propagated to the in-file ToC; cursor rules regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

…gate blind spot Base-path mismatch is the documented worst-case failure (wall of 404s), yet the only pre-scan defense was a manual curl resolve-check buried in openapi-specs.md. Fold its result into the mandatory user-confirmation summary in scan-planning.md so the highest-value check is no longer skippable. Also document in scan-quality.md that the gate re-derives its no-spec expectation with the same greps discovery uses, so it inherits discovery's detection blind spots — the user-confirmed discovery summary is the real backstop, not the gate. ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

…ch; record rule LLMs copy runnable code blocks verbatim, so an app-specific literal baked into a command is a trap — it runs against a different repo and silently does the wrong thing. The resolve-check curl in openapi-specs.md used literal /authors and /api/v1/authors; replace with <spec-path>/<base-path> placeholders that fail loudly on verbatim paste, keeping the concrete values as labeled e.g. comments. Add one framing line to scan-planning.md's derivation table (illustrative starting points — not literal src/ or openapi.yaml). Record the underlying principle in the skill-authoring skill: runnable blocks use placeholders that fail loudly; concrete values live in prose/comments as e.g.; literals only when genuinely universal. ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

… by reading, not by starting the app Skill-eval discovery cells (firefly-iii, memos) fail a BLOCKING read-only guard on every model — the agent stands up the app/container during discovery (docker compose up, php artisan serve) even when told not to, to learn host/port. A three-way eval comparison (HEAD vs this session's base af3f3d1 vs old pre-rewrite main) shows the behavior is identical across all three: not caused by recent edits, long-standing skill-wide. Fix: state in SKILL.md Step 1a and scan-planning.md that discovery is read-only and static — host, port, and run command come from reading docker-compose.yml port mappings, .env/.env.example, and the README; starting the app is a scan-time step (Step 1c), never part of discovery. This is also better real-world behavior: no need to spin up Docker to read a port that's declared in config. ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

…-708) The read-only discovery guard denied any command containing the bare word `docker` (also podman/nerdctl), which false-failed read-only inspection the agent legitimately uses to read port mappings during discovery — `docker ps`, `docker images`, `docker compose config`, `docker inspect`. For a Docker-packaged app (firefly-iii) that single denial flips the BLOCKING discovery_stayed_read_only check and fails an otherwise-correct, fully read-only discovery. Narrow to commands that actually start a container/app (docker run/start/create, docker compose up/run, docker-compose up, podman/nerdctl run/start, podman-compose up); allow read-only inspection. Adds test_denies_container_start + test_allows_container_readonly_inspection. ENG-708. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

…ismatch, hw-22 auth-wall) The post-scan quality gate had no eval coverage — it needs a completed scan, which CI can't run. Cover the gate's *reasoning* instead: two cells inline a synthetic completed-scan result and ask the agent to apply the gate. No harness change needed — uses applies_to-scoped process-checks (the same mechanism the discovery cells use). - hw-21: spec paths all 404 while /api/v1-prefixed variants resolve → agent must name the base-path-mismatch signal. - hw-22: auth-wall flag + 22/254 coverage on a reachable app → agent must name auth-wall (config-class) and report the coverage evidence. - Shared: must not declare the scan "done and secure" while a gap is open. Checks are scoped via applies_to so they don't touch the other 24 cells. Bumped the hardcoded prompt count in test_cli_resilience (24→26). ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

…ile no-secure check hw-21/hw-22 are reasoning-only (no live scan), but grade() applied the global scan-flow checks (hawk version, hawk op app list, scan --json-output...) which a paper exercise never performs — failing the cells flakily by model. Mirror the discovery-cell precedent: add an own_checks_only cell flag routing these through grade_discovery (own applies_to checks + expected only), same as answer_key cells. Also drop gate_no_false_done_secure: output_negative substring-matched the CORRECT behavior — the agent quoting the rule ("never claim \"done and secure\"") tripped it. Telling a genuine false claim from a quoted rule needs the semantic rubric grader (not run in CI), so the check produced false failures. Keep the robust positive checks (names base-path-mismatch / auth-wall / coverage evidence). ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

… discovery answer Per the devkit/eval author's guidance: evals exist to catch regressions, not to pin one phrasing, and the scan-planning discovery drives a slightly different (more repo-accurate) answer. The firefly-iii key led run_command/host with docker-compose that isn't in the cloned repo — so an agent that correctly reports the in-repo path (`php artisan serve` on :8000) and notes Docker packaging is a separate image was judged "partial" against external knowledge. Reframe so the in-repo local-dev path OR the external Docker image both count as correct. Behavior-neutral for the skill; adjusts eval ground truth to the accurate target. ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

Users reported the autonomous loop runs too quietly. Add a canonical "Progress output" convention to SKILL.md: prefix every status line with `StackHawk | ` (ASCII only, no emoji — renders in Cursor, Windows cmd, PowerShell), emit one line entering each phase (discovery, config, scan, quality gate, findings, each fix batch, rescan, report) with the key number, and before a scan/rescan set a duration expectation + let hawk's native progress stream through (the honest "live progress" — the agent can't interject mid-synchronous scan). Status output, never a prompt. Propagate branded lines through autonomous-loop.md's phases (scan start/complete, gate result, findings, rescan) which were previously silent or plain. Greppable prefix so CI/users can filter loop status. ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

…ublic repo) The shipped reference said "(in dogfood: 22 scanned URIs against 254 routes)" — internal-testing language in a customer-facing skill. Reword to a generic illustration ("well under 10% of routes reached on a typical REST API") that keeps the point (spec-less scans reach little) without the internal framing. ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

…fouls block Cross-model inconsistency in the hawkscan discovery matrix (firefly-iii/memos sonnet blocking-fail while opus passes) traced to discovery_stayed_read_only being blocking on ANY guard denial. But a denial has two flavors: a blocked-and- recovered app-START attempt (php artisan serve / go run / docker run) is a soft, model-personality foul — the guard blocks it, discovery still completes read-only and correct; a write/scan/network-egress denial is a hard foul (discovery left its lane). Add a `guard_scope` field to guard_clean checks ("hard" | "app_start" | "all", default "all" — backward compatible). Scope discovery_stayed_read_only to "hard" (still blocking, now only real out-of-scope actions), and add discovery_no_app_start scoped to "app_start" at WARNING severity so the reflex is surfaced without flapping the matrix across models. +5 test cases. ENG-708. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

…ry cells as fails Two matrix-consistency fixes surfaced running the full cross-agent matrix: 1. agy is OAuth-only and can't authenticate headless (always red). Comment out the eval-agy job, drop `agy` from the platform options, and remove it from the report job's `needs`. `platform: all` now = claude-code + codex + cursor. Re-enable later per the header note once agy creds work in CI. 2. target_repo (discovery) cells clone + read-only-guard a real repo — only the claude-code harness supports that; codex/cursor short-circuit them. But the harness still GRADED the short-circuited cell, so its expected-signal checks failed against empty output → 4 phantom blocking fails per codex/cursor hawkscan run (the bulk of their low scores). Skip those cells on non-claude harnesses in cli.py so they render "not present", not failed. ENG-708. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

… reasoning fixture The data-seed matrix weakness (haiku/sonnet/codex-o3 at 12/16) traced almost entirely to phase2_minimal_seed_proposed, a blocking output_contains with 15 generic substrings. The cells run in an EMPTY env (0/16 target_repo, not a git repo, hawk perch seed unavailable), so the agent can only DOCUMENT the multi-step workflow, never reach a real Phase-2 proposal. Pass/fail then hinged on whether a model's prose incidentally contained a signal word (opus hit "minimum seed"; sonnet hit none) — a phrasing lottery correlated with verbosity, not skill quality. Two fixes, matching how hawkscan handles live-only behavior: 1. Tier the phase-2/phase-3/phase-4 execution+emission checks `extended` so single-turn CI cells grade only what an empty-env turn can achieve (trigger, preflight, discovery, honesty, anti-patterns); the deep flow is graded only in full_auto/dogfood against a real repo. 2. Add ds-17: an own_checks_only Phase-2 reasoning fixture that inlines a repo digest (users/orgs/api_keys + a bearer-JWT login) and grades the minimal-seed PROPOSAL (ds17_minimal_seed_proposed blocking + credential/confirmation-gate warnings) — reachable and meaningful in one turn. Skill unchanged: the weakness was an eval mismatch, not a skill deficiency. ENG-708. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

… + hand-off hawkscan-ci over-triggered in the eval matrix on doc-only, research, and local-scan prompts (hci-21 "update the README about CI scanning" fired on all 4 models; hci-22 "what CI providers does HawkScan support?"; hci-17 "run a scan before deploy"). Root cause: the frontmatter description listed CI+scan trigger phrases but had no "Do NOT trigger for" clause (skill-authoring requires one; the base hawkscan skill has one). Add explicit exclusions — documentation-only changes, informational/research questions, and running a local scan (that's the hawkscan skill; hawkscan-ci only edits CI pipeline config). Also reframe the opener to "WIRE ... into a CI/CD pipeline config file" and soften "requires a working local scan path" → if no stackhawk.yml exists yet, still trigger and route local-config to hawkscan (fixes the hci-14 hand-off). Mirrored into the cursor MAPPINGS description; regenerated. Description 1001 chars (<=1024). Trigger behavior is soft/model-dependent — will verify empirically. ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

Minor release: the scan-planning discovery + post-scan quality gate + OpenAPI spec-accuracy rewrite of the hawkscan skill, StackHawk-branded loop progress output, and the skill-eval matrix-consistency work (guard-denial classification, non-claude discovery-cell skip, agy disabled in CI, data-seed Phase-2 reasoning fixture, hawkscan-ci trigger exclusions). bump-version.sh --minor updated VERSION + all platform manifests + SKILL.md frontmatter atomically; cursor rules regenerated (idempotent). ENG-704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012sAkR2BzkbLxzYUuGzLAQa

linear-code · 2026-07-20T12:32:08Z

ENG-704

claude · 2026-07-20T12:32:22Z

Claude finished @kcberg's task in 5m 38s —— View job

Claude Code Review — Looks Good ✅

This PR rewrites the hawkscan skill around code-first scan discovery (scan-planning.md, openapi-specs.md) and a post-scan quality gate (scan-quality.md, SKILL.md Step 4.5) with branded StackHawk | progress output, plus a batch of eval-harness matrix-consistency fixes (guard-scope classification, discovery-cell skipping on non-claude-code harnesses, agy disabled in CI, data-seed tiering). Version bump 2.3.4 → 2.4.0 is consistent across all manifests and SKILL.md frontmatter.

Reviewed

Correctness — eval harness logic (grading.py guard-scope split, discovery_guard.py regex narrowing, cli.py own_checks_only/target_repo skip paths) traced through by hand; all internally consistent, no obvious bugs.
Security — no injection/secret-exposure concerns; discovery_guard.py changes narrow (not widen) the read-only guard's denial surface correctly (only actual container/app-start commands are exempted from the blanket docker match, read-only inspection commands like docker ps/docker inspect are now correctly allowed).
Performance — n/a (docs + eval harness only).
Testing — new tests (test_discovery_guard.py, test_guard_clean_check.py) directly cover the new guard-scope and container-start regex behavior; counts look right.
Conventions — version/manifest consistency verified across all files (VERSION, all plugin.json, marketplace.json, SKILL.md frontmatter all read 2.4.0); the 8 quality-gate reason identifiers are used consistently between SKILL.md, scan-quality.md, and autonomous-loop.md.

Minor suggestions (non-blocking, posted inline)

SKILL.md "Step 4.5: Quality Gate" physically runs before Step 4 (parse findings), not after — the "4.5" label reads as "between 4 and 5" which could mislead a reader skimming just the headers. Consider a name that doesn't imply ordinal position, or an explicit callout.
A few --app <APP_ID> examples in SKILL.md and hawk-op-shortcuts.md weren't swept to the new <NAME|UUID> placeholder style used elsewhere in the same files — harmless (an ID still works), just a documentation-consistency nit.
evals/hawkscan/process-checks.json's new gate_coverage_evidence_reported check includes generic signal tokens ("22", "coverage", "reached") that could pass on unrelated output since output_contains matches on any signal — a tighter regex requiring the numeric pair together would make it a stronger discriminator.

Not independently verified

Sandbox tool permissions blocked non-git shell execution (pytest, the cursor-rule generator, hawk itself) in this review environment, so I relied on static review rather than re-running uv run pytest / bash scripts/generate-cursor-rules.sh myself. The PR body's reported results (197 unit tests passing, cursor rules idempotent, 26/26 and 17/17 skill-eval green) line up with what the diff shows, and I traced the affected logic by hand.

claude · 2026-07-20T12:42:50Z


 ---

+## Step 4.5: Quality Gate


[suggestion] Step 4.5: Quality Gate is placed and executes before Step 4: Parse Findings (per scan-quality.md: "sitting between Step 3's scan run and Step 4's fix-task generation"), but the "4.5" label conventionally reads as "between 4 and 5." A reader skimming just the headers could reasonably expect this gate to run after Step 4 rather than before it. If renumbering the rest of the doc is too disruptive, consider a name that doesn't imply ordinal position (e.g. "Step 3b: Quality Gate") or a one-line callout at the top of the section stating explicitly "this runs before Step 4, not after."

claude · 2026-07-20T12:42:55Z

+| Set specific tech flags       | `hawk op app tech-flags set --app <NAME\|UUID> Key=true`                          |
 | Triage a finding              | `hawk op scan triage --scan <ID> --hash <HASH> --status false-positive --note ""` |
 | Bulk triage from file         | `hawk op scan triage --scan <ID> --from-file triage.yaml`                         |
 | Annotate w/o triage perm      | `hawk op finding note --scan <ID> --hash <HASH> --note "..."`                     |


[suggestion] This clarifies that --app accepts a name or UUID, which is great, but several other --app examples in this same file (e.g. lines 202-203: hawk op env list --app <APP_ID>, hawk op env create --app <APP_ID> ...) and in plugins/api/skills/api/references/hawk-op-shortcuts.md (e.g. lines 152, 164, 194, 257-260) still use the <APP_ID> placeholder for the exact same flag. Not a functional bug since a resolved ID still works, but it undercuts the new guidance here — an agent skimming those other spots could conclude --app needs an ID specifically. Consider sweeping the remaining <APP_ID> placeholders to <NAME|UUID> (or a shared term) for consistency, or note explicitly that <APP_ID> is just shorthand for "whichever identifier you already have."

claude · 2026-07-20T12:43:03Z

+      "type": "output_contains",
+      "applies_to": ["hw-22"],
+      "signals": ["22", "254", "coverage", "reached"],
+      "severity": "warning",


[suggestion] signals here includes bare "22" and "254" alongside generic words like "coverage"/"reached". Since output_contains passes on ANY signal match, this check will pass almost any output that mentions the word "coverage" at all (very likely for a gate-reasoning prompt), which makes it a weak discriminator for whether the model actually reported the specific 22/254 evidence. Consider requiring the numeric pair together (e.g. a regex/adhoc check for 22.*254 or 22 of 254) so this only passes when the concrete coverage numbers are actually cited.

Bwvolleyball · 2026-07-20T19:13:17Z

+
+Do **not** split the difference — e.g. stripping the context path from `app.host` and
+hardcoding it only into `loginPath`/`testPath`. That fixes auth but leaves every
+`openApiConf` path unprefixed, which is exactly how a scan ends up 71% 404s while login still


yes, this is exactly how a scan ends up with precisely 71% 404's!

I need at least 2 significant digits to verify this claim

There are 2? a 7 and a 1?

kcberg and others added 29 commits July 13, 2026 16:31

fix(hawkscan): count-proof autonomous loop label

0c6c7b7

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WWyT8CN77z9xrJhRLjrtJC

kcberg requested a review from a team as a code owner July 20, 2026 12:32

claude Bot reviewed Jul 20, 2026

View reviewed changes

Bwvolleyball reviewed Jul 20, 2026

View reviewed changes

Bwvolleyball approved these changes Jul 20, 2026

View reviewed changes

kcberg merged commit ba4bab4 into main Jul 20, 2026
4 checks passed

kcberg deleted the feat/scan-planning-quality-gate branch July 20, 2026 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

hawkscan: scan-planning discovery + post-scan quality gate + branded output (v2.4.0) [ENG-704]#76

hawkscan: scan-planning discovery + post-scan quality gate + branded output (v2.4.0) [ENG-704]#76
kcberg merged 29 commits into
mainfrom
feat/scan-planning-quality-gate

kcberg commented Jul 20, 2026

Uh oh!

linear-code Bot commented Jul 20, 2026

Uh oh!

claude Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

claude Bot Jul 20, 2026

Uh oh!

claude Bot Jul 20, 2026

Uh oh!

claude Bot Jul 20, 2026

Uh oh!

Bwvolleyball Jul 20, 2026

Uh oh!

d-co-white Jul 20, 2026

Uh oh!

Bwvolleyball Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

kcberg commented Jul 20, 2026

What & why

Type of change

Tests

Related

Uh oh!

linear-code Bot commented Jul 20, 2026

Uh oh!

claude Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude Code Review — Looks Good ✅

Reviewed

Minor suggestions (non-blocking, posted inline)

Not independently verified

Uh oh!

claude Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Bwvolleyball Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

d-co-white Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Bwvolleyball Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

claude Bot commented Jul 20, 2026 •

edited

Loading