Use runtime CLI signals for audits

README.md

@@ -31,6 +31,45 @@ In any Claude Code session, type:
 
 Claude will walk you through setup interactively — pick repos, choose tasks, confirm before anything is created.
 
+## Required CLI tools for Night Shift runs
+
+Some Night Shift audit tasks can consume runtime signals from local CLI tools (performance, accessibility, and code-security checks). Install these tools on the machine that runs Night Shift routines.
+
+- `semgrep` (code security and code-pattern findings)
+- `lighthouse` (performance metrics for key pages)
+- `@axe-core/cli` (accessibility checks for key pages)
+
+### Install
+
+```bash
+# Semgrep (Homebrew)
+brew install semgrep
+
+# Lighthouse + axe CLI (npm)
+npm install -g lighthouse @axe-core/cli
+```
+
+If you prefer not to install globally, use `npx` in task commands:
+
+```bash
+npx lighthouse --version
+npx @axe-core/cli --help
+```
+
+### Verify
+
+```bash
+semgrep --version
+lighthouse --version
+axe --version
+```
+
+### Notes
+
+- `semgrep` is available via Homebrew.
+- `@axe-core/cli` is installed via npm (not Homebrew).
+- Use latest tool versions by default unless your org needs pinned versions for reproducibility.
+
 Night Shift supports two backends:
 
 | Backend | How it runs | Requirements |

docs/roadmap.md

@@ -101,3 +101,19 @@ Today Night Shift only does what its prompts define — it discovers work on its
 - **Client-facing reports** — could be added later, but internal adoption comes first
 - **Cost optimization** — model choice, turn limits, skipping idle repos — optimize after the system is proven valuable
 - **Multi-model strategy** — using cheaper models for simpler tasks — premature optimization for now
+
+## Next candidate phase: unified quality-signal ingestion (Plan B)
+
+**Goal:** Move from task-local CLI signals to a shared, validated signal layer consumed by multiple tasks.
+
+Current v1 direction (already useful): `improve-accessibility` consumes Axe CLI findings and `find-security-issues` consumes Semgrep findings directly in each task, with heuristic fallback.
+
+Plan B evolves this into one shared ingestion contract:
+
+- Add a prelude step that writes a normalized `/tmp/night-shift-signals.json`
+- Include tool metadata, timestamps, target URL/path, severity/impact, and parse errors
+- Enforce freshness and context checks (age window, app scope, target matching)
+- Let `improve-performance`, `improve-accessibility`, and `find-security-issues` consume the same schema
+- Keep graceful fallback to heuristic review when signals are missing or invalid
+
+**Why later:** Better consistency and observability, but meaningfully more implementation complexity than v1.

tasks/find-security-issues.md

@@ -24,25 +24,36 @@ Regardless of app scope, once per repo per night, grep the whole repo for accide
    ```
    gh pr list --search "night-shift/security in:title" --state open
    ```
-2. Look for patterns (inside `<app_path>` when scoped; skip the "Exposed secrets" row here — it's handled by the pre-step above):
+2. Collect Semgrep findings first (best effort), then use manual pattern review as fallback:
+   - Run Semgrep in JSON mode (scope to `<app_path>` when scoped):
+   ```
+   # scoped
+   semgrep scan --config auto --json "<app_path>" > /tmp/night-shift-semgrep.json
+   # unscoped
+   semgrep scan --config auto --json . > /tmp/night-shift-semgrep.json
+   ```
+   - If Semgrep is unavailable or fails, continue with manual OWASP pattern review (do not fail the task).
+   - Use Semgrep findings as the primary prioritization signal when present.
+
+3. Look for patterns (inside `<app_path>` when scoped; skip the "Exposed secrets" row here — it's handled by the pre-step above):
    - **Auth bypass / broken access control** — routes/handlers that read user IDs from request without authorization checks (IDOR)
    - **Injection** — raw SQL string building, shell exec with user input, unsafe template eval
    - **XSS** — `dangerouslySetInnerHTML`, `innerHTML`, unescaped output in templates
    - **Exposed secrets** — API keys, tokens, private URLs in client bundles, public repos, or `NEXT_PUBLIC_*`
    - **CSRF** — state-changing routes without origin/CSRF protection
    - **Missing rate limiting** on auth, signup, password reset, expensive endpoints
    - **Insecure deserialization, SSRF, open redirects**
-3. Pick **one** real, non-speculative issue tonight. Skip anything that requires guessing about intent. When scoped, the issue and its fix must live under `<app_path>`.
-4. Create a branch (include the app slug when scoped):
+4. Pick **one** real, non-speculative issue tonight. Skip anything that requires guessing about intent. When scoped, the issue and its fix must live under `<app_path>`. When Semgrep findings exist, prioritize the highest-severity, clearly valid finding first.
+5. Create a branch (include the app slug when scoped):
    ```
    # scoped:
    git checkout -b night-shift/security-<app-slug>-YYYY-MM-DD
    # unscoped:
    git checkout -b night-shift/security-YYYY-MM-DD
    ```
-5. Fix the issue at the smallest reasonable scope. Add a regression test.
-6. Run the scoped **test suite** and the scoped **build command**. Both must pass.
-7. Push and open a PR (prefix the title with `<app_path> — ` when scoped). The wrapper has already created the standard labels for this repo — just attach them. **Always use `--body-file`, never inline `--body`.** End the body with the Night Shift footer:
+6. Fix the issue at the smallest reasonable scope. Add a regression test.
+7. Run the scoped **test suite** and the scoped **build command**. Both must pass.
+8. Push and open a PR (prefix the title with `<app_path> — ` when scoped). The wrapper has already created the standard labels for this repo — just attach them. **Always use `--body-file`, never inline `--body`.** End the body with the Night Shift footer:
    ```
    cat > /tmp/night-shift-pr-body.md <<'EOF'
    ## Plain summary
@@ -59,6 +70,9 @@ Regardless of app scope, once per repo per night, grep the whole repo for accide
 
    ## Verification
    <how the new test demonstrates the fix>
+
+   ## Detection source
+   <semgrep | heuristic-fallback | mixed>
 
    ---
    _Run by Night Shift • audits/find-security-issues_

tasks/improve-accessibility.md

@@ -24,9 +24,20 @@ Only open a PR for violations that are clearly demonstrable from the code and wh
    ```
    If one exists for the same app, exit silently — do not stack PRs.
 
-2. Read the source of each configured **key page** (scoped to `<app_path>` when set) and the components they render.
+2. Collect runtime a11y signals from Axe CLI (best effort), then use source review as fallback:
+   - Exclude pages that require login/authentication.
+   - For each remaining key page URL, run Axe and store JSON output:
+     ```
+     npx -y @axe-core/cli "<absolute-production-url>" \
+       --stdout --format json > "/tmp/night-shift-axe-<page-slug>.json"
+     ```
+   - If Axe fails on a page (timeout/network/tooling), continue with remaining pages and report partial results in the PR.
+   - If Axe is unavailable in the environment, continue with heuristic WCAG code review (do not fail the task).
+   - Use Axe findings as the primary prioritization signal when present.
 
-3. Audit against the **WCAG 2.1 AA success criteria**. Check each category:
+3. Read the source of each configured **key page** (scoped to `<app_path>` when set) and the components they render.
+
+4. Audit against the **WCAG 2.1 AA success criteria**. Check each category:
 
    **Perceivable**
    - Images missing meaningful `alt` text (decorative images should have `alt=""`)
@@ -59,23 +70,23 @@ Only open a PR for violations that are clearly demonstrable from the code and wh
    - Duplicate `id` attributes on the same page
    - Heading hierarchy that skips levels (h1 → h3) or uses headings for styling
 
-4. Fix **all** clear violations. Prioritize by impact — issues that block access entirely (keyboard traps, missing form labels, broken focus management) before cosmetic issues (contrast tweaks on secondary text). Every violation must be real and demonstrable from reading the code, not hypothetical. Fix at the component level so all consumers benefit.
+5. Fix **all** clear violations. Prioritize by impact — issues that block access entirely (keyboard traps, missing form labels, broken focus management) before cosmetic issues (contrast tweaks on secondary text). Every violation must be real and demonstrable from reading the code, not hypothetical. Fix at the component level so all consumers benefit. When Axe findings exist, prioritize them first.
 
-5. **Verify fixes don't introduce regressions:**
+6. **Verify fixes don't introduce regressions:**
    - If the project has a11y tests (e.g. jest-axe, @axe-core/react, Playwright a11y assertions), add tests that would have caught the original violations.
    - Otherwise, add tests that exercise the fixes — at minimum render tests asserting the corrected attributes/elements are present.
    - Read the surrounding code and any existing tests for each component. Make sure fixes don't break existing behavior: conditional rendering, event handlers, CSS class names, prop interfaces.
    - Do not change existing `<title>`, metadata, heading text, or visible copy unless the fix specifically requires it.
 
-6. Collect all fixes in one branch (include app slug when scoped):
+7. Collect all fixes in one branch (include app slug when scoped):
    ```
    # scoped:
    git checkout -b night-shift/a11y-<app-slug>-YYYY-MM-DD
    # unscoped:
    git checkout -b night-shift/a11y-YYYY-MM-DD
    ```
-7. Run the scoped **test suite** and the scoped **build command**. Both must pass.
-8. Push and open the PR (prefix title with `<app_path> — ` when scoped). The wrapper has already created the standard labels for this repo — just attach them. **Always use `--body-file`, never inline `--body`.** End the body with the Night Shift footer:
+8. Run the scoped **test suite** and the scoped **build command**. Both must pass.
+9. Push and open the PR (prefix title with `<app_path> — ` when scoped). The wrapper has already created the standard labels for this repo — just attach them. **Always use `--body-file`, never inline `--body`.** End the body with the Night Shift footer:
    ```
    cat > /tmp/night-shift-pr-body.md <<'EOF'
    ## Plain summary
@@ -92,6 +103,7 @@ Only open a PR for violations that are clearly demonstrable from the code and wh
 
    ## Verification
    - <how to confirm each fix in the rendered page>
+   - Signal source: <axe-cli | heuristic-fallback | mixed>
 
    ---
    _Run by Night Shift • code-fixes/improve-accessibility_

tasks/improve-performance.md

@@ -30,7 +30,44 @@ Only open a PR when you can point to a concrete, low-risk win that will clearly
    - **Render-blocking resources** — synchronous scripts, blocking CSS, large inline blobs.
    - **Client/server boundary** — components marked client-only that could be server, large data fetched on the client that could be fetched on the server.
    - **Database / API calls** — N+1 patterns, missing indexes implied by query shape, missing caching where appropriate.
-   - **Lighthouse-style checks** for each configured **key page** by reading the source (no live browser needed).
+   - **Lighthouse CLI benchmark** for each configured **key page** against the production URL (do not replace with source-only heuristics).
+     - Benchmark all configured key pages each run (no page cap).
+     - Exclude pages that require login/authentication from the benchmark set.
+     - Measure both profiles for each key page:
+       - Mobile (default Lighthouse settings)
+       - Desktop (`--preset=desktop`)
+     - Include all core Lighthouse performance KPIs in prioritization:
+       - Performance score
+       - LCP
+       - INP
+       - CLS
+       - TBT
+     - Use this command pattern (per key page URL):
+       ```
+       # Run 3 times per page/profile and compare medians
+       for i in 1 2 3; do
+         lighthouse "<absolute-production-url>" \
+           --quiet \
+           --chrome-flags="--headless=new" \
+           --output=json \
+           --output-path="/tmp/lh-mobile-${i}.json"
+       done
+
+       for i in 1 2 3; do
+         lighthouse "<absolute-production-url>" \
+           --quiet \
+           --preset=desktop \
+           --chrome-flags="--headless=new" \
+           --output=json \
+           --output-path="/tmp/lh-desktop-${i}.json"
+       done
+       ```
+     - Use latest available Lighthouse CLI version.
+     - Treat measurement as best effort (network variance is expected on production targets), and use median values from the 3 runs when reporting/comparing.
+     - If a page benchmark fails (timeout, 403, or similar), continue the run and report partial results in the PR.
+     - Treat all listed KPIs with equal weight.
+     - Prioritize fixes by largest relative improvement potential across key pages and both profiles, not fixed absolute thresholds.
+     - It is acceptable to ship a change that improves overall Lighthouse score even if one or more individual KPIs regress, but the PR must include a clear warning naming the regressed KPI(s), affected page(s), and estimated regression size.
 3. Fix all clear, low-risk issues in one branch (include app slug when scoped):
    ```
    # scoped:
@@ -54,6 +91,7 @@ Only open a PR when you can point to a concrete, low-risk win that will clearly
 
    ## Expected impact
    - <bundle size delta, render path improvements, etc.>
+  - <best-effort Lighthouse deltas for key pages (mobile + desktop): score, LCP, INP, CLS, TBT>
 
    ---
    _Run by Night Shift • audits/improve-performance_

tests/test_nightshift.py

@@ -42,7 +42,7 @@
 MANIFEST_PATH = REPO_ROOT / "manifest.yml"
 TASKS_DIR = REPO_ROOT / "tasks"
 BUNDLES_DIR = REPO_ROOT / "bundles"
-SKILL_PATH = REPO_ROOT / "skill" / "SKILL.md"
+SKILL_PATH = REPO_ROOT / "skills" / "night-shift" / "SKILL.md"
 README_PATH = REPO_ROOT / "README.md"
 HOWTO_PATH = REPO_ROOT / "HOW-TO.md"
 
@@ -70,12 +70,12 @@ def all_task_ids() -> set[str]:
 
 
 class TestManifestStructure(unittest.TestCase):
-    def test_twelve_tasks(self):
-        self.assertEqual(len(load_manifest()["tasks"]), 12)
+    def test_manifest_has_expected_task_count(self):
+        self.assertEqual(len(load_manifest()["tasks"]), 15)
 
-    def test_four_bundles(self):
+    def test_manifest_has_expected_bundles(self):
         self.assertEqual(set(load_manifest()["bundles"].keys()),
-                         {"plans", "docs", "code-fixes", "audits"})
+                         {"plans", "docs", "code-fixes", "audits", "triage-ci"})
 
     def test_every_task_has_file(self):
         for tid in all_task_ids():
@@ -302,6 +302,7 @@ class TestBundleFilter(unittest.TestCase):
         "document-decisions", "suggest-improvements", "add-tests",
         "improve-accessibility", "translate-ui", "find-security-issues",
         "find-bugs", "improve-seo", "improve-performance",
+        "work-on-issues", "work-on-jira-issues", "triage-ci-failures",
     }
 
     def test_fall_back_returns_full_bundle(self):
@@ -479,12 +480,12 @@ def test_three_repos_mixed_selections(self):
         )
         plans_parsed = parse_allowlist(plans_prompt)
         self.assertFalse(plans_parsed.fell_back)
-        # Only repo-a has build-planned-features (audits-off leaves plans intact)
+        # Only repo-a has plans tasks (audits-off leaves plans intact)
         self.assertEqual(set(plans_parsed.repos), {repo_a})
         tasks, _ = filter_bundle_for_repo(
             list(plans_ids), plans_parsed, repo_a, self.manifest_ids,
         )
-        self.assertEqual(tasks, ["build-planned-features"])
+        self.assertEqual(sorted(tasks), sorted(self.by_bundle["plans"]))
 
         # --- Docs + code-fixes trigger ---
         docs_fix_ids = set(self.by_bundle["docs"]) | set(self.by_bundle["code-fixes"])