test(evals): negative-control regressions catch the cases they should

[kaiapeacock-eng]

Summary

Phase 5 of Week 2 — proof that the eval suite actually catches the regressions it was built to catch. Each test forks the existing nextjs-app-router/vanilla golden into a tmpdir, applies a deliberate regression, and asserts the full scorer stack reports the failure shape we expect.

Three controls:

1. Wrong SDK family — substitute @amplitude/analytics-browser for @amplitude/unified in the golden's package.json. L0-correct-sdk-package hard-fails with a detail naming the wrong family.
2. Hardcoded API key — plant the eval-only API key literal in the entry file. L0-no-hardcoded-key hard-fails.
3. Confirmed event without a track() call — strip a track('Sign In') while keeping it in the proposed plan. L1-confirmed-events-tracked fails with a detail naming 'Sign In'.

If a future scorer change makes a negative control unexpectedly pass, the suite caught a regression in itself — that's the point.

Test plan

• pnpm exec vitest run evals/ → 34/34 across 6 files
• pnpm lint clean
• All three negative controls fire as expected

Stacked: base is ba-104-layer3-and-ci (#570). Full chain: this → #570 → #567 → #560 → main.

🤖 Generated with Claude Code

---

Note

Low Risk
Low risk: adds new Vitest coverage only, exercising the existing replay/score pipeline without changing production scoring logic.

Overview
Adds a new Vitest negative-control suite that clones the nextjs-app-router/vanilla golden scenario into a temp directory, injects intentional regressions, and runs the full runReplay + score stack to assert the expected failure modes.

The tests verify three canary cases: wrong SDK family triggers L0-correct-sdk-package hard-fail, hardcoded API key triggers L0-no-hardcoded-key hard-fail, and missing track() for a confirmed event causes L1-confirmed-events-tracked to fail with the missing event called out.

^{Reviewed by Cursor Bugbot for commit 4aa30e6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[kelsonpw]

LGTM. Three sharp negative-control tests (wrong SDK family, hardcoded API key, missing track() for a confirmed event) all asserting on the failure shape — name in the detail, hard-fail on the L0 ones, soft-fail with event name on the L1. Self-validating in the right way: if a future scorer change makes one of these unexpectedly pass, the suite caught a regression in itself.\n\nNote: Node 22/24 unit-test failure (DataIngestionCheckScreen tracking-plan polling test) is a known flake unrelated to this PR — same test failed on 570 and 575 in the same window. Eval scorer + Ring 1 gate are green, which is what matters here. Stacked on #570; will merge cleanly once the chain lands.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Workflow paths filter doesn't include itself for self-validation
  • Added '.github/workflows/evals-pr-scenarios.yml' to its own paths trigger so changes to the workflow file will trigger CI validation.

Or push these changes by commenting:

@cursor push 002ac15775

Preview (002ac15775)

diff --git a/.github/workflows/evals-pr-scenarios.yml b/.github/workflows/evals-pr-scenarios.yml
--- a/.github/workflows/evals-pr-scenarios.yml
+++ b/.github/workflows/evals-pr-scenarios.yml
@@ -21,6 +21,7 @@
       - 'evals/**'
       - 'package.json'
       - '.github/workflows/evals-pr.yml'
+      - '.github/workflows/evals-pr-scenarios.yml'
 
 # Run-once-per-ref semantics: a new push to the PR cancels the prior
 # run. PR-gate evals are deterministic against golden replays today,

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 1c178b2. Configure here.}