Skip to content

TESTING.md

Latest commit

 

History

History
192 lines (139 loc) · 6.23 KB

TESTING.md

File metadata and controls

192 lines (139 loc) · 6.23 KB
Raw

Testing the pre-mortem Skill

Method: Scenario-Based LLM Eval

LLM skills can't be unit-tested like functions — the output is probabilistic prose. The method used here is scenario-based assertion testing, adapted from Hamel Husain's LLM eval approach:

"Unit tests for LLMs are assertions (like you would write in pytest)."

Key principle: you don't need 100% pass rate. Pass rate is a product decision. A critical assertion failing (security risk not surfaced) matters more than a nit assertion failing (exact word count). Grade accordingly.

Test Levels

Level What When
L1 — Assertions run-tests.sh checks output against assertions.txt Every time you change SKILL.md
L2 — Human eval Read actual output, judge quality Before shipping a SKILL.md version
L3 — Regression Compare to golden files After model upgrades

This repo covers L1 and L3. L2 is manual — see "Human Eval Checklist" below.

Directory Structure

tests/
  scenarios/
    01-ai-search/
      input.txt        ← feature description fed to the skill
      flags.txt        ← CLI flags (--depth, --format, etc.)
      assertions.txt   ← what the output must contain
      notes.txt        ← why this scenario, acceptance criteria, failure modes
    02-vague-input/
    03-depth-quick/
    04-depth-deep/
  fixtures/
    required-sections.txt  ← baseline assertions applied to ALL scenarios
  outputs/
    01-ai-search.actual.md   ← paste actual skill output here
    01-ai-search.golden.md   ← saved known-good output (for regression)
  run-tests.sh         ← assertion runner
  TESTING.md           ← this file

How to Run a Test

The skill runs inside Claude Code (a language model), so output is captured manually.

Step 1 — Run the skill

Open Claude Code in any project directory and invoke:

/pre-mortem "<paste contents of input.txt here>" <paste flags.txt here>

Example for scenario 01:

/pre-mortem "Add AI-powered semantic search to replace our current keyword search. Users type a query, the backend calls an embedding model, retrieves nearest-neighbor results from a vector DB, and ranks them by relevance score." --depth thorough --format report

Step 2 — Save the output

Copy the full skill output into:

tests/outputs/01-ai-search.actual.md

Step 3 — Run assertions

bash tests/run-tests.sh

Or for one scenario:

bash tests/run-tests.sh 01-ai-search

Step 4 — Read the results

  PASS  CONTAINS: What goes wrong
  PASS  CONTAINS_ANY: latency|response time|slow|performance
  FAIL  CONTAINS_ANY: rollback|fallback|keyword search
  ...
  Results: 1 scenarios, 1 failed, 0 skipped

A FAIL means the skill missed something. Read notes.txt for that scenario to understand whether it's critical (fix SKILL.md) or acceptable (lower priority).

Assertion Directives

Directive Meaning
CONTAINS: text Output must contain this exact string (case-insensitive)
CONTAINS_ANY: a|b|c Output must match at least one of these (regex OR)
NOT_CONTAINS: text Output must NOT contain this (catches hallucination)
MIN_SECTIONS: N Output must have at least N risk blocks
MAX_SECTIONS: N Output must have at most N risk blocks

Blank lines and # comments in assertions.txt are ignored.

Updating Golden Files

When you're happy with an actual output (it passed all assertions + human eval), save it as the golden file for regression testing:

bash tests/run-tests.sh --update 01-ai-search

This copies outputs/01-ai-search.actual.md → outputs/01-ai-search.golden.md.

Future model upgrades: run assertions on new output, then diff against golden to catch regressions in quality (not just structure).

Human Eval Checklist (L2)

Run this mentally after reading each actual output:

  • Risks are specific to the feature, not generic ("scope creep" tied to this system)
  • At least one uncomfortable truth surfaced (political, assumption, people risk)
  • Mitigations are actions, not advice ("run load test on throttled 3G" not "test performance")
  • Signal to watch is observable within days/weeks, not "monitor metrics"
  • Top threat is the actual top threat, not the easiest one to name
  • Immediate action is doable this week by one person

If 4+ items fail: revise SKILL.md prompt framing, especially Tone and Style section.

Adding a New Scenario

  1. Create tests/scenarios/NN-name/
  2. Write input.txt — real feature description, 1-5 sentences
  3. Write flags.txt — the flags to test (one line)
  4. Write assertions.txt — domain-specific checks (see existing scenarios for patterns)
  5. Write notes.txt — why this scenario, acceptance criteria, what failure looks like
  6. Run the skill, save output, run assertions

Good scenario coverage:

  • Happy path (clear feature, thorough depth) — scenario 01
  • Edge case: vague input — scenario 02
  • Flag behavior: --depth quick — scenario 03
  • Flag behavior: --depth deep + adversarial — scenario 04
  • Domain: security-heavy feature (public API)
  • Domain: data migration
  • Domain: user-facing UI change

Why Not Automated End-to-End?

Claude Code runs inside an interactive session. Fully automated invocation (piping input → capturing output) requires either:

  • A CI harness with Claude Code CLI + API key (possible, not set up here)
  • Mock output replay (defeats the purpose — we're testing the model's behavior)

For now: manual capture, automated assertion. This is the right tradeoff for a skill at this stage. Add CI when the skill is stable and you have a budget for API calls in CI.

Pass Rate Guidance

Assertion type Required pass rate
Structural (CONTAINS required sections) 100%
Domain-specific risk presence ≥ 80%
NOT_CONTAINS (no hallucination) 100%
Section count (MIN/MAX) ≥ 90%

If structural assertions fail: the skill broke — fix SKILL.md immediately. If domain assertions fail at 60-70%: improve the category prompts in Step 2. If NOT_CONTAINS fails: the skill is hallucinating — highest priority fix.