Why the kit is shaped the way it is. Everything here was learned the expensive way on a real pre-release review of a commercial desktop product; the specifics are generalized, the scars are real. Read this once before your first run; it is the difference between executing the prompt and understanding it.
A maximum-depth automated review ran every conventional phase: architecture audit, adversarial code review, thousands of simulated sessions, a persona panel, a verdict. Afterward, a human spent thirty minutes with the product and found four defects the whole run had missed.
None of the four was a bug in the usual sense. Each was an incoherence between two individually-correct parts: a counter that was technically right but stopped meaning what any user would read it to mean at exactly the moment they would look at it; a completion flow that silently discarded something the user was still working toward; a result screen that owed numbers it did not show; two features whose reasonable individual rules combined into the user's own annotations eating their actions. Every part matched its spec. The experience was still wrong.
Spec-verification cannot catch this class, because the spec agrees with the behavior. Code-derived expectations cannot catch it, because the code is the source of the expectation. Persona questionnaires cannot catch it, because answering questions is not using the product. The think-aloud phase (Phase 4) is the mechanization of that thirty minutes, and its three load-bearing rules exist precisely to break the circularity:
- The charter comes before the docs. Expectations are written down from category conventions and common sense before any internal document is read. Once you have read the docs, you can no longer be surprised on the user's behalf; the charter is pre-registration, and the timestamped ordering is what makes "expected vs observed" auditable instead of retrospective rationalization.
- The expectation precedes the action. An expectation written after the observation is a rationalization. Every action gets its EXPECT line first; the transcript is the audit trail.
- A doc citation never closes a finding. If the docs agree with the surprising behavior, the incoherence is designed in, which is worse, not fine. Only a rationale that would satisfy the user at that moment closes a finding.
It helps to name the layers explicitly. Mechanically correct: the code does what the code intends (unit tests live here). Specified correct: the code matches the written rules (spec tests, doc-sync audits live here). Experientially correct: the product matches what a reasonable user, taught by the product itself and by every comparable product they have used, expects at this moment. Standard testing pyramids cover the first two. Nearly everything users call "janky", "unfair", or "broken" in reviews lives in the third, and the third is exactly where a team's own documentation is useless as an oracle, because the docs describe intent, and the intent may be wrong.
The obvious objection to charter-first testing: the tester must learn the project to test it at all, yet must somehow not let the project's documentation shape its expectations. How can both hold at once?
The answer is that "context" is two different things wearing one name. Operational context is how to act: how to build, which commands drive the product, how to read state. The tester can absorb all of it without harm, because knowing how to click teaches nothing about what should happen next. Normative context is what is supposed to happen: specs, design docs, requirement lists, the test suite. That is the material under audit, and it is the only kind that poisons judgment. The docs having holes is not the danger; the danger is the docs filling the tester's head, because a head full of intent cannot be surprised on the user's behalf. Classify every piece of project information into one of the two kinds and the paradox dissolves: take everything operational, refuse everything normative.
With human testers this split is a discipline, and discipline leaks. With AI agents it is a construction: run the tester as a clean-room subagent whose entire context is a curated briefing pack (the user-facing corpus, a user-vocabulary manual for the drive surface, and the charter the tester writes itself), and contamination becomes impossible rather than resisted. The agent cannot be influenced by a document that was never in its prompt. This is one of the few places where AI testing is structurally stronger than human testing, not a cheaper approximation of it.
Two seams need guarding. The briefing pack itself can smuggle normative content, so it gets audited line by line against one question: could a real user know this? And mid-session the tester will need display facts it cannot see, so a question protocol connects the two roles: the tester asks, the Operator answers with the fact alone, no rationale, no "because". Every breach of the diet, including choosing to run single-mind at all, goes into a contamination ledger the report states plainly. The single-mind fallback (expectation written before acting, docs consulted only afterward and only for facts) is workable and was field-proven, but it is an approximation by ordering; the subagent form is an enforcement by construction.
The same failure keeps arriving in different clothes: the thing being tested supplied the expectation it is tested against.
- A test whose expected value was captured from the code's own output proves nothing; when implementation and test share one wrong assumption, both agree forever. Expected values come from the rules document, hand-verified fixtures, a reference implementation, or a brute-force oracle written from the rules. Type golden values from the document; never paste them from program output.
- A green suite is evidence about the code, not about the rules. Suites encode wrong rules and pass for years. When a rule is corrected, grep the suite for assertions encoding the old rule; when a user report contradicts a green suite, reproduce the report first and argue with the suite later.
- A harness that drives the product using the product's own components cannot judge the product with them. Sample sessions and re-verify outcomes by independent means; three genuinely independent derivations agreeing (shipped code, reference model, brute-force enumeration) is the gold standard.
- The product's own reasoning engine saying "stuck" or "impossible" is a claim, not a fact. In one measured run, sixteen of seventeen "nothing more can be done" reports were the engine being blind, and zero were genuinely dead ends. Separate "the assistant found nothing" from "nothing exists" everywhere: in your analysis and, just as important, in the product's own UI strings.
A related trap with its own name: certify along one path, consume along many. A validator that certifies a property by walking one canonical trajectory (greedily taking every forced step, always in the optimal order) may hold a hidden dependency on that ordering; real users traverse differently, and the property quietly fails for them. Drive certified properties along non-canonical orderings, and treat "more information produced fewer conclusions" as an anomaly class worth an explicit assertion.
The driver you write for a run is the newest, least-tested code in the room, and it will generate your most dramatic false findings. From one campaign: a suspected product hang that was the driver's own sweep loop spinning on a silently-refused action (the core answered the same position in one millisecond); doubled progression numbers caused by the driver re-applying credit on every render of the end screen; a fatal-outcome label printed for an action the product had correctly ignored; a "retry" the driver implemented with different semantics than the real client, invalidating a session's worth of expectations.
The guards are cheap and worth building into any driver on day one: a no-progress break in every loop (state unchanged after a full round with work still pending stops and prints what was blocked, instead of spinning); one-time effects latched at the terminal transition, never in a render path; outcome labels derived from state-before vs state-after, never from the action name; friendly refusals on bad driver input; and a terminal view that matches the real client's information content, so end-state findings are argued from full evidence. Before any dramatic finding leaves the room, freeze the exact state and probe each layer independently with timings. And when a harness bug does explain the observation, harvest it before discarding: the same silent behavior that trapped your automation traps human users.
One more: unstable output is not evidence. Any output line that varies between identical replays (cross-run state pollution, timestamps) is excluded from findings by policy, not case-by-case judgment.
The deepest root cause is rarely the right pre-release fix. In one run, a real incompleteness in a reasoning engine was correctly deferred to the next version, while the BLOCKER was a single UI string that, at those positions, told the user a provable falsehood contradicting the product's central public promise. The string was an under-an-hour fix; the engine was weeks. Rank consequences by embarrassment and promise-contradiction, separately from the internal defect's depth. Related habits: grade crashes by reachability (a boot-path crash with no in-app recovery is its own class); let independent rediscovery (two testers, or a tester and a persona, hitting the same root cause blind) outrank any single loud complaint; and keep a WORKS-LEAVE-IT ledger with the same rigor as the defect list, because unrecorded verified-good behavior gets re-litigated forever.
Numbers in a report get misread as facts, so earn them: a zero-event result over n trials is a bound (rule of three: below roughly 3/n at 95% confidence), never "cannot happen", and the public caveat stays in the copy even after measuring zero. A measured rate is compared to its configured target only alongside its sampling error. Latency figures captured under all-cores parallel load overstate wall time severalfold; user-facing timing claims come from a dedicated single-threaded run, and the contaminated columns get annotated, not deleted (deleting hides the mistake; trusting repeats it). When contaminated numbers have already reached testers, correct by a prominent note plus inline withdrawals and keep the originals verbatim; silently rewritten findings become zombies that resurface.
An AI run cannot feel input latency, judge visual fit, or hear the audio. The honest response
is not hedging every sentence; it is an economy: label every such judgment [PROXY], allow it
to state its mechanism ("plausible because the threshold is raw pixels"), and roll every one
into a ranked human checklist with pre-committed escalation rules. The run's job in those
domains is to make sure human attention is spent only where humans are irreplaceable. A
simulated cohort that finds every findable defect beforehand is what buys the human testers
the freedom to talk about feel. The reverse discipline matters equally: displayed values,
enabled controls, panel contents, and error texts are computed state, and claiming [PROXY]
blindness about derivable state is how experience defects survive.
- Every finding carries its own refutation attempt, written by someone who did not author it; several would-be blockers die correctly this way in every run.
- Findings are S/E/O: symptom with transcript citations, expectation with its source tier quoted, observation with any code fact labeled and cited, then impact and severity. Reject any finding lacking a citation on either side.
- Unmet wants are first-class output, phrased as the user's question verbatim. Today's unanswerable question ("how close is this to auto-completing?") is the early form of tomorrow's defect.
- Fix humility: findings list candidate fixes as a cost-annotated range and never prescribe the design; the cheapest tier that resolves the friction (copy < content < toggle < engine) gets named on every entry.
- The commissioning prompt is archived verbatim beside the reports, and the run closes by auditing itself against it, naming every skipped technique. A skipped pass that is not named is a lie of omission.
- The harness is a deliverable equal to the report: committed, seeded, rerunnable in one command, with the anomaly list as its standing regression set. One-shot review scripts rot, and rot makes the next release's regressions invisible.
- Subagents get written contracts (scope, allowed sources, output format, what they must not do); the independent-oracle rule extends to minds, so the agent verifying a behavior is never the one that read the implementation.
The kit does not attempt: penetration testing or exploit development against systems you do not own (the adversarial catalog is for your own product's robustness); load testing at production scale; compliance certification; or replacing human user research. It makes human testing radically cheaper by arriving at it with everything findable already found; it never substitutes for it.