Skip to content

docs: add payments case study #1 (success theater) - #27

Merged
MarouaBoud merged 1 commit into
mainfrom
docs/case-study-payments
Sep 14, 2026
Merged

MarouaBoud merged 1 commit into
mainfrom
docs/case-study-payments

Conversation

@MarouaBoud

Copy link
Copy Markdown
Member

The tool has been build-complete and CI-green for a while but had never been pointed at a target in a written-up, reproducible way. This is the first case study — the launch-track "prove it works in the wild" deliverable.

What's here

  • docs/case-studies/01-payments-success-theater.md — a 60-agent run against the payments world, graded B, written up entirely from the real report. Headline: 100% task success, 24% misuse — 42% of agents told to charge_customer called list_charges instead (and 37% the reverse, which charges the card). Exactly-once held under an 11-fault chaos storm; the 9-agent adversarial cohort was contained (0 destructive tools reached).
  • docs/case-studies/payments.stampede.yaml — the exact config, so the whole thing is reproducible with one command.
  • README.md — a "See it in action" section linking the study, replacing the old hypothetical misuse example with the real one.

Honesty notes (called out in the doc)

  • This is the deterministic --dry-run path, so all dollar figures are $0 — the "4.7× cost spread" is a token spread and the denial-of-wallet amplification is structural, not monetary. The misuse / persona / exactly-once / containment findings are real and brain-independent. The doc states this plainly per the toolkit's "honest over impressive" rule.
  • Every number is read straight from the run's JSON report; the reproduce commands were run verbatim and produce byte-identical output. The run_id is intentionally not quoted as a reproducibility anchor because it hashes the full config (output paths included) — the findings are the stable guarantee.

Docs-only; no code touched.

@MarouaBoud
MarouaBoud merged commit 0541fe1 into main Sep 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant