feat(cli): ordo replay — replay recorded decisions against a rule change

[Pama-Lee]

The safety net for changing a rule: replay last week's real decisions and see which ones flip.

ordo replay <captured.jsonl> reads a JSONL of captured decisions — {rule_name, input, code, output}, the shape ordo-server's upcoming --capture-io writes (PR-2), and also what a business app's own decision log looks like — and re-runs every input through the current project ruleset.

What it reports

Each record lands in a bucket:

• consistent — same decision as captured
• flipped — code/output changed vs the captured baseline (with a diff, worded exactly like ordo test)
• errored — execution failed
• unknown-ruleset — named a rule not in this project
• replayed — input-only capture (no baseline to compare), still usable for --write-tests
• plus a count of skipped unparseable lines

Flags

• --write-tests — fixate the captured decisions as tests/<rule>.json regression cases (deduped by input). Production traffic → regression suite.
• --fail-on-flip — exit non-zero when any decision flips (CI gating); default is report-only, exit 0.
• --ruleset <name> — replay all records against one rule, ignoring per-record rule_name.
• source - reads JSONL from stdin; --json emits the full bucketed summary + per-record diffs.

Why it works so cleanly

A test case is {input, expect:{code, output}} — the same shape as a captured {input, code, output} decision. So capture → regression is a near-passthrough. Reuses the execution + diff logic of ordo test via a new pub(crate) test_runner::diff_result wrapper (keeps TestCase/TestExpectation private); compiles each distinct ruleset once and caches it.

Tests

tests/replay.rs drives the real binary through every bucket, the --write-tests → ordo test loop (incl. idempotent dedup), stdin (replay -), --fail-on-flip exit code, and --ruleset override. cargo test -p ordo-cli green (35 tests); clippy --all-targets -- -D warnings clean.

Verify locally

```bash
ordo init demo && cd demo
printf '%s\n' '{"rule_name":"loan-approval","input":{"amount":5000},"code":"APPROVED"}'
'{"rule_name":"loan-approval","input":{"amount":20000},"code":"APPROVED"}' > cap.jsonl # 2nd is a planted flip
ordo replay cap.jsonl # 1 consistent, 1 flipped (APPROVED → REJECTED)
ordo replay cap.jsonl --write-tests && ordo test
```

Part 1 of 2 — PR-2 adds ordo-server --capture-io to produce the JSONL this consumes.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
ordo_docs	Ready	Preview, Comment	Jul 5, 2026 10:28am
ordo-app	Error		Jul 5, 2026 10:28am