Paper vs replay reconciliation: the last go-live gate

.gitignore

@@ -21,5 +21,6 @@ all_usdt.json
 reports/
 user_data/audit/
 user_data/risk_state.json
+*.reconcile.json
 .pytest_cache/
 .ruff_cache/

README.md

@@ -29,8 +29,9 @@ panel cannot start a live bot at all.
 | `user_data/config.private.json` | API login, JWT secret, exchange keys - **never share/commit** (loaded automatically via `add_config_files`) |
 | `panel/` | control panel (FastAPI + plain JS) |
 | `scripts/research.py` | IS/OOS backtests, walk-forward, fee stress, bias checks, go-live gates |
-| `ridgeline/` | shared package: risk limits (`risk.py`, `freqtrade_risk.py`), audit trail, tear-sheet reports, alerts |
+| `ridgeline/` | shared package: risk limits (`risk.py`, `freqtrade_risk.py`), audit trail, tear-sheet reports, paper-vs-replay reconciliation, alerts |
 | `scripts/report.py`, `scripts/watchdog.py` | tear-sheet generator; health/risk watchdog with Telegram alerts |
+| `scripts/reconcile.py` | the last go-live gate: compares a paper session with a replay of the same weeks |
 | `tests/`, `.github/workflows/ci.yml` | test suite (risk, audit, lookahead, universe, panel, hygiene) run by GitHub Actions |
 | `Dockerfile`, `docker-compose.yml` | 24/7 deployment: bot + monitor-mode panel + watchdog |
 | `docs/OPERATIONS.md` | runbook: risk limits, kill switch, alerts, VPS deployment, incidents, monthly routine |
@@ -150,6 +151,36 @@ Then it must paper-trade for **4+ weeks** with results close to a replay of
 the same weeks. If nothing passes, the answer is "don't go live yet" - not
 "loosen the gates".
 
+### Paper vs replay (the last gate)
+
+Use the panel's **Paper trading → Paper vs replay** card, or run:
+
+```bash
+python scripts/reconcile.py <session-id>
+```
+
+It replays the exact period the session ran, with the same strategy, pairs and
+starting balance. Then it matches the trades one by one (same pair, entries
+within 1.5 candles). The session is "close to the replay" when all of these hold:
+
+- it ran for at least 28 days
+- at least 80% of trades happened in both
+- realised return is within 2 percentage points of the replay
+- fills cost at most 0.3% per side compared with the replay's candle prices
+
+The result also splits the return gap into three parts:
+- trades in both, which shows execution differences
+- paper-only trades
+- replay-only trades
+
+It flags anything the replay can't reproduce:
+- manual pauses, resumes and flattens
+- drawdown halts
+- strategy code or parameters that changed mid-session
+
+The Research tab shows the newest result for each strategy.
+See [docs/OPERATIONS.md](docs/OPERATIONS.md#paper-vs-replay).
+
 Plain backtests still work as usual:
 
 ```bash
@@ -229,6 +260,7 @@ The tests cover:
 - audit tamper detection
 - a lookahead check: signals must not change when future candles are removed
 - point-in-time universe selection
+- paper-vs-replay trade matching, gap breakdown and checks
 - the panel's dry-run guard and kill-switch sequence
 - repo hygiene: no secrets, and no live config in git
 

RESEARCH.md

@@ -113,6 +113,10 @@ The gap between the IS results is the most important finding:
 ## Next steps
 1. **Paper-trade TrendRider4h for 4+ weeks** from the panel. Use Reset wallet
    and pick TrendRider4h. `config.json` now holds today's point-in-time top 25.
+   - Then run **Paper vs replay** on that session, or
+     `python scripts/reconcile.py <session>`.
+   - That's the last gate: the paper results must be close to a replay of
+     the same weeks.
 2. **Re-run `research.py gates` monthly.** New data adds OOS trades, and the
    100-trade gate will be settled honestly.
 3. **Optionally re-hyperopt TrendRider4h** on point-in-time IS universes. Treat

docs/OPERATIONS.md

@@ -10,6 +10,7 @@ How to run ridgeline 24/7, watch it, and act when something goes wrong.
 - [Deploying on a VPS (Docker)](#deploying-on-a-vps-docker)
 - [Incident checklist](#incident-checklist)
 - [Monthly routine](#monthly-routine)
+- [Paper vs replay](#paper-vs-replay)
 - [Reports](#reports)
 
 ---
@@ -161,11 +162,73 @@ git pull && docker compose up -d --build
 python scripts/research.py universe --download --write-config   # refresh data + live pair list
 python scripts/research.py gates                                 # re-test every strategy on new OOS data
 python scripts/report.py --db user_data/paper/<session>.sqlite --month YYYY-MM
+python scripts/reconcile.py <session>                            # paper vs replay of the same weeks
 ```
 
 If you change the pair list or the strategy, restart the bot: `docker compose
 restart bot`, or Reset in the panel.
 
+## Paper vs replay
+
+The last go-live gate: 4+ weeks of paper trading whose results are close to a
+replay (backtest) of the same weeks. Run it from **Paper trading → Paper vs
+replay** in the panel, or from the command line:
+
+```bash
+python scripts/reconcile.py <session-id>                       # a panel session
+python scripts/reconcile.py --db user_data/tradesv3.docker.sqlite --log user_data/logs/bot.log   # the Docker bot
+```
+
+**What it does.**
+1. Tops up candle data for the session's pairs.
+2. Replays the session's period, starting at the candle its first decision
+   used and using the same strategy, pairs, starting balance and max open trades.
+3. Matches trades: same pair, entries at most 1.5 candles apart.
+
+The result is saved next to the session's database as
+`<session>.reconcile.json`. The Research tab shows the newest result for each
+strategy.
+
+**Where the settings come from.** A session started from the panel records
+its strategy, pair list, balance, start time and a fingerprint of the
+strategy code in `<session>.meta.json`. That matters because `config.json`'s
+pair list changes monthly. Older sessions and the Docker bot fall back to the
+bot's log and database. `--strategy`, `--pairs`, `--balance`, `--start` and
+`--end` override any of them.
+
+**Checks.** All must pass:
+
+| Check | Target | Why |
+|---|---|---|
+| Length | ≥ 28 days | Long enough to see a few trades and some market change |
+| Trades in both | ≥ 80% | Signals fire the same way live as in the backtest |
+| Return gap | ≤ 2 percentage points | The backtest's results are reachable |
+| Fill cost | ≤ 0.3% per side | Fills land near the candle prices the backtest assumes |
+
+Fill cost is the average difference between paper and replay prices. It's
+positive when paper paid more on entry or received less on exit. Exits only
+count when both sides left for the same reason.
+
+**Reading a gap.** The return gap is split into three parts:
+- **trades in both**: execution, meaning fills, stop timing and exit prices
+- **paper-only trades**
+- **replay-only trades**
+
+One-sided trades usually come from:
+- an entry order that didn't fill within its 10-minute timeout
+- a different equity, so a risk limit or the open-trade cap blocked one side
+- signals right at the start or end of the period
+
+The result notes things the replay can't reproduce, so check those first:
+- manual pause, resume or flatten (from the audit trail)
+- drawdown halts, which need a manual resume live but resume automatically in backtests
+- strategy code or parameters changed during the session
+- paper trades newer than the latest closed candle
+
+**If it fails.** Don't loosen the targets. Find out why the two diverged. If
+the backtest assumptions are wrong (for example, real fills are consistently
+worse), fix the research model and re-run the gates before trying again.
+
 ## Reports
 
 Two ways to get a tear sheet:

panel/app.py

@@ -12,8 +12,9 @@
 
 from panel.bot import MONITOR_MODE, PaperBot, PaperBotError
 from panel.common import USERDIR, list_strategies, load_json
-from panel.replay import REPLAY_DIR, ReplayJobs
+from panel.replay import REPLAY_DIR, BacktestError, ReplayJobs, SessionReplays
 from ridgeline import audit
+from ridgeline.reconcile import session_info
 from ridgeline.report import build_tearsheet, load_trades_backtest, load_trades_db, session_balance
 
 STATIC = Path(__file__).parent / "static"
@@ -23,6 +24,7 @@
 
 bot = PaperBot()
 replays = ReplayJobs()
+session_replays = SessionReplays()
 
 
 class PaperStart(BaseModel):
@@ -149,11 +151,30 @@ def _valid_month(month: Optional[str]) -> Optional[str]:
     return month or None
 
 
-@app.get("/api/report", response_class=HTMLResponse)
-def paper_report(session: str, month: Optional[str] = None):
-    db = USERDIR / "paper" / f"{_valid_id(session)}.sqlite"
+def _session_db(session_id: str) -> Path:
+    db = USERDIR / "paper" / f"{_valid_id(session_id)}.sqlite"
     if not db.exists():
         raise HTTPException(404, "No such paper session")
+    return db
+
+
+@app.post("/api/paper/sessions/{session_id}/reconcile")
+def reconcile_start(session_id: str):
+    db = _session_db(session_id)
+    try:
+        return session_replays.submit(session_info(db))
+    except (ValueError, BacktestError) as e:
+        raise HTTPException(409, str(e))
+
+
+@app.get("/api/paper/sessions/{session_id}/reconcile")
+def reconcile_get(session_id: str):
+    return session_replays.get(session_id, _session_db(session_id)) or {"status": "none"}
+
+
+@app.get("/api/report", response_class=HTMLResponse)
+def paper_report(session: str, month: Optional[str] = None):
+    db = _session_db(session)
     try:
         trades = load_trades_db(db)
         return build_tearsheet(trades, session_balance(db), title="Ridgeline - paper trading report",
@@ -202,6 +223,25 @@ def replay_history():
     return replays.history()
 
 
+def _latest_reconciliations() -> dict:
+    """Newest paper-vs-replay result per strategy (panel sessions and the
+    Docker bot's database), for the last go-live gate."""
+    out = {}
+    for path in [*(USERDIR / "paper").glob("*.reconcile.json"), *USERDIR.glob("*.reconcile.json")]:
+        r = load_json(path, None) or {}
+        if r.get("status") != "done":
+            continue
+        strategy = r["session"]["strategy"]
+        if strategy not in out or r["generated"] > out[strategy]["generated"]:
+            out[strategy] = {"session_id": r["session"]["session_id"], "generated": r["generated"],
+                             "days": r["period"]["days"], "close_to_replay": r["close_to_replay"],
+                             "passed": sum(c["pass"] for c in r["checks"]), "checks": len(r["checks"])}
+    return out
+
+
 @app.get("/api/research")
 def research():
-    return load_json(USERDIR / "research" / "latest.json", None) or {}
+    report = load_json(USERDIR / "research" / "latest.json", None) or {}
+    if report:
+        report["paper_checks"] = _latest_reconciliations()
+    return report

panel/bot.py

@@ -29,6 +29,7 @@
 
 from panel.common import FREQTRADE, ROOT, USERDIR, config_args, load_json, merged_config
 from ridgeline import audit
+from ridgeline.reconcile import strategy_fingerprint
 from ridgeline.risk import RiskLimits, build_state, limits_summary, load_halt, set_halt
 
 MONITOR_MODE = os.environ.get("PANEL_MODE", "").lower() == "monitor"
@@ -110,6 +111,18 @@ def start(self, balance: float, strategy: str) -> dict:
         }, indent=2))
         args = [*config_args(), "-c", str(session_cfg)]
         self._verify_dry_run(args)
+        # What this session runs, for reconciling it with a replay later:
+        # config.json's pair list changes monthly, and code can change mid-session.
+        cfg = merged_config()
+        (PAPER_DIR / f"{session_id}.meta.json").write_text(json.dumps({
+            "session_id": session_id,
+            "strategy": strategy,
+            "balance": balance,
+            "pairs": cfg.get("exchange", {}).get("pair_whitelist", []),
+            "max_open_trades": cfg.get("max_open_trades"),
+            "started": datetime.now(timezone.utc).isoformat(timespec="seconds"),
+            "code": strategy_fingerprint(strategy),
+        }, indent=2))
 
         log_path = PAPER_DIR / f"{session_id}.log"
         cmd = [*FREQTRADE, "trade", *args, "--strategy", strategy, "--logfile", str(log_path)]
@@ -305,6 +318,7 @@ def sessions(self) -> list[dict]:
         rows = []
         for db in sorted(PAPER_DIR.glob("*.sqlite"), reverse=True):
             cfg = load_json(db.with_suffix(".config.json"), {}) or {}
+            meta = load_json(db.with_suffix(".meta.json"), {}) or {}
             rows.append({"session_id": db.stem, "balance": cfg.get("dry_run_wallet"),
-                         "size_kb": round(db.stat().st_size / 1024, 1)})
+                         "strategy": meta.get("strategy"), "size_kb": round(db.stat().st_size / 1024, 1)})
         return rows

panel/replay.py

@@ -1,6 +1,8 @@
 # Fast historical replay: run a Freqtrade backtest with a chosen starting
 # balance and date range, then reduce the result to what the UI and the
 # research scripts need (metrics, equity curve vs buy-and-hold, trades).
+# Also replays a paper session's exact period to reconcile it against paper
+# trading (ridgeline/reconcile.py has the comparison itself).
 
 import json
 import re
@@ -15,8 +17,12 @@
 import pandas as pd
 
 from panel.common import DATADIR, FREQTRADE, USERDIR, config_args
+from ridgeline import audit
+from ridgeline.reconcile import SessionInfo, load_paper_trades, load_replay_trades, reconcile, strategy_timeframe
 
 REPLAY_DIR = USERDIR / "replays"
+# Daily EMA200 filters need ~200 days of history before the first candle.
+WARMUP_DAYS = 250
 
 
 class BacktestError(RuntimeError):
@@ -67,6 +73,92 @@ def run_backtest(
     return summarize(outdir, strategy)
 
 
+def download_data(pairs: list[str], since: pd.Timestamp, log_path: Path,
+                  timeframes: tuple[str, ...] = ("1h", "4h", "1d")) -> None:
+    """Top up candle data for these pairs to the latest closed candle."""
+    cmd = [*FREQTRADE, "download-data", *config_args(), "--pairs", *pairs,
+           "--timeframes", *timeframes, "--timerange", f"{since:%Y%m%d}-"]
+    with open(log_path, "w", encoding="utf-8", errors="replace") as log:
+        proc = subprocess.run(cmd, stdout=log, stderr=subprocess.STDOUT)
+    if proc.returncode != 0:
+        tail = log_path.read_text(encoding="utf-8", errors="replace").splitlines()[-15:]
+        raise BacktestError("Data download failed:\n" + "\n".join(tail))
+
+
+def replay_window(info: SessionInfo) -> tuple[str, pd.Timestamp]:
+    """The strategy's timeframe and the replay's first candle: the one the
+    session's first decision was based on. Raises if it hasn't closed yet."""
+    timeframe = strategy_timeframe(info.strategy)
+    start = info.start.floor(pd.Timedelta(timeframe))
+    first_close = start + pd.Timedelta(timeframe)
+    if first_close > pd.Timestamp.now(tz="UTC"):
+        raise BacktestError(f"Nothing to replay yet: the session's first {timeframe} candle closes at "
+                            f"{first_close:%Y-%m-%d %H:%M} UTC.")
+    return timeframe, start
+
+
+def replay_session(info: SessionInfo, outdir: Optional[Path] = None, download: bool = True) -> dict:
+    """Replay a paper session's exact period (same strategy, pairs, balance)
+    and reconcile it with what paper trading did."""
+    timeframe, start = replay_window(info)
+    outdir = Path(outdir or REPLAY_DIR / f"session-{info.session_id}")
+    outdir.mkdir(parents=True, exist_ok=True)
+    if download:
+        # BTC/USDT feeds the strategies' market-regime filter even when it isn't traded.
+        download_data(sorted({*info.pairs, "BTC/USDT"}), start - pd.Timedelta(days=WARMUP_DAYS),
+                      outdir / "download.log")
+    run_backtest(info.strategy, f"{int(start.timestamp())}-{int(info.end.timestamp())}", info.balance,
+                 pairs=info.pairs, max_open_trades=info.max_open_trades, outdir=outdir)
+    replay, meta = load_replay_trades(outdir)
+    return reconcile(load_paper_trades(info.db), replay, info=info, replay_end=meta["end"],
+                     timeframe=timeframe, events=audit.tail(100_000))
+
+
+def reconcile_path(db: Path) -> Path:
+    """Where a session's latest reconciliation is kept: next to its database."""
+    return Path(db).with_suffix(".reconcile.json")
+
+
+class SessionReplays:
+    """Paper-vs-replay reconciliations for the panel, one at a time. The result
+    is written next to the session's database, so it outlives the panel."""
+
+    def __init__(self):
+        self._pool = ThreadPoolExecutor(max_workers=1)
+        self._lock = threading.Lock()
+        self._running: dict[str, dict] = {}
+
+    def submit(self, info: SessionInfo) -> dict:
+        replay_window(info)  # fail fast if there's nothing to replay yet
+        with self._lock:
+            if info.session_id in self._running:
+                return self._running[info.session_id]
+            job = {"status": "running", "session_id": info.session_id,
+                   "started": datetime.now(timezone.utc).isoformat(timespec="seconds")}
+            self._running[info.session_id] = job
+        self._pool.submit(self._run, info)
+        return job
+
+    def _run(self, info: SessionInfo) -> None:
+        try:
+            result = replay_session(info)
+        except Exception as e:  # surfaced to the UI as-is
+            result = {"status": "error", "error": str(e), "session": info.to_dict(),
+                      "generated": datetime.now(timezone.utc).isoformat(timespec="seconds")}
+        try:
+            reconcile_path(info.db).write_text(json.dumps(result, indent=2, default=str))
+        finally:
+            with self._lock:
+                self._running.pop(info.session_id, None)
+
+    def get(self, session_id: str, db: Path) -> Optional[dict]:
+        with self._lock:
+            if session_id in self._running:
+                return self._running[session_id]
+        path = reconcile_path(db)
+        return json.loads(path.read_text()) if path.exists() else None
+
+
 def summarize(outdir: Path, strategy: str) -> dict:
     from freqtrade.data.btanalysis import load_backtest_stats