Backtestable Crypto

Built and open-sourced by the team behind Millennium AI

Your backtest is probably lying to you. This tool proves it.

backtest-truth is a tiny, zero-dependency linter that catches the mistakes that make trading backtests look amazing and lose money live: lookahead bias, unrealistic costs, and overfitting.

It works with any backtesting engine (vectorbt, backtrader, freqtrade, your own loop) because it checks two things every engine has in common:

1. Your strategy code — a static scan for future-data leaks.
2. Your results — an "honesty scorecard" for fees, validation, and stats.

pip install backtest-truth

---

Why this exists

Most backtests that show a Sharpe of 4 and a smooth equity curve are not strategies — they are bugs. The usual culprits:

• using tomorrow's price to decide today's trade (shift(-1)),
• a centered moving average that peeks into the future,
• a threshold computed over the whole series (which "knows" the future),
• zero fees / zero slippage,
• one parameter set out of 5,000 that happened to win (overfitting),
• no out-of-sample test at all.

backtest-truth flags every one of these in seconds, with an explanation and a fix — so you find out before the market does.

---

1. Scan your strategy for lookahead bias

backtest-truth check examples/leaky_strategy.py

Static lookahead scan: 3 finding(s), 2 error(s)

ERROR   BT001 [examples/leaky_strategy.py:9]: shift() with a negative period pulls future data into the present.
        → Use a positive shift to look back; shift signals +1 before applying to returns.
ERROR   BT002 [examples/leaky_strategy.py:12]: rolling(center=True) averages future bars into the current one.
        → Drop center=True (default is trailing) for causal indicators.
WARNING BT003 [examples/leaky_strategy.py:15]: .max() over a full column uses the entire series, including the future.
        → Compute thresholds on a trailing/expanding window, or train-only.

A clean, causal strategy passes:

backtest-truth check examples/clean_strategy.py
# ✅ Static lookahead scan: no issues found.

False positive? Suppress one line explicitly:

adj = df["close"].shift(-1)   # bt:ignore BT001  (intentional, labelled target)

2. Score the honesty of your results

Describe your backtest in a small JSON file (or build the dict in code):

backtest-truth score examples/report_bad.json

Honesty score: 0/100   Grade: F — do not trust this backtest

Scorecard: 6 finding(s), 4 error(s)
ERROR   BT101 [fees_pct]: No trading fees modeled — every real fill costs money.
ERROR   BT201 [walk_forward]: No out-of-sample or walk-forward validation — results are in-sample only.
ERROR   BT303 [metrics.max_drawdown_pct]: Zero max drawdown — smells like lookahead.
ERROR   BT304 [oos_sharpe]: Out-of-sample Sharpe (0.3) collapses vs in-sample (4.0) — likely overfit.
...

Use it in code:

from backtest_truth import score_backtest

report = score_backtest({
    "fees_pct": 0.1, "slippage_pct": 0.05,
    "walk_forward": True, "regime_breakdown": True,
    "metrics": {"sharpe": 1.2, "trades": 400, "max_drawdown_pct": 18.0},
})
print(report.score, report.grade)   # 100 'A — credible'

3. Drop it into CI

backtest-truth exits non-zero on any error finding, so you can gate merges on it:

# .github/workflows/backtest-truth.yml
- run: pip install backtest-truth
- run: backtest-truth check strategies/

---

What it checks

Code	Meaning
BT001	Future indexing (shift(-1), series[i+1])
BT002	Centered rolling window
BT003	Whole-series statistic used as a threshold
BT004	fit() before train/test split
BT005	fit_transform() on the full series
BT101	Zero / missing fees
BT102	Zero / missing slippage
BT201	No out-of-sample / walk-forward
BT202	No per-regime breakdown
BT203	Parameter sweep without multiple-testing correction
BT301	Implausibly high Sharpe
BT302	Too few trades
BT303	Zero drawdown
BT304	Out-of-sample collapse (overfit)

Run backtest-truth rules for the full list.

---

The discipline behind it

This linter encodes a checklist we use in production. The principles:

1. No lookahead. A signal at bar t may only use data available at the close of t or earlier. Shift signals forward before applying to returns.
2. Realistic costs. Model fees and slippage. A strategy that only works at zero cost is not a strategy.
3. Out-of-sample is the only honest number. In-sample performance is a description of the past, not a prediction.
4. Plateau, not peak. Prefer parameter regions where neighbours also work; a lone spike is noise.
5. Report drawdown and trade count. Smooth curves and tiny samples lie.

⚠️ Not financial advice. backtest-truth checks the credibility of your measurement, not whether a strategy will make money. Passing all checks means your backtest is honest — not that your strategy is profitable.

Development

pip install -e ".[dev]"
pytest

License

MIT — see LICENSE. Built and open-sourced by the team behind Millennium AI.