Tune on point-in-time pairs; re-tuned TrendRider4h overfits

README.md

@@ -23,6 +23,7 @@ panel cannot start a live bot at all.
 | Path | What |
 |---|---|
 | `user_data/strategies/TrendRider4h.py` | 4h Donchian breakout trend-follower, BTC bear-market filter, ATR chandelier exit |
+| `user_data/strategies/TrendRider4hPIT.py` | TrendRider4h re-tuned on point-in-time pairs (candidate; own parameter file) |
 | `user_data/strategies/TrendDipBuyer1h.py` | 1h buy-the-dip inside an uptrend (mean reversion) |
 | `user_data/strategies/EmaRsiMomentum.py` | original 1h EMA-crossover + RSI strategy (hyperoptable) |
 | `user_data/strategies/MomentumRotation1d.py` | daily rotation into the 3 strongest coins by risk-adjusted momentum while BTC is in an uptrend (candidate) |
@@ -153,9 +154,17 @@ downloaded, so some survivorship bias remains.
 Tuning against OOS would make it in-sample too, and the numbers would stop
 meaning anything.
 
-Hyperopt saves the best parameters to `user_data/strategies/<Strategy>.json`,
-which every later backtest/dry-run picks up automatically. Delete that file to
-go back to the defaults in the `.py`.
+Hyperopt tunes on the point-in-time pairs at the start of the in-sample period,
+the same pairs the in-sample backtest uses. It never uses `config.json`'s live
+list, because that's today's top coins.
+
+It saves the best parameters to `user_data/strategies/<Strategy>.json`, which
+every later backtest and dry-run picks up automatically. Delete that file to go
+back to the defaults in the `.py`.
+
+To try new parameters without touching a strategy that's paper trading, tune a
+subclass with its own name instead. For example, `TrendRider4hPIT` is
+TrendRider4h tuned on point-in-time pairs.
 
 ### Go-live gates
 

RESEARCH.md

@@ -1,6 +1,6 @@
 # Research results
 
-**Generated:** 2026-10-11 by `python scripts/research.py gates`. Raw data is in
+**Generated:** 2026-10-15 by `python scripts/research.py gates`. Raw data is in
 `user_data/research/latest.json`, and the panel's **Research** tab shows the same numbers.
 The same run with a 65 USDT (≈100 AUD) account is in `latest-65.json`
 (`research.py gates --balance 65`). The case study PDF is built from both
@@ -31,7 +31,8 @@ The same run with a 65 USDT (≈100 AUD) account is in `latest-65.json`
 | Strategy | IS return | OOS return | OOS max DD | OOS profit factor | OOS trades | Walk-forward | Gates |
 |---|---|---|---|---|---|---|---|
 | **TrendRider4h** | +5.6% | **+17.3%** | 9.8% | 1.43 | 99 | 64% | **8 / 9** |
-| MomentumRotation1d (new) | +24.4% | +13.1% | **5.2%** | **1.59** | 63 | 57% | 7 / 9 |
+| MomentumRotation1d | +24.4% | +13.1% | **5.2%** | **1.59** | 63 | 57% | 7 / 9 |
+| TrendRider4hPIT (re-tuned) | **+45.6%** | +13.8% | 9.9% | 1.33 | 98 | 50% | 7 / 9 |
 | EmaRsiMomentum | +7.7% | +5.1% | 18.1% | 1.13 | 74 | 43% | 6 / 9 |
 | TrendDipBuyer1h | −8.0% | −17.8% | 19.2% | 0.73 | 300 | 36% | 4 / 9 |
 
@@ -59,6 +60,28 @@ the next monthly re-run will settle it either way.
 MomentumRotation1d fails the sample-size gate (63 trades) and the walk-forward
 gate (57% against 60%).
 
+## Re-tuning TrendRider4h on point-in-time pairs
+
+TrendRider4h's parameters were tuned on 15 hand-picked pairs, chosen with
+today's knowledge. `research.py hyperopt` also had a bug: it tuned on
+`config.json`'s live list, today's top coins. It now tunes on the
+point-in-time universe at the start of the in-sample period.
+
+TrendRider4hPIT is the same strategy with its own parameter file. It was tuned
+that way for 300 trials (Calmar objective), using in-sample data only:
+
+| | Breakout | ADX | Trailing stop | Exit window | IS | OOS | OOS PF | Walk-forward |
+|---|---|---|---|---|---|---|---|---|
+| TrendRider4h | 15 | 23 | 4.9 × ATR | 6 | +5.6% | **+17.3%** | **1.43** | **64%** |
+| TrendRider4hPIT | 28 | 22 | 3.9 × ATR | 5 | **+45.6%** | +13.8% | 1.33 | 50% |
+
+**Re-tuning made it worse where it counts.** In-sample return went up eightfold;
+out of sample it fell by 3.5 points, and its out-of-sample quarters went from
+3 of 6 profitable to 2 of 6. That is overfitting: the tuning fitted the
+in-sample period rather than finding a better strategy. TrendRider4h stays the
+lead candidate; TrendRider4hPIT is kept as evidence. The 65 USDT run gives
+the same picture (+17.5% against +14.0%).
+
 ## Small accounts (≈100 AUD)
 
 Same gates, starting with 65 USDT (about 100 AUD at an assumed 0.65 USD per AUD):
@@ -67,6 +90,7 @@ Same gates, starting with 65 USDT (about 100 AUD at an assumed 0.65 USD per AUD)
 |---|---|---|---|---|---|---|
 | TrendRider4h | 65 → 76.35 | +17.5% | +17.3% | 9.7% | 99 | 8 / 9 |
 | MomentumRotation1d | 65 → 74.08 | +14.0% | +13.1% | 7.1% | 64 | 7 / 9 |
+| TrendRider4hPIT | 65 → 74.07 | +14.0% | +13.8% | 9.8% | 98 | 7 / 9 |
 | EmaRsiMomentum | 65 → 68.29 | +5.1% | +5.1% | 18.1% | 74 | 6 / 9 |
 | TrendDipBuyer1h | 65 → 53.49 | −17.7% | −17.8% | 19.1% | 300 | 4 / 9 |
 
@@ -198,8 +222,8 @@ The gap between the IS results is the most important finding:
 - **Survivorship bias.** Binance doesn't serve data for delisted coins, so
   coins that died between 2023 and 2026 are missing from the candidate pool.
 - **Tuning on hand-picked pairs.** TrendRider4h's parameters were tuned on the
-  hand-picked 15, not on point-in-time universes. Re-tuning on point-in-time IS
-  universes is a valid next step. It must not touch OOS.
+  hand-picked 15. Re-tuning on point-in-time pairs (TrendRider4hPIT) did worse
+  out of sample, so the original parameters stay.
 - **No modelled slippage.** Backtests fill at the candle price plus fees.
   Paper trading is where slippage shows up.
 
@@ -212,8 +236,8 @@ The gap between the IS results is the most important finding:
      the same weeks.
 2. **Re-run `research.py gates` monthly.** New data adds OOS trades, and the
    100-trade gate will be settled honestly.
-3. **Optionally re-hyperopt TrendRider4h** on point-in-time IS universes. Treat
-   it as a new candidate and judge it only on OOS.
+3. **Don't re-tune TrendRider4h again on this in-sample period.** The
+   point-in-time re-tune overfitted. More tuning would just fit it harder.
 4. **Keep MomentumRotation1d as a candidate, not a replacement.** It needs more
    OOS trades and a better walk-forward record. Don't re-tune its lookback on
    the OOS period.

scripts/case_study.py

@@ -338,6 +338,27 @@ def passed(s):
                         f"that: it made only {pct(wf_r[up]['profit_pct'])} in that strongest quarter (and {pct(wf_r[down]['profit_pct'])} "
                         "in the worst one). Its rally-capture rationale is not supported by these results.")
 
+    retune = strategies.get("TrendRider4hPIT")
+    retune_section = ""
+    if retune is not None and "TrendRider4h" in strategies:
+        orig = strategies["TrendRider4h"]
+        rows = [[esc(name), pct(s["is"]["profit_pct"]), pct(s["oos"]["profit_pct"]), f'{s["oos"]["profit_factor"]:.2f}',
+                 pct(s["walkforward"]["pct_profitable"], 0, sign=False), f"{passed(name)} / {n_gates}"]
+                for name, s in (("TrendRider4h", orig), ("TrendRider4hPIT", retune))]
+        better = retune["oos"]["profit_pct"] > orig["oos"]["profit_pct"]
+        retune_section = (
+            "<h3>5.7 Re-tuning on point-in-time pairs</h3>"
+            + table(["Version", "IS return", "OOS return", "OOS PF", "WF profitable", "Gates"], rows,
+                    caption="Table 8. TrendRider4h tuned on hand-picked pairs vs re-tuned on point-in-time in-sample pairs (300 trials).")
+            + "<p>TrendRider4h's parameters were originally tuned on a hand-picked pair list, and the tuning command itself "
+            "used today's live pair list. Both leak hindsight. Re-tuning on the point-in-time universe "
+            f"raised the in-sample return from {pct(orig['is']['profit_pct'])} to {pct(retune['is']['profit_pct'])}, "
+            f"but the out-of-sample return {'rose' if better else 'fell'} from {pct(orig['oos']['profit_pct'])} to "
+            f"{pct(retune['oos']['profit_pct'])}"
+            + ("." if better else ", and the walk-forward record weakened. A large in-sample gain that doesn't carry over to new "
+               "data is the signature of overfitting, so the original parameters stay.")
+            + "</p>")
+
     verdict = (f"{len(eligible)} of {len(strategies)} strategies passed every gate" if eligible
                else f"none of the {len(strategies)} strategies passed every gate, so none is eligible for real money")
     rot_text = ""
@@ -380,9 +401,10 @@ def passed(s):
 <p>Marketing for trading bots tends to report backtests, which are easy to make look good. This study asks whether a strategy still
 shows an edge when it is tested the way a sceptical reviewer would test it, and whether that holds for a very small account.
 Four questions guide it:</p>
+
 <ol>
 <li><b>RQ1.</b> How is the 2026 automated-trading market structured, and what risks does it carry for retail users?</li>
-<li><b>RQ2.</b> Does any of four rule-based strategies show an edge that survives point-in-time, out-of-sample testing and nine go-live gates?</li>
+<li><b>RQ2.</b> Does any of {len(strategies)} rule-based strategies show an edge that survives point-in-time, out-of-sample testing and nine go-live gates?</li>
 <li><b>RQ3.</b> Can the system trade an account of about 100 AUD, and with what results?</li>
 <li><b>RQ4.</b> Does taking more risk per trade produce proportionally more profit?</li>
 </ol>
@@ -510,6 +532,7 @@ def passed(s):
 were chosen on in-sample data only.</li>
 <li><b>EmaRsiMomentum:</b> a 1-hour EMA crossover with RSI, ADX and 4-hour EMA200 filters and a take-profit ladder.</li>
 <li><b>TrendDipBuyer1h:</b> buys 1-hour pullbacks inside an established uptrend (mean reversion).</li>
+{'<li><b>TrendRider4hPIT</b> (new): TrendRider4h with its parameters re-tuned on point-in-time in-sample pairs, to test whether the original tuning on hand-picked pairs was flattering it.</li>' if retune else ''}
 </ul>
 
 <h2 class="pagebreak">5. Results</h2>
@@ -552,6 +575,7 @@ def passed(s):
        caption="Table 7. MomentumRotation1d in-sample return by lookback (top 3, 65 USDT start, measured before the warm-up fix).")}
 <p>The 14-day setting was the in-sample best, but neighbouring settings fell sharply, from +37% at 14 days to −4% at 21 days. A
 peak this narrow suggests part of the in-sample result is luck, so its out-of-sample figure should be read with that caution.</p>
+{retune_section}
 
 <h2>6. Discussion</h2>
 <p><b>RQ1.</b> The market sells infrastructure rather than proven edges. Fixed fees, curve-fitted marketplace strategies and centralised

scripts/research.py

@@ -322,8 +322,12 @@ def bias(strategies: list[str]) -> dict:
 
 
 def hyperopt(strategy: str, epochs: int, spaces: list[str], loss: str) -> None:
+    # Tune on the same point-in-time pairs the in-sample backtest uses. Without
+    # --pairs Freqtrade would use config.json's live list, i.e. today's top
+    # coins by volume, which leaks the future into the tuning.
     cmd = [*FREQTRADE, "hyperopt", *config_args(), "--strategy", strategy,
            "--timerange", tr(*IS_RANGE), "--enable-protections",
+           "--pairs", *universe_at(IS_RANGE[0]),
            "--hyperopt-loss", loss, "--spaces", *spaces, "--epochs", str(epochs),
            "--min-trades", "60", "-j", "-2", "--random-state", "42"]
     print(" ".join(cmd))

tests/test_universe.py

@@ -66,3 +66,17 @@ def test_recursive_gate_ignores_fewer_startup_candles_than_the_strategy_uses():
     # 5.978% is at 199 candles, below the 360 the strategy always gets.
     assert research.worst_recursive_deviation(RECURSIVE_TABLE) == 0.392
     assert research.worst_recursive_deviation("no table here") is None
+
+
+def test_hyperopt_tunes_on_point_in_time_pairs(monkeypatch):
+    """Without --pairs Freqtrade would tune on config.json's live list (today's
+    top coins), leaking the future into the parameters."""
+    import research
+
+    calls = []
+    monkeypatch.setattr(research, "universe_at", lambda when: ("AAA/USDT", "BBB/USDT") if when == research.IS_RANGE[0] else ())
+    monkeypatch.setattr(research.subprocess, "run", lambda cmd, **kw: calls.append(cmd))
+    research.hyperopt("TrendRider4hPIT", 5, ["buy"], "CalmarHyperOptLoss")
+    cmd = calls[0]
+    assert cmd[cmd.index("--pairs") + 1:cmd.index("--pairs") + 3] == ["AAA/USDT", "BBB/USDT"]
+    assert cmd[cmd.index("--timerange") + 1] == research.tr(*research.IS_RANGE)