Regime-Aware Dynamic Multi-Factor Strategy

HMM regime detection + rolling Fama-French 5-factor regression + risk-aware portfolio optimization on the S&P 500 universe.
Monthly walk-forward backtest (2017–2025).

⚠️ Data Disclaimer
All raw data used in this project are sourced from publicly available datasets (Kenneth R. French Data Library and Yahoo Finance via yfinance). No proprietary or licensed terminal data (e.g., Wind, Bloomberg, CEIC) are included. For full traceability, a fetch_data.py script is provided to reproduce the dataset from primary sources. If you are adapting this framework to markets with restricted data (e.g., A-shares via Wind), you must replace the data layer with your own licensed feeds and must not redistribute raw tick data in this repository.

---

Architecture

graph LR
    A[FF5 Factors] --> B[HMM / GMM<br/>Regime Detection]
    C[SP500 Prices] --> D[Rolling FF5<br/>Regression]
    B --> E[Regime Factor<br/>Means μ]
    D --> F[Stock Betas β]
    E & F --> G[Score = β · μ]
    G --> H[Top-K Selection]
    H --> I[Min-Risk +<br/>Signal Tilt]
    I --> J[Portfolio]
    J --> K[Monthly<br/>Backtest]

Loading

---

Key Results

Metric	Signal-Tilt Strategy	S&P 500 Benchmark	Δ
CAGR	17.51%	13.19%	+4.3%
Sharpe	0.72	0.70	+0.02
Max Drawdown	-35.90%	-24.17%	-11.7%
Hit Rate	60.7%	—	—
Annual Turnover	676%	—	—

⚠️ Caveat: The backtest above assumes zero transaction costs. With 676% annual turnover, even modest trading costs (10–20 bps) materially erode the reported alpha. See Limitations.

---

Quick Start

Prerequisites

• Python >= 3.10
• Git

1. Clone & Install

git clone https://github.com/<your-username>/regime-aware-multi-factor.git
cd regime-aware-multi-factor
pip install -r requirements.txt

2. Data

All raw inputs are publicly available and included in data/ for convenience (total < 2 MB). You can also regenerate them from primary sources:

python fetch_data.py

Data Source	File	Size	License
Kenneth R. French	F-F_Research_Data_5_Factors_2x3.csv	52 KB	Free academic / non-commercial
Yahoo Finance (yfinance)	sp500_monthly_prices.xlsx	1.1 MB	Public API, CC0-equivalent for price data
Yahoo Finance (yfinance)	SP500_Benchmark_Returns.csv	4 KB	Public API
Pre-computed (GMM)	ff5_factors_with_regime.xlsx	16 KB	Derived from above

3. Run the Main Backtest

python main_update.py

This executes a monthly walk-forward backtest from 2017-02 to 2025-12 and produces:

• data/backtest_signal_tilt.xlsx — detailed NAV, returns, drawdowns, turnover
• Console summary of CAGR, Sharpe, Max Drawdown, etc.

4. Run Additional Scripts

# Sensitivity analysis (parameter grid heatmaps)
python sensitivity_analysis.py

# Visualize holdings for a specific month
python plot_monthly_holdings_example.py --month 2024-12 --save

# Export annual December position snapshots
python export_december_snapshots.py

---

Project Structure

.
├── data/
│   ├── F-F_Research_Data_5_Factors_2x3.csv   # FF5 monthly factors (Kenneth French)
│   ├── sp500_monthly_prices.xlsx             # S&P 500 constituents monthly close
│   ├── SP500_Benchmark_Returns.csv           # S&P 500 index monthly returns
│   ├── ff5_factors_with_regime.xlsx          # Pre-computed GMM regime labels
│   └── backtest_signal_tilt.xlsx             # ← Generated by main_update.py
├── Backtest_utils.py                         # Config dataclass, metrics, turnover
├── Cluster_analysis.py                         # GMM/HMM regime detection wrapper
├── HMM_regime.py                               # HMM-specific implementation
├── Regression.py                               # Rolling FF5 regression + clock alignment
├── Dynamic_stock_selection.py                  # Regime-aware scoring (β · μ)
├── Optimize_Portfolio.py                       # Risk + signal tilt optimization
├── Factor_covariance.py                        # Factor model covariance (Σ = BΩB' + Ψ)
├── Regime_forecast.py                          # RF/XGB next-regime predictor
├── Regime_alignment.py                         # Permutation-invariant label alignment
├── sensitivity_analysis.py                     # Parameter grid heatmaps
├── validation.py                               # Walk-forward regime validation
├── fetch_data.py                               # Data acquisition pipeline
├── export_december_snapshots.py                # Annual position snapshots
├── plot_monthly_holdings_example.py            # Single-month visualization
├── main_update.py                              # Main entry point
├── requirements.txt
├── LICENSE
└── README.md

---

Methodology

1. Dual-Clock Alignment

Price data are stamped at the first day of each month but represent the realized return of the preceding calendar month. align_ff5_clock() shifts factor timestamps forward by one month so that returns[t] and factors[t] correspond to the same economic period.

2. Regime Detection

HMM (GaussianHMM) with stickiness prior = 0.85
  → 3 latent regimes
  → Regime summary: mean of [Mkt-RF, SMB, HML, RMW, CMA] per regime

Alternative: Set CLUSTER_METHOD = "gmm" in main_update.py to use Gaussian Mixture Models.

3. Rolling FF5 Regression

For each stock $i$ over a 60-month lookback:

$$R_{i,t} - R_{f,t} = \alpha_i + \beta_{i,mkt},MktRF_t + \beta_{i,smb},SMB_t + \beta_{i,hml},HML_t + \beta_{i,rmw},RMW_t + \beta_{i,cma},CMA_t + \epsilon_{i,t}$$

Ridge regularization is supported via ridge_alpha in BacktestConfig.

4. Regime-Aware Scoring

Given current regime $s$ and its historical factor mean $\mu_s$:

$$Score_i = \beta_i \cdot \mu_s$$

Stocks with higher scores have factor exposures that align with the "winning" factors of the current regime. Top-$k$ (default 50) are selected for portfolio optimization.

5. Portfolio Optimization

$$ \min_w ; w'\Sigma w ; - ; \lambda_{signal} \cdot z_{signal}' w $$

Subject to:

• $\sum_i w_i = 1$ (fully invested)
• $0 \le w_i \le w_{max}$ (long-only, 10% cap)

Covariance $\Sigma$ is estimated via a factor model:

$$\Sigma = B , \Omega , B' + \Psi$$

• $B$: beta exposure matrix from FF5 regression
• $\Omega$: factor covariance (Ledoit-Wolf shrinkage, 12-month lookback)
• $\Psi$: diagonal idiosyncratic variance matrix with floor 1e-4

---

Configuration

All hyperparameters are centralized in BacktestConfig (Backtest_utils.py):

Parameter	Default	Description
top_k	50	Number of stocks selected by regime score
beta_lookback	60	Months for rolling FF5 regression
cov_lookback	12	Months for factor covariance estimation
w_max	0.10	Maximum position weight per stock
lambda_signal	0.005	Signal-tilt strength in optimization
cluster_method	"hmm"	Regime model: "hmm" or "gmm"
cost_bps	0.0	Transaction cost in basis points (⚠️ set > 0 for realistic backtests)

---

Limitations

1. Zero transaction costs: The published backtest uses cost_bps = 0. With 676% annual turnover, even 10 bps (bidirectional) would erode ~0.7–2.7% of annual alpha. Always re-run with realistic costs before drawing conclusions.
2. Regime forecasting is not wired into the main backtest: Regime_forecast.py (RF/XGB) predicts the next regime, but main_update.py currently uses the current (contemporaneous) regime. For production use, wire in the forecaster to avoid look-ahead bias.
3. Regime label alignment: Regime_alignment.py solves HMM/GMM permutation invariance but is not invoked in the main loop. Labels may swap across rolling windows.
4. Survivorship bias: The price universe reflects the S&P 500 as of the data pull date. Delisted or dropped constituents may not be present.
5. Short covariance lookback: cov_lookback = 12 months may be noisy during regime transitions.

---

Future Work

• Wire Regime_forecast.py into the main walk-forward loop
• Add realistic transaction cost modeling
• Implement regime-dependent beta estimation ($\beta_i^{(s)}$ rather than $\beta_i$)
• Introduce sector-neutrality constraints
• Extend to A-shares / CSI 300 with localized factor definitions

---

License

MIT — feel free to use, modify, and cite.

Acknowledgments

• Fama-French 5-factor data: Kenneth R. French Data Library
• S&P 500 price data: Yahoo Finance via yfinance
• Course project for MFIT6000C Algorithmic Trading, HKUST

---

Last updated: July 2026