BUFN736 - Fall 2025
A comprehensive quantitative backtest pipeline for factor-based portfolio construction and performance evaluation. This project implements a complete workflow from data loading through portfolio construction to performance analysis.
This notebook contains a standalone quantitative backtest pipeline that:
- Loads stock data from WRDS (CRSP, Compustat) and factor data from Open Asset Pricing (OSAP)
- Filters the investment universe based on market cap, price, exchange, and share codes
- Computes industry-adjusted z-scores for factors
- Selects factors using Fama-MacBeth cross-sectional regressions
- Constructs long/short portfolios based on selected factors
- Evaluates out-of-sample performance using CAPM and Carhart 4-factor models
- Generates comprehensive visualizations and summary statistics
- CRSP Monthly Data: Stock returns, prices, market capitalization, exchange codes
- OSAP Factors: 110+ quantitative factors including:
- Value factors (BM, EP, SP, etc.)
- Profitability factors (GP, OperProf, roaq, etc.)
- Growth/Investment factors (AssetGrowth, Investment, etc.)
- Accruals, Capital Structure, Distress, Momentum, Liquidity, Risk, Analyst, and Size factors
- Compustat Fundamentals: Annual financial statement data for custom factor construction
- Minimum price filter ($5.00)
- Minimum market cap filter ($100M)
- Exchange restrictions (NYSE, AMEX, NASDAQ)
- Common stock only (share codes 10, 11)
- Industry-adjusted z-scoring using Fama-French 49 industry classification
- Multicollinearity filtering based on correlation thresholds
- Core Factor Set Selection: Determines a consistent set of factors based on overall coverage (average coverage ≥60%, present in ≥70% of years)
- Fama-MacBeth Estimation: Runs cross-sectional regressions using the same core factor set across all years (no floating factors)
- Factor selection requires both significant t-statistics (|t-stat| ≥ 1.5) and sufficient temporal coverage (≥15 years)
- In-sample period: 1985-2009
- Out-of-sample period: 2010-2024
- Long/short portfolios based on top/bottom deciles of factor scores
- Equal-weighted portfolios
- Rebalanced annually at year-end
- Raw performance metrics (returns, volatility, Sharpe ratio)
- CAPM alpha and beta
- Carhart 4-factor model (Market, SMB, HML, UMD)
- Information ratio
- Year-by-year performance breakdown
- Rolling statistics (12-month rolling Sharpe, volatility)
pandas
numpy
statsmodels
pathlib
openassetpricing (oap)
wrds
openpyxl (for Excel file reading)
matplotlib (for visualizations)
seaborn (for visualizations)
- WRDS Account: Required for accessing CRSP and Compustat data
- OSAP Access: Required for downloading factor signals
SIC_49_Industry.xlsx- SIC to Fama-French 49 industry mapping (must be in project root)ffdata.csv- Fama-French factor data (market, SMB, HML, UMD) (must be in project root)
Portfolio_Project/
├── main.ipynb # Main notebook with complete pipeline
├── README.md # This file
├── SIC_49_Industry.xlsx # SIC to FF49 industry mapping
├── ffdata.csv # Fama-French factors
├── data/ # Data directory (created automatically)
│ ├── crsp_monthly_raw.parquet # Raw CRSP monthly data
│ ├── annual_returns.parquet # Annual returns computed from CRSP
│ ├── december_snapshot.parquet # December snapshots for universe filtering
│ ├── compustat_annual.parquet # Compustat annual data
│ ├── osap_signals_december.parquet # OSAP factors (December snapshots)
│ └── factors_with_zscores_filtered.parquet # Processed factors with z-scores
└── output/ # Output directory (created automatically)
├── selected_factors.csv # Selected factors from Fama-MacBeth
├── selected_factors_summary.csv # Summary of selected factors
├── portfolio_returns_monthly.csv # Monthly portfolio returns
├── performance_results.csv # Detailed performance metrics
├── performance_summary_table.csv # Summary performance table
├── yearly_performance.csv # Year-by-year performance
├── fama_macbeth_coefficients.csv # Fama-MacBeth coefficients
├── fama_macbeth_statistics.csv # Fama-MacBeth statistics
├── factor_correlation_matrix.csv # Factor correlation matrix
└── plots/ # Visualization directory
├── cumulative_returns.png
├── monthly_returns.png
├── factor_performance.png
├── correlation_matrix.png
└── rolling_statistics.png
Key configuration parameters (defined in the notebook):
# Time periods
START_YEAR = 1985
END_YEAR = 2024
IN_SAMPLE_START = 1985
IN_SAMPLE_END = 2009
OUT_SAMPLE_START = 2010
OUT_SAMPLE_END = 2024
# Universe filters
MIN_PRICE = 5.0
MIN_MARKET_CAP = 100.0
VALID_EXCHANGES = [1, 2, 3] # NYSE, AMEX, NASDAQ
VALID_SHARE_CODES = [10, 11] # Common stock only
# Factor selection
INITIAL_FACTOR_COUNT = 110
FINAL_FACTOR_COUNT_MIN = 10
FINAL_FACTOR_COUNT_MAX = 20
MIN_T_STAT = 1.5
FACTOR_MIN_COVERAGE = 0.6
# Core factor set (for Fama-MacBeth)
CORE_MIN_AVG_COVERAGE = 0.6 # Minimum average coverage across years (60%)
CORE_MIN_YEARS_PRESENT = 0.7 # Minimum fraction of years present (70%)
MIN_YEARS_REQUIRED = 15 # Minimum years with valid data for final selection
# Portfolio construction
TOP_PERCENTILE = 0.10
BOTTOM_PERCENTILE = 0.10- Review and adjust configuration constants as needed
- Ensure required data files are in the project root
- Load CRSP monthly data from WRDS
- Load OSAP factor signals (December snapshots)
- Optionally load Compustat data for custom factors
- Data is saved to parquet files in
data/directory
- Apply price, market cap, exchange, and share code filters
- Create filtered dataset for analysis
- Compute custom factors from Compustat data
- Merge with existing OSAP factors
- Map SIC codes to Fama-French 49 industries
- Compute industry-adjusted z-scores for all factors
- Remove highly correlated factors
- Filter factors based on coverage requirements
- Determine Core Factor Set: Selects factors with average coverage ≥60% and present in ≥70% of years (1985-2009)
- Fixed Factor Set: Uses the same core factor set in every yearly cross-sectional regression (no year-by-year factor filtering)
- Missing Data Handling: Drops stocks (rows) with missing data, but keeps all core factors in the regression specification
- Run cross-sectional regressions year-by-year (1985-2009) with consistent RHS variables
- Factor Selection: Selects factors based on both t-statistics (|t-stat| ≥ 1.5) and minimum years requirement (≥15 years)
- Generate factor selection summary with core set information
- Construct long/short portfolios using selected factors (2010-2024)
- Rebalance annually at year-end
- Generate monthly portfolio returns
- Compute raw performance metrics
- Run CAPM and 4-factor regressions
- Calculate risk-adjusted returns
- Export all results to CSV files in
output/directory
- Generate performance charts
- Create correlation matrices
- Produce summary tables
Handles data loading from WRDS and OSAP:
load_crsp_monthly(): Load CRSP monthly stock dataload_osap_signals_yearly(): Load OSAP factors as yearly snapshotsload_compustat_annual(): Load Compustat annual data
Filters investment universe:
filter_universe(): Apply price, market cap, exchange filters
Computes industry-adjusted z-scores:
map_sic_to_industry(): Map SIC codes to FF49 industriescompute_all_zscores(): Compute z-scores for all factors
Runs Fama-MacBeth estimation with core factor set:
determine_core_factor_set(): Determines a consistent set of factors based on coverage criteria (avg coverage ≥60%, present in ≥70% of years)estimate(): Runs Fama-MacBeth estimation using the core factor set across all yearsselect_factors(): Selects factors based on both t-statistics (|t-stat| ≥ 1.5) and minimum years requirement (≥15 years)core_factors: Attribute storing the list of factors in the core set (accessible afterestimate())
Constructs portfolios:
construct_portfolio(): Build long/short portfolios
Evaluates performance:
evaluate(): Compute comprehensive performance metricsprint_results(): Display formatted results
performance_summary_table.csv: Summary of all performance metricsyearly_performance.csv: Year-by-year performance breakdownportfolio_returns_monthly.csv: Monthly portfolio returns
selected_factors.csv: Complete list of selected factorsselected_factors_summary.csv: Summary statistics for selected factorsfama_macbeth_coefficients.csv: Fama-MacBeth coefficients by yearfama_macbeth_statistics.csv: Fama-MacBeth statisticsfactor_correlation_matrix.csv: Correlation matrix of factors
All plots are saved in output/plots/:
- Cumulative returns over time
- Monthly returns distribution
- Factor performance comparison
- Factor correlation heatmap
- Rolling performance statistics
- The pipeline is designed to run end-to-end, but each step can be run independently if intermediate data files exist
- Data files are saved in parquet format for efficient storage and loading
- The notebook includes extensive error handling and progress reporting
- All dates and years are clearly marked for in-sample vs. out-of-sample periods
- Core Factor Set: The Fama-MacBeth estimation uses a fixed core factor set determined upfront, ensuring consistent regression specifications across all years. This prevents factors from "floating" in and out of the model based on year-by-year coverage, providing more stable and interpretable results.
- Factor Selection Criteria: Final factor selection requires both statistical significance (|t-stat| ≥ 1.5) and sufficient temporal coverage (≥15 years out of 25 in-sample years), ensuring selected factors are both economically significant and consistently available.
This project is for academic use in BUFN736 - Fall 2025.