Portfolio Optimization System

A comprehensive, professional-grade portfolio optimization platform combining Modern Portfolio Theory, Machine Learning, and Advanced Backtesting in an elegant web interface.

🌟 Overview

This system provides institutional-quality portfolio optimization tools accessible through both a web interface and Python API. It integrates classical portfolio theory with modern machine learning techniques to deliver robust, data-driven investment strategies.

Key Features

🎯 Multiple Optimization Methods
- Mean-Variance Optimization (Markowitz)
- Black-Litterman Model with investor views
- Risk Parity & Hierarchical Risk Parity (HRP)
🤖 Machine Learning Integration
- Random Forest for return prediction
- LSTM neural networks for time series
- Ensemble models combining multiple approaches
- 50+ technical indicators and features
📊 Professional Backtesting
- Realistic transaction costs
- Multiple rebalancing frequencies
- Comprehensive performance metrics
- Trade-by-trade analysis
🌐 Modern Web Interface
- Interactive portfolio builder
- Real-time optimization
- Educational content
- Experiment validation tools
📈 Comprehensive Stock Database
- 600+ stocks across all sectors
- Major ETFs and indices
- Preset portfolios (Tech Giants, Mag 7, etc.)
- Real-time data via Yahoo Finance

🚀 Installation

Prerequisites

Python 3.8 or higher
pip package manager
4GB+ RAM recommended

Setup

Clone the repository

git clone <repository-url>
cd portfolio_optimization

Create virtual environment (recommended)

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies

pip install -r requirements.txt

Dependencies

Core libraries include:

Data: yfinance, pandas, numpy
ML: scikit-learn, tensorflow, xgboost, lightgbm
Optimization: cvxpy, scipy
Web: flask, flask-cors
Visualization: matplotlib, seaborn, plotly
Technical Analysis: ta

⚡ Quick Start

Web Application

Start the Flask server:

python3 flask_app.py

Access the application at http://localhost:5000

Python API

from src.main import PortfolioOptimizer

# Initialize optimizer
optimizer = PortfolioOptimizer(
    tickers=['AAPL', 'GOOGL', 'MSFT', 'AMZN', 'META'],
    start_date='2020-01-01',
    end_date='2024-12-31'
)

# Collect data
optimizer.collect_data()

# Optimize portfolio
weights = optimizer.optimize_portfolio(
    method='mean_variance',
    objective='max_sharpe'
)

print("Optimal Weights:", weights)

# Run backtest
results = optimizer.backtest(
    method='mean_variance',
    rebalance_frequency='monthly'
)

print("Sharpe Ratio:", results['metrics']['sharpe_ratio'])
print("Total Return:", results['metrics']['total_return'])

🏗️ System Architecture

Core Components

portfolio_optimization/
├── src/
│   ├── main.py                    # Main PortfolioOptimizer class
│   ├── optimization/              # Optimization algorithms
│   │   ├── mean_variance.py       # Markowitz optimization
│   │   ├── black_litterman.py     # Black-Litterman model
│   │   └── risk_parity.py         # Risk parity & HRP
│   ├── data/                      # Data collection & processing
│   │   ├── data_collector.py      # Yahoo Finance integration
│   │   └── feature_engineering.py # Technical indicators
│   ├── backtesting/               # Backtesting engine
│   │   └── backtest_engine.py     # Realistic backtesting
│   └── visualization/             # Plotting utilities
├── flask_app.py                   # Web application server
├── stock_database.py              # 600+ stock database
├── validate_forecasts.py          # ML validation script
└── templates/                     # Web interface
    ├── index.html                 # Portfolio optimizer
    ├── learn.html                 # Educational content
    ├── experiments.html           # Validation tools
    └── about.html                 # Documentation

Data Flow

Data Collection → Yahoo Finance API fetches historical prices
Feature Engineering → 50+ technical indicators computed
ML Training → Models predict expected returns
Optimization → Algorithms find optimal weights
Backtesting → Strategy validated on historical data
Visualization → Results displayed in web interface

🎯 Optimization Methods

1. Mean-Variance Optimization (Markowitz)

Classical portfolio theory balancing risk and return.

Objectives:

max_sharpe: Maximize Sharpe ratio
min_volatility: Minimize portfolio variance
max_return: Maximize return for target volatility

Features:

Long-only or market-neutral constraints
Position size limits (max weight per asset)
Efficient frontier calculation
Diversification ratio metrics

Example:

weights = optimizer.optimize_portfolio(
    method='mean_variance',
    objective='max_sharpe',
    max_weight=0.4  # Max 40% per asset
)

2. Black-Litterman Model

Combines market equilibrium with investor views.

Features:

Absolute views: "AAPL will return 15%"
Relative views: "TSLA will outperform GM by 10%"
Confidence-weighted view integration
Reduced estimation error vs. pure Markowitz

Example:

# Add investor views
views = [
    {'type': 'absolute', 'asset': 'AAPL', 'return': 0.15, 'confidence': 0.7},
    {'type': 'relative', 'asset1': 'TSLA', 'asset2': 'GM', 
     'outperformance': 0.10, 'confidence': 0.5}
]

weights = optimizer.optimize_portfolio(
    method='black_litterman',
    views=views,
    risk_aversion=2.5
)

3. Risk Parity

Allocates capital so each asset contributes equally to portfolio risk.

Variants:

Standard Risk Parity: Equal risk contribution
Hierarchical Risk Parity (HRP): Uses clustering for stability

Features:

More diversified than market-cap weighting
Robust to estimation error
No expected returns required

Example:

weights = optimizer.optimize_portfolio(
    method='risk_parity',
    use_hrp=True  # Use hierarchical clustering
)

🤖 Machine Learning Models

Feature Engineering

50+ Technical Indicators:

Trend: SMA, EMA, MACD, ADX
Momentum: RSI, Stochastic, ROC
Volatility: Bollinger Bands, ATR, Keltner Channels
Volume: OBV, Volume SMA, VWAP
Custom: Lagged returns, rolling statistics

Models

1. Random Forest

Ensemble of decision trees
Handles non-linear relationships
Feature importance analysis
Fast training and prediction

2. LSTM Neural Networks

Captures temporal dependencies
Sequence-to-sequence architecture
Dropout for regularization
Suitable for time series

3. Ensemble Model

Combines Random Forest + LSTM
Weighted averaging of predictions
Improved robustness
Reduced overfitting

Training & Prediction

# Engineer features
optimizer.engineer_features()

# Train models
optimizer.train_models(
    train_ratio=0.8,
    use_lstm=True,
    use_rf=True,
    use_ensemble=True
)

# Use ML predictions for optimization
weights = optimizer.optimize_portfolio(
    method='mean_variance',
    use_predictions=True  # Use ML-predicted returns
)

🌐 Web Application

Pages

1. Portfolio Optimizer (`/`)

Interactive stock selection
Real-time optimization
Multiple optimization methods
Downloadable results
Performance visualization

2. Learn (`/learn`)

Portfolio theory fundamentals
Optimization method explanations
Interactive examples
Mathematical formulations

3. Experiments (`/experiments`)

ML model validation
Train/test split analysis
Prediction accuracy metrics
Model comparison tools

4. About (`/about`)

System documentation
API usage guide
Feature descriptions
Getting started tutorial

Features

Stock Database: 600+ stocks organized by sector
Preset Portfolios: Tech Giants, Mag 7, Diversified, etc.
Date Range Selection: Flexible historical periods
Advanced Options: ML integration, constraints, views
Export: CSV download of weights and allocations

📡 API Reference

REST Endpoints

`GET /api/stocks`

Get comprehensive stock database.

Response:

{
  "Technology": {
    "AAPL": "Apple Inc.",
    "GOOGL": "Alphabet Inc.",
    ...
  },
  "Financial Services": {...},
  ...
}

`GET /api/presets`

Get preset portfolios.

Response:

{
  "Tech Giants": ["AAPL", "GOOGL", "MSFT", "AMZN", "META"],
  "Mag 7": ["AAPL", "GOOGL", "MSFT", "AMZN", "NVDA", "TSLA", "META"],
  ...
}

`POST /api/optimize`

Optimize portfolio weights.

Request:

{
  "tickers": ["AAPL", "GOOGL", "MSFT"],
  "start_date": "2020-01-01",
  "end_date": "2024-12-31",
  "method": "mean_variance",
  "objective": "max_sharpe",
  "use_ml": true,
  "max_position_weight": 0.4
}

Response:

{
  "session_id": "1234567890.123",
  "weights": {
    "AAPL": 0.35,
    "GOOGL": 0.30,
    "MSFT": 0.35
  },
  "statistics": {
    "expected_return": 0.18,
    "volatility": 0.22,
    "sharpe_ratio": 0.82
  }
}

`POST /api/backtest`

Run portfolio backtest.

Request:

{
  "session_id": "1234567890.123",
  "method": "mean_variance",
  "rebalance_frequency": "monthly"
}

Response:

{
  "metrics": {
    "total_return": 0.45,
    "annual_return": 0.12,
    "sharpe_ratio": 0.85,
    "max_drawdown": -0.18,
    "win_rate": 0.62,
    "n_trades": 24
  },
  "timeseries": {
    "dates": ["2020-01-01", ...],
    "cumulative_returns": [1.0, 1.02, ...],
    "drawdown": [0, -0.01, ...]
  },
  "trades": [...]
}

`POST /api/experiment`

Run ML validation experiment.

Request:

{
  "tickers": ["AAPL", "GOOGL", "MSFT"],
  "train_start_date": "2020-01-01",
  "train_end_date": "2023-12-31",
  "test_start_date": "2024-01-01",
  "test_end_date": "2024-12-31",
  "ml_model": "random_forest"
}

Response:

{
  "portfolio_metrics": {
    "predicted_return": 0.15,
    "actual_return": 0.18,
    "error": 0.03,
    "mae": 0.025,
    "r2": 0.72
  },
  "asset_details": [
    {
      "ticker": "AAPL",
      "predicted": 0.20,
      "actual": 0.22,
      "error": 0.02
    },
    ...
  ]
}

`GET /api/health`

Health check endpoint.

Response:

{
  "status": "healthy",
  "timestamp": "2024-12-05T23:16:44",
  "active_sessions": 3
}

💡 Usage Examples

Example 1: Basic Optimization

from src.main import PortfolioOptimizer

# Create optimizer
optimizer = PortfolioOptimizer(
    tickers=['AAPL', 'MSFT', 'GOOGL', 'AMZN'],
    start_date='2020-01-01',
    end_date='2024-12-31'
)

# Collect data
optimizer.collect_data()

# Optimize for maximum Sharpe ratio
weights = optimizer.optimize_portfolio(
    method='mean_variance',
    objective='max_sharpe'
)

print(weights)

Example 2: ML-Enhanced Optimization

# Engineer features
optimizer.engineer_features()

# Train ML models
optimizer.train_models(
    use_rf=True,
    use_ensemble=True
)

# Optimize using ML predictions
weights = optimizer.optimize_portfolio(
    method='mean_variance',
    use_predictions=True,
    objective='max_sharpe'
)

Example 3: Black-Litterman with Views

# Define investor views
views = [
    {
        'type': 'absolute',
        'asset': 'AAPL',
        'return': 0.20,  # Expect 20% return
        'confidence': 0.8
    },
    {
        'type': 'relative',
        'asset1': 'GOOGL',
        'asset2': 'META',
        'outperformance': 0.05,  # GOOGL outperforms META by 5%
        'confidence': 0.6
    }
]

# Optimize with views
weights = optimizer.optimize_portfolio(
    method='black_litterman',
    views=views,
    risk_aversion=2.5
)

Example 4: Comprehensive Backtest

# Run backtest with monthly rebalancing
results = optimizer.backtest(
    method='mean_variance',
    rebalance_frequency='monthly'
)

# Access metrics
metrics = results['metrics']
print(f"Total Return: {metrics['total_return']:.2%}")
print(f"Sharpe Ratio: {metrics['sharpe_ratio']:.2f}")
print(f"Max Drawdown: {metrics['max_drawdown']:.2%}")
print(f"Win Rate: {metrics['win_rate']:.2%}")

# Get trade history
trades = optimizer.backtest_results['mean_variance']['engine'].get_trades()
print(trades.head())

Example 5: Risk Parity with HRP

# Optimize using Hierarchical Risk Parity
weights = optimizer.optimize_portfolio(
    method='risk_parity',
    use_hrp=True
)

# Get risk contribution analysis
opt_obj = optimizer.optimizers['risk_parity']
risk_analysis = opt_obj.get_risk_contribution_analysis()
print(risk_analysis)

🧪 Validation & Testing

Forecast Validation Script

The system includes a comprehensive validation script that trains models on historical data and tests predictions on out-of-sample periods.

Run validation:

python3 validate_forecasts.py

What it does:

Trains ML models on 2020-2023 data
Makes predictions for 2024-2025
Compares predictions to actual returns
Generates comprehensive visualizations
Calculates error metrics (MAE, MAPE, R²)

Preset portfolios tested:

Tech Giants (AAPL, GOOGL, MSFT, AMZN, META)
Mag 7 (+ NVDA, TSLA)
Financial (JPM, BAC, WFC, GS, MS)
Healthcare (JNJ, UNH, PFE, ABBV, TMO)
Diversified (SPY, TLT, GLD, VNQ, IEF)

Output:

Prediction accuracy metrics
Portfolio-level error analysis
Asset-by-asset comparison
Visualization saved to outputs/forecast_validation.png

Web-Based Experiments

Access /experiments page to:

Configure train/test splits
Select ML models
Compare model performance
Visualize prediction accuracy
Export results

📁 Project Structure

portfolio_optimization/
│
├── src/                           # Core source code
│   ├── main.py                    # Main PortfolioOptimizer class
│   ├── __init__.py
│   │
│   ├── optimization/              # Optimization algorithms
│   │   ├── __init__.py
│   │   ├── mean_variance.py       # Markowitz optimization
│   │   ├── black_litterman.py     # Black-Litterman model
│   │   └── risk_parity.py         # Risk parity & HRP
│   │
│   ├── data/                      # Data collection & processing
│   │   ├── __init__.py
│   │   ├── data_collector.py      # Yahoo Finance integration
│   │   └── feature_engineering.py # Technical indicators
│   │
│   ├── backtesting/               # Backtesting engine
│   │   ├── __init__.py
│   │   └── backtest_engine.py     # Realistic backtesting
│   │
│   ├── models/                    # ML models (generated)
│   │
│   └── visualization/             # Plotting utilities
│       ├── __init__.py
│       └── visualizer.py
│
├── templates/                     # Web interface templates
│   ├── base.html                  # Base template
│   ├── index.html                 # Portfolio optimizer
│   ├── learn.html                 # Educational content
│   ├── experiments.html           # Validation tools
│   └── about.html                 # Documentation
│
├── static/                        # Static web assets
│   ├── css/
│   │   └── style.css
│   ├── js/
│   │   ├── app.js                 # Main application logic
│   │   └── experiments.js         # Experiment page logic
│   └── images/
│
├── examples/                      # Usage examples
│   ├── __init__.py
│   └── example_usage.py
│
├── outputs/                       # Generated outputs
│   └── forecast_validation.png
│
├── config/                        # Configuration files
│
├── flask_app.py                   # Flask web application
├── stock_database.py              # 600+ stock database
├── validate_forecasts.py          # ML validation script
├── requirements.txt               # Python dependencies
├── setup.py                       # Package setup
├── .gitignore
└── README.md                      # This file

🎓 Key Concepts

Modern Portfolio Theory (MPT)

Developed by Harry Markowitz, MPT provides a framework for constructing portfolios that maximize expected return for a given level of risk.

Key principles:

Diversification reduces risk
Efficient frontier represents optimal portfolios
Sharpe ratio measures risk-adjusted returns

Black-Litterman Model

Addresses limitations of MPT by:

Starting with market equilibrium
Incorporating investor views with confidence levels
Reducing sensitivity to input estimation errors
Producing more stable, intuitive allocations

Risk Parity

Alternative to mean-variance optimization:

Allocates based on risk contribution, not capital
Each asset contributes equally to portfolio risk
More diversified than market-cap weighting
Doesn't require expected return estimates

Hierarchical Risk Parity (HRP)

Advanced risk parity using machine learning:

Uses hierarchical clustering on correlation matrix
Recursive bisection for allocation
More stable than traditional optimization
Robust to estimation error

🔧 Configuration

Environment Variables

# Flask configuration
export FLASK_ENV=development
export FLASK_DEBUG=1

# Data configuration
export DATA_CACHE_DIR=./cache
export MAX_DATA_AGE_DAYS=7

Config File

Create config/config.yaml:

data:
  default_start_date: "2020-01-01"
  cache_enabled: true
  cache_dir: "./cache"

optimization:
  default_method: "mean_variance"
  risk_free_rate: 0.02
  max_weight: 0.4

ml:
  train_ratio: 0.8
  random_state: 42
  n_estimators: 100

backtesting:
  initial_capital: 100000
  transaction_cost: 0.001
  rebalance_frequency: "monthly"

📊 Performance Metrics

Portfolio Metrics

Total Return: Cumulative return over period
Annual Return: Annualized return
Volatility: Standard deviation of returns
Sharpe Ratio: Risk-adjusted return
Sortino Ratio: Downside risk-adjusted return
Calmar Ratio: Return / max drawdown
Max Drawdown: Largest peak-to-trough decline
Win Rate: Percentage of profitable periods
Profit Factor: Gross profit / gross loss
VaR (95%): Value at Risk at 95% confidence
CVaR (95%): Conditional Value at Risk

ML Metrics

MAE: Mean Absolute Error
MAPE: Mean Absolute Percentage Error
R²: Coefficient of determination
Directional Accuracy: % of correct direction predictions

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Development Setup

# Install development dependencies
pip install -r requirements-dev.txt

# Run tests
pytest tests/

# Check code style
flake8 src/
black src/

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Harry Markowitz - Modern Portfolio Theory
Fischer Black & Robert Litterman - Black-Litterman Model
Marcos López de Prado - Hierarchical Risk Parity
Yahoo Finance - Market data API
scikit-learn, TensorFlow - ML frameworks

📞 Support

For questions, issues, or suggestions:

Open an issue on GitHub
Contact the development team
Check the /about page in the web interface

🗺️ Roadmap

Planned Features

📚 References

Academic Papers

Markowitz, H. (1952). "Portfolio Selection". Journal of Finance
Black, F., & Litterman, R. (1992). "Global Portfolio Optimization". Financial Analysts Journal
López de Prado, M. (2016). "Building Diversified Portfolios that Outperform Out of Sample". Journal of Portfolio Management

Books

Modern Portfolio Theory and Investment Analysis - Elton et al.
Advances in Active Portfolio Management - Grinold & Kahn
Machine Learning for Asset Managers - López de Prado

Built with ❤️ for quantitative finance enthusiasts

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
config		config
examples		examples
src		src
static		static
templates		templates
.gitignore		.gitignore
Archive.zip		Archive.zip
Final Testimonial for MIT Emerging Talent.pdf		Final Testimonial for MIT Emerging Talent.pdf
README.md		README.md
Retrospective for Portfolio Optimization System.pdf		Retrospective for Portfolio Optimization System.pdf
final presentation.pdf		final presentation.pdf
flask_app.py		flask_app.py
requirements.txt		requirements.txt
setup.py		setup.py
stock_database.py		stock_database.py
validate_forecasts.py		validate_forecasts.py

Folders and files

Latest commit

History

Repository files navigation

Portfolio Optimization System

🌟 Overview

Key Features

📋 Table of Contents

🚀 Installation

Prerequisites

Setup

Dependencies

⚡ Quick Start

Web Application

Python API

🏗️ System Architecture

Core Components

Data Flow

🎯 Optimization Methods

1. Mean-Variance Optimization (Markowitz)

2. Black-Litterman Model

3. Risk Parity

🤖 Machine Learning Models

Feature Engineering

Models

1. Random Forest

2. LSTM Neural Networks

3. Ensemble Model

Training & Prediction

🌐 Web Application

Pages

1. Portfolio Optimizer (/)

2. Learn (/learn)

3. Experiments (/experiments)

4. About (/about)

Features

📡 API Reference

REST Endpoints

GET /api/stocks

GET /api/presets

POST /api/optimize

POST /api/backtest

POST /api/experiment

GET /api/health

💡 Usage Examples

Example 1: Basic Optimization

Example 2: ML-Enhanced Optimization

Example 3: Black-Litterman with Views

Example 4: Comprehensive Backtest

Example 5: Risk Parity with HRP

🧪 Validation & Testing

Forecast Validation Script

Web-Based Experiments

📁 Project Structure

🎓 Key Concepts

Modern Portfolio Theory (MPT)

Black-Litterman Model

Risk Parity

Hierarchical Risk Parity (HRP)

🔧 Configuration

Environment Variables

Config File

📊 Performance Metrics

Portfolio Metrics

ML Metrics

🤝 Contributing

Development Setup

📄 License

🙏 Acknowledgments

📞 Support

🗺️ Roadmap

Planned Features

📚 References

Academic Papers

Books

About

Resources

Stars

Watchers

Forks

1. Portfolio Optimizer (`/`)

2. Learn (`/learn`)

3. Experiments (`/experiments`)

4. About (`/about`)

`GET /api/stocks`

`GET /api/presets`

`POST /api/optimize`

`POST /api/backtest`

`POST /api/experiment`

`GET /api/health`