Portfolio Optimizer API

A portfolio optimization service that combines Modern Portfolio Theory (Markowitz / Max Sharpe optimization) with LLM-driven news sentiment analysis and real-time market data streaming via Kafka.

The service exposes a single REST endpoint that computes the optimal asset allocation for a given basket of tickers, adjusting expected returns using sentiment extracted from recent news headlines by an LLM (OpenAI gpt-4o-mini).

---

Table of Contents

• Architecture
• Key Design Decisions
• Tech Stack
• Project Structure
• Getting Started
• Configuration
• API Reference
• How the Optimization Works
• Operational Notes
• Roadmap

---

Architecture

The system is composed of three independent services orchestrated via Docker Compose, decoupled through a Kafka message broker.

flowchart TB
    subgraph External["External Data Sources"]
        YF[("Yahoo Finance\n(yfinance)")]
        OAI[("OpenAI API\ngpt-4o-mini")]
    end
 
    subgraph Infra["Kafka Broker (KRaft mode, single node)"]
        TOPIC["Topic: market.prices"]
    end
 
    subgraph Producer["market-producer service"]
        MP["MarketDataProducer\n(polls every 60s)"]
    end
 
    subgraph API["portfolio-api service"]
        LC["Lifespan Handler\n(warms historical cache on boot)"]
        CONS["Kafka Consumer\n(background task, in-memory state)"]
        EP["POST /api/v1/optimize"]
        SVC["PortfolioService"]
        SENT["SentimentAnalyzer"]
        POOL["ProcessPoolExecutor\n(CPU-bound optimization)"]
    end
 
    Client(["Client"])
 
    YF -- "1m OHLC bars" --> MP
    MP -- "produce" --> TOPIC
    TOPIC -- "consume" --> CONS
    CONS -- "updates" --> API
 
    Client -- "HTTP POST" --> EP
    EP --> SVC
    SVC -- "fetch news + sentiment" --> SENT
    SENT -- "headlines" --> YF
    SENT -- "sentiment prompt" --> OAI
    SVC -- "historical cache + live prices + sentiment" --> POOL
    POOL -- "SLSQP Max-Sharpe optimization" --> SVC
    SVC -- "PortfolioResponse" --> Client
 
    LC -. "on startup" .-> API

Loading

Request Lifecycle

sequenceDiagram
    participant C as Client
    participant API as FastAPI Endpoint
    participant Svc as PortfolioService
    participant Sent as SentimentAnalyzer
    participant LLM as OpenAI
    participant Pool as ProcessPoolExecutor
 
    C->>API: POST /api/v1/optimize {tickers, ...}
    API->>Svc: generate_optimal_portfolio(req)
    Svc->>Svc: filter tickers against live Kafka cache
    Svc->>Sent: analyze_multiple_tickers(tickers)
    par per ticker
        Sent->>Sent: fetch_news_async(ticker)
        Sent->>LLM: chat.completions.create(prompt)
        LLM-->>Sent: {sentiment, confidence}
    end
    Sent-->>Svc: {ticker: sentiment}
    Svc->>Pool: _run_heavy_math(hist_df, live_prices, sentiment, req)
    Pool->>Pool: annualize returns/covariance
    Pool->>Pool: alpha-adjust returns with sentiment
    Pool->>Pool: SLSQP Max-Sharpe optimization
    Pool-->>Svc: (weights, return, volatility)
    Svc-->>API: PortfolioResponse
    API-->>C: 200 OK {optimal_weights, sharpe_ratio, ...}

Loading

---

Key Design Decisions

Decision	Rationale
CPU-bound math offloaded to ProcessPoolExecutor	scipy.optimize.minimize (SLSQP) is CPU-bound and would block the asyncio event loop if run in-process. Delegating to a process pool keeps the API responsive under concurrent load.
Live prices via Kafka, not synchronous API calls	Decouples market-data ingestion from request handling. The API never blocks on Yahoo Finance during a request — it only reads from an in-memory dict kept warm by a background consumer.
Historical data pre-fetched at startup (lifespan)	Avoids a cold, multi-second yfinance download on the first user request; the cache is warmed once and reused.
Sentiment as an additive alpha term	adjusted_return = mean_return + alpha * sentiment_score — a simple, explainable, tunable way to blend qualitative signal into a quantitative optimizer without destabilizing the covariance structure.
Sentiment failures degrade gracefully	Any OpenAI/API/parsing failure returns a neutral 0.0 sentiment rather than failing the whole request — the optimizer still runs on pure historical statistics.
frozen=True dataclass for config	Prevents accidental runtime mutation of shared configuration (tickers, risk-free rate, etc.) across concurrent requests/processes.
Explicit dropna() before covariance calculation	Prevents NaN propagation from silently corrupting the covariance matrix and producing an unstable/undefined optimization result.

---

Tech Stack

Layer	Technology
API Framework	FastAPI (async, Pydantic v2 schemas)
Numerical Optimization	NumPy, SciPy (SLSQP)
Market Data	yfinance
Streaming	Apache Kafka (KRaft mode, aiokafka)
LLM Sentiment	OpenAI gpt-4o-mini via openai AsyncClient
Concurrency	asyncio, ProcessPoolExecutor, asyncio.to_thread
Containerization	Docker, Docker Compose

---

Project Structure

.
├── src/
│   ├── api.py                 # FastAPI app, lifespan, DI, HTTP layer
│   ├── service.py              # Orchestration: live data + sentiment + heavy math
│   ├── portfolio_math.py        # Pure Markowitz / Max-Sharpe optimization logic
│   ├── sentiment_analyzer.py     # OpenAI-backed news sentiment scoring
│   ├── data_provider.py         # yfinance historical/news data access
│   ├── kafka_utils.py           # Async Kafka consumer (live price updates)
│   ├── market_producer.py        # Standalone producer polling Yahoo Finance -> Kafka
│   ├── schemas.py               # Pydantic request/response models
│   └── config.py                # Centralized, frozen application configuration
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
└── start.sh                # Boots Kafka + API + producer with a generated cluster ID

---

Getting Started

Prerequisites

• Docker & Docker Compose
• An OpenAI API key
• uuidgen available on your shell (used by start.sh to generate a Kafka cluster ID)

Run

export OPENAI_API_KEY="your-key-here"
chmod +x start.sh
./start.sh

This will:

1. Generate a fresh Kafka CLUSTER_ID.
2. Build and start three containers: kafka (broker), api (FastAPI), and producer (market data feed). The API will be available at http://localhost:8000.

Note: on startup, the API pre-fetches 5 years of historical data for 15 tickers via yfinance, which can take a noticeable amount of time on first boot — this is expected.

---

Configuration

All configuration lives in config.py and is overridable via environment variables:

Variable	Default	Description
KAFKA_BOOTSTRAP_SERVERS	localhost:9092	Kafka broker address
OPENAI_API_KEY	(empty)	OpenAI API key — required for sentiment scoring

Other parameters (tickers, history_period, alpha, risk_free_rate, trading_days_per_year) are compiled into the frozen PortfolioConfig dataclass and can also be overridden per-request via the request body (see below).

---

API Reference

POST /api/v1/optimize

Request body

Field	Type	Default	Constraints
tickers	List[str]	—	required, min 2 items
history_period	str	"5y"	pattern ^\d+[a-zA-Z]$
trading_days_per_year	int	252	> 0
alpha	float	0.02	>= 0
risk_free_rate	float	0.045	>= 0

Example request

{
  "tickers": ["AAPL", "MSFT", "GOOGL", "AMZN"],
  "risk_free_rate": 0.045
}

Example response

{
  "optimal_weights": {
    "AAPL": 0.31,
    "MSFT": 0.42,
    "AMZN": 0.27
  },
  "expected_return": 0.1842,
  "volatility": 0.1523,
  "sharpe_ratio": 0.9013
}

Error responses

Status	Cause
400	No live market data available yet for requested tickers, or historical data missing (ValueError)
500	Unhandled internal error (optimizer non-convergence, unexpected exception)

---

How the Optimization Works

1. Filter requested tickers against the live prices currently held in memory (populated by the Kafka consumer). Tickers with no live price are dropped; if none remain, the request fails fast with 400.
2. Merge the warmed historical price cache with the latest live tick to form a single time series per ticker.
3. Annualize log-returns and covariance (mean * trading_days, cov * trading_days).
4. Sentiment-adjust expected returns: for each ticker, adjusted_return += alpha * sentiment_score, where sentiment_score ∈ [-1, 1] comes from an LLM analysis of recent headlines.
5. Optimize using SLSQP to maximize the Sharpe ratio, subject to Σweights = 1 and 0 ≤ weight ≤ 1 (long-only, fully invested, no leverage).
6. Return weights above a 0.01 threshold, plus expected return, volatility, and Sharpe ratio.

---

Operational Notes

• Statelessness caveat: live market data and historical cache live in a single process's memory (app_state). Horizontal scaling of the api service would currently require each replica to consume its own Kafka feed and re-warm its own cache — there is no shared cache layer (e.g., Redis) yet.
• Single-broker Kafka: the provided docker-compose.yml runs Kafka in KRaft mode with a single node (KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1) — suitable for local development, not for production durability.
• Cold start cost: initial historical data fetch for 15 tickers over 5 years happens synchronously in the lifespan handler, delaying readiness on first boot.
• LLM cost/latency: sentiment analysis makes one LLM call per ticker per request (temperature=0.0 for determinism); consider caching sentiment results with a TTL for high-traffic scenarios.

---

Roadmap

• Shared cache (Redis) for horizontal scalability of the API layer
• TTL-based caching of sentiment scores to reduce OpenAI cost/latency
• Multi-broker Kafka setup with replication for production durability
• Structured logging / tracing (OpenTelemetry)
• Health check endpoint (/health) reporting cache warmth and Kafka consumer lag
• Authentication / rate limiting on the public endpoint

---