LiveBacktestGEX

A live forward-testing system for index options regimes. Each minute during U.S. equity session hours, the bots pull live market data from the Schwab Market Data API, compute gamma exposure (GEX) and related regime features, classify volatility with a pre-trained Hidden Markov Model (HMM), and simulate mean-reversion trades on paper (no broker orders). Results are logged to stdout, a local DuckDB database, and optionally Discord.

Two parallel deployments target different underlyings:

Variant	Live script	HMM training	Index symbol	OFI proxy
NQ	live_backtester_nq.py	train_hmm_nq.py	$NDX	QQQ bid/ask size ratio
ES	live_backtester_es.py	train_hmm_es.py	$SPX	SPY bid/ask size ratio

The NQ and ES backtesters are structurally the same; only symbols, model filenames, and Docker images differ.

---

What “live forward test” means here

This is not automated live trading. It is a forward test / paper simulator that:

1. Runs continuously (typically in Docker on a server).
2. Uses real-time Schwab quotes, minute bars, and options chains.
3. Applies the same signal logic you would use in production.
4. Updates a virtual balance, records fills with a fixed slippage assumption, and exits on targets, stops, regime change, or end-of-day flatten.

Comparing logged trades to your RTF exports (kept locally, not in git) is how you validate behavior over time.

---

High-level flow

Each 60-second loop (only active Mon–Fri, 9:30 AM–4:00 PM US/Eastern):

flowchart TD
    A[Wake / sleep until market open] --> B[Fetch quote + 1m history]
    B --> C[Compute net GEX from options chain]
    B --> D[Compute OFI from QQQ or SPY]
    B --> E[Update price buffer]
    E --> F[Regime: GEX sign + Hurst + HMM state]
    F --> G{Position open?}
    G -->|No| H[Check entry signals]
    H -->|Signal| I[Simulate OPEN with Kelly sizing]
    G -->|Yes| J[Check TP / SL / gamma flip / EOD]
    J -->|Exit| K[Update balance + DuckDB + Discord]
    I --> L[Sleep 60s]
    K --> L
    H -->|No signal| L
    J -->|Hold| L

Loading

---

Repository layout

File	Role
live_backtester_nq.py / live_backtester_es.py	Main forward-test loop: data, regimes, signals, simulated PnL
train_hmm_nq.py / train_hmm_es.py	One-off (or periodic) HMM training from Schwab history; writes .pkl
hmm_pretrained_nq.pkl / hmm_pretrained_es.pkl	Serialized GaussianHMM (gitignored; required at runtime and in Docker build)
requirements.txt	Python dependencies
.env.example	Template for Schwab + Discord secrets
Dockerfile.nq / Dockerfile.es	Slim images that run the corresponding backtester
runcommands.txt	Example docker buildx / docker run commands for local build and server deploy

---

Data sources and features

Schwab API client

Both live and training scripts authenticate with OAuth2 refresh token flow (SCHWAB_APP_KEY, SCHWAB_APP_SECRET, SCHWAB_REFRESH_TOKEN). Access tokens are refreshed about every 28 minutes.

Used endpoints:

• Quotes — last price for $NDX / $SPX and OFI symbols.
• Price history — 1-minute closes (1 day for live loop; 10 days for HMM training).
• Option chains — near-the-money, 40 strikes, for GEX.

Net GEX

From the options chain, for each call and put contract:

• Calls: net_gex += gamma × open_interest × 100 × spot
• Puts: net_gex -= gamma × open_interest × 100 × spot

Regime label: POSITIVE if net_gex > 0, else NEGATIVE. Dealers’ gamma positioning is used as a coarse “pinning vs acceleration” backdrop.

Order flow imbalance (OFI)

A simple proxy: rolling mean (last 5 samples) of bid_size / ask_size on QQQ (NQ) or SPY (ES). Used as a filter on reversion entries (e.g. short only if OFI < 0.5, long if OFI > 2.0).

Hurst exponent

Estimated on the live price buffer (up to 100 ticks) via log-log regression of lagged price differences. Values < 0.5 suggest mean-reverting behavior; required for entry together with other filters.

HMM volatility state

• Training (train_hmm_*.py): fits a 2-state GaussianHMM on 1-minute returns (~10 days of bars). Saves hmm_pretrained_*.pkl after checking convergence and that high/low vol variances differ meaningfully (ratio ≥ 1.5 preferred).
• Inference (live): last 20 minute returns are predicted; the state with lower learned variance is labeled Low Volatility, the other High Volatility.

Entry logic currently requires Low Volatility + Positive GEX + Hurst < 0.5 (mean-reversion “REVERSION” bucket).

OU-style bands

Mean μ and standard deviation σ over the last 20 prices (live buffer when ≥10 samples, else intraday 1m history). Bands: μ ± 2σ. Targets use μ; stops are 3σ from entry.

Risk sizing

Half-Kelly from fixed win rates (REVERSION: 79%, MOMENTUM: 59% — momentum path exists in sizing but current signals only emit REVERSION). Capped at 5% of virtual balance per trade. Simulated fill: ±0.25 points slippage on entry.

Safety / session rules

• Gap filter: If session open gap > 0.5%, no new trades for the first ~15 minutes (buffer length < 15).
• Stop cooldown: 5 minutes after a stop-loss exit.
• EOD flatten: Close any open position at 3:45 PM Eastern.
• Regime flip exit: Close if GEX sign changes vs entry.
• Auth failure: Critical log + Discord alert + graceful shutdown.

---

Trade simulation details

When a signal fires:

1. OPEN — position stored in memory with entry, target, stop, strategy type, and entry gamma regime.
2. Each minute while open — check target, stop, gamma flip, or EOD.
3. CLOSE — PnL = size × pct_move (direction-adjusted); balance updated; row inserted into DuckDB trades table.

Discord embeds (if DISCORD_WEBHOOK_URL is set) announce opens and closes with color by exit reason (TARGET, STOP_LOSS, EOD_FLATTEN, REGIME_FLIP).

Default starting balance: $100,000 (virtual).

---

DuckDB persistence

File: quant_trading_data.db (created in the working directory; gitignored).

Table trades:

Column	Description
timestamp	Close time
symbol	$NDX or $SPX
action	LONG / SHORT
price	Exit price
size	Capital at risk for the trade
regime	GEX regime at exit
pnl	Realized PnL dollars

---

Setup

1. Python environment

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Credentials

Copy the template and fill in values from your Schwab Developer app:

cp .env.example .env
# Edit .env — never commit this file

Required:

• SCHWAB_APP_KEY
• SCHWAB_APP_SECRET
• SCHWAB_REFRESH_TOKEN

Optional:

• DISCORD_WEBHOOK_URL — trade alerts

Export for local runs:

set -a && source .env && set +a

3. Train HMM models (before first live run or Docker build)

python train_hmm_nq.py   # writes hmm_pretrained_nq.pkl
python train_hmm_es.py   # writes hmm_pretrained_es.pkl

These files are not in the repository. You must generate them on each machine that runs or builds the containers. Re-train periodically if you want the HMM to track recent volatility structure.

4. Run locally (forward test)

python live_backtester_nq.py
# or
python live_backtester_es.py

The process runs until killed (SIGINT/SIGTERM closes DuckDB cleanly). Outside market hours it sleeps until the next 9:30 AM Eastern open.

---

Docker deployment

Images bundle the backtester script and the matching .pkl. Build only after the corresponding pickle exists in the build context.

See runcommands.txt for example multi-arch build and server docker run with --env-file .env. Typical pattern:

• Image davidburca/gex-backtesting-nq → Dockerfile.nq → live_backtester_nq.py
• Image davidburca/gex-backtesting-es → Dockerfile.es → live_backtester_es.py

Containers are intended to run with --restart unless-stopped during the trading week.

---

Dependencies

Package	Use
numpy / pandas	Math, buffers
duckdb	Trade log database
requests	Schwab REST + Discord
hmmlearn	Gaussian HMM train/infer
pytz	US/Eastern session clock
schwab-py	Listed in requirements (OAuth helpers available; live code uses direct REST)

---

Operational notes

• Rate limits: Schwab may return 429 Too Many Requests on heavy chain usage; the loop logs errors and continues next minute.
• Refresh tokens: Expired or revoked refresh tokens stop the bot after a critical auth error.
• Not investment advice: Research and simulation tooling only; past simulated performance does not guarantee future results.
• RTF logs: Files like Trades 5:20.rtf are local Discord/log exports for your review; they stay out of git via *.rtf.

---

Quick reference: entry conditions (current logic)

All must hold for a REVERSION entry:

• HMM = Low Volatility
• GEX = POSITIVE
• Hurst < 0.5
• Price outside μ ± 2σ band
• OFI filter: SHORT if price > upper band and OFI < 0.5; LONG if price < lower band and OFI > 2.0
• No active gap suspension, stop cooldown, or existing position

---

License

Add a license file if you plan to open-source or share the repository publicly.