Microprice

Does order-flow imbalance predict short-horizon returns after costs? A research and execution framework for market microstructure.

This is a research project, not a trading bot. We're testing whether order-flow imbalance (OFI) predicts 10–60s mid-price returns after controlling for spread, volatility and momentum, and after transaction costs. A clean negative result counts as a finding.

Role	Owner	Spec
Quant Research (QR)	Derek (@dryhour)	QUANT_RESEARCH.md
Quant Engineering (QE)	Josh	QUANT_ENGINEERING.md

Timeline: 8 weeks, Wed Sep 30, 2026 → Tue Nov 24, 2026. Each week runs Wednesday to Tuesday. Deadlines are end of day Tuesday, and we do a short sync that day to check the gate before starting the next week.

---

Hypotheses

ID	Hypothesis
H1	OFI predicts the sign and size of the next h-second mid-price return
H2	OFI adds information beyond spread, volatility, momentum and top-of-book imbalance
H3	Predictive power decays with horizon, fastest in liquid names
H4	The signal is still profitable after realistic costs
H5	The signal is regime-dependent and decays over time

Full tests and methodology are in QUANT_RESEARCH.md §1 and §4.

---

Repository layout

microprice/
├── common/              # shared: paths, version metadata, frozen universe + holdout guard
├── ingest/              # QE: WebSocket collector, historical loaders
├── book/                # QE: order-book reconstruction
├── features/            # QE implements, QR specs (feature code)
├── backtest/            # QE: event-driven engine, execution sim, costs, portfolio
├── api/                 # QE: FastAPI service
├── dashboard/           # QE: Streamlit/React dashboard
├── research/
│   ├── notebooks/       # QR: exploration (EDA, IC studies)
│   ├── studies/         # QR: scripts that produce paper results; trials.csv lives in research/
│   ├── hand_calcs/      # QR: hand-computed feature examples for QE's unit tests
│   └── output/          # generated tables (IC, data quality)
├── paper/
│   └── figures/         # generated by scripts only
├── configs/             # strategy.yaml, cost scenarios, run configs
├── scripts/             # pipeline runners (week2_pipeline.sh)
├── tests/               # unit, golden, property, leakage, determinism tests; fixtures/ = raw samples
├── docs/                # architecture.md, data.md, performance.md
├── data/                # git-ignored, regenerated from raw
│   ├── raw/             #   raw/{venue}/{symbol}/{date}/  (immutable, append-only)
│   ├── book/            #   reconstructed L2 snapshots
│   ├── features/        #   features/{symbol}/{date}.parquet   (QE → QR)
│   └── signals/         #   signals/{run_id}.parquet           (QR → QE)
└── results/             # git-ignored: results/{run_id}/        (QE → QR)

Planned later (not created yet): Makefile (make data | features | backtest | walkforward | paper), docker-compose.yml.

Getting started

python3 -m venv .venv && .venv/bin/pip install uv
UV_PROJECT_ENVIRONMENT=.venv .venv/bin/uv sync --all-extras --group dev
.venv/bin/uv run pre-commit install          # ruff, ruff format, mypy on commit
.venv/bin/python -m pytest -q                 # tests marked `data` skip without data/

.venv/bin/python -m ingest.tardis --wait-for-quota   # historical raw data (universe v1)
.venv/bin/python -m book.build                       # raw → data/book/   (1 s grid L2)
.venv/bin/python -m features.build                   # book → data/features/
.venv/bin/python -m book.golden tardis BTC-USD 2024-11-01 --max-rows 300000
.venv/bin/python -m ingest.coinbase_ws               # live collector (runs until stopped)
.venv/bin/python -m ingest.capture_report            # live capture summary + golden test

Data notes are in docs/data.md. Tardis's free tier has a daily transfer quota (~18 GB). The full universe needs about two quota windows, and the loader waits for the reset by itself.

The role docs write the contract paths as features/… and signals/…. Here they live under data/ so they don't collide with the features/ code package. The formats are unchanged.

---

Interface contract (v1)

Change it only by agreement, and bump the version.

Artifact	Producer → Consumer	Format
data/features/{symbol}/{date}.parquet	QE → QR	ts_ns, symbol, one column per feature, labels y_10s, y_30s, y_60s
data/signals/{run_id}.parquet	QR → QE	ts_ns, symbol, signal (target position in [-1, 1]), model_id
configs/strategy.yaml	QR → QE	thresholds, holding period, max position, cost scenario
results/{run_id}/	QE → QR	trades.parquet, pnl.parquet, metrics.json, config.yaml, git SHA

• ts_ns is the local receive time, int64 nanoseconds UTC. Exchange time is stored separately.
• Every result must be reproducible from (git SHA, config, data version).

---

8-week plan

Week	Dates	Focus	QR (Derek)	QE (Josh)
1	Sep 30 – Oct 6	Research setup	Hypothesis + literature review	Data architecture + ingestion
2	Oct 7 – Oct 13	Data	Explore market data + identify features	Historical data pipeline
3	Oct 14 – Oct 20	Signal	Order-book / microstructure signals	Feature computation infrastructure
4	Oct 21 – Oct 27	Modeling	Statistical / predictive models	Initial backtesting engine
5	Oct 28 – Nov 3	Strategy	Combine signals + portfolio logic	Execution + slippage / fees
6	Nov 4 – Nov 10	Validation	Walk-forward + out-of-sample testing	Optimize bottlenecks
7	Nov 11 – Nov 17	Robustness	Statistical significance + stress tests	Productionize / reproducibility
8	Nov 18 – Nov 24	Finalization	Research paper + conclusions	Dashboard + documentation

---

TODO

Tick a box when it's done and merged. A week's Gate must be fully ticked before either of us starts the next week's work that depends on it. Handoffs with an earlier date are what the other person is blocked on, so hit those first.

Week 1: Research setup (due Tue Oct 6)

QR — Derek

• Finalize research question and H1–H5, with the economic rationale for each → research/hypotheses.md (Fri Oct 2)
• Literature review notes in research/: Cont, Kukanov & Stoikov (2014); Stoikov (2018); Cartea, Jaimungal & Penalva; Bailey & López de Prado (deflated Sharpe) → research/literature_review.md (Tue Oct 6; done Oct 11)
• Sign off feature spec and label spec: horizons, clock vs. event time, mid vs. microprice → research/feature_label_spec.md (Fri Oct 2)
• Choose symbol and date universe before looking at any results, and write it down → research/universe.md, configs/universe.yaml (Tue Oct 6; done Oct 11)
• Define the final holdout dates (last ~20%) and record them → 5 of 24 dates, 2026-06-01 … 2026-10-01, enforced in code by common.universe (Tue Oct 6; done Oct 11)
• Create research/trials.csv with its column headers → research/trials.csv, columns in research/trials.md (Tue Oct 6; done Oct 11)

QE — Josh

• Choose a data source (crypto L2 recommended) and document the choice in docs/data.md → Coinbase: live Advanced Trade WS + historical Tardis.dev (docs/data.md §1) (Fri Oct 2; done Oct 11)
• Write the raw message schema doc (exchange_ts, recv_ts_ns, sequence numbers) → docs/data.md §2 (Fri Oct 2; done Oct 11)
• Send QR a raw data sample → tests/fixtures/raw/ (live + Tardis), described in docs/data.md §7 (Fri Oct 2; done Oct 11)
• Collector running with reconnect/backoff, sequence-gap detection, snapshot resync → ingest/coinbase_ws.py, first 1 h run on 2026-10-11 with 0 gaps (Tue Oct 6; done Oct 11)
• Raw append-only Parquet archive at data/raw/{venue}/{symbol}/{date}/ → ingest/archive.py, ingest/tardis.py (Tue Oct 6; done Oct 11)
• Repo tooling: pyproject.toml (uv/poetry), ruff, mypy, pre-commit, CI skeleton → pyproject.toml + uv.lock, .pre-commit-config.yaml, .github/workflows/ci.yml (Tue Oct 6; done Oct 11)

Gate

• Both: interface contract v1 agreed (Fri Oct 2). Implemented as written (paths under data/); needs Derek and Josh to confirm at the sync.
• QE: 24h capture completed with gaps detected and resynced. Not yet run: a 1 h trial ran Oct 11 (0 gaps, 18 audits passed). Start the 24 h run with caffeinate -i .venv/bin/python -m ingest.coinbase_ws --duration 93600 (the Mac must stay awake), then run python -m ingest.capture_report and fill in docs/data.md §8. Gap → resync is tested end to end against a fake exchange.
• QR: symbols, dates and holdout frozen

Week 2: Data (due Tue Oct 13)

QR — Derek

• EDA notebook: spread and volatility profiles, intraday seasonality, message rates (Tue Oct 13). Written (research/notebooks/01_eda.ipynb); run scripts/week2_pipeline.sh after the Tardis quota resets (~22:00 UTC daily) to download the rest of the in-sample data and execute it.
• Data-quality review of the reconstructed book: gaps, crossed books, outliers (Fri Oct 9). BTC-USD and ETH-USD reviewed in research/data_quality.md; scripts/week2_pipeline.sh generates the tables for the other four symbols, which then need a read.
• Univariate ICs of v1 features vs. y_10s, y_30s, y_60s, logged in trials.csv (Tue Oct 13). research/studies/univariate_ic.py is ready; scripts/week2_pipeline.sh runs it after the liquidity buckets are frozen; it refuses to run before.
• Final list of features to build for Week 3, with exact definitions → research/week3_features.md (Fri Oct 9; done Oct 11)
• Hand-computed examples for the v1 features → research/hand_calcs/v1_features.md (spec §6, Fri Oct 9; done Oct 11)

QE — Josh

• L2 book reconstruction (snapshot + deltas) with invariant checks → book/ (Fri Oct 9; done Oct 11)
• Historical pipeline: raw → book → features, partitioned Parquet with version metadata (code_git_sha, source_version, created_at) → ingest.tardis → book.build → features.build (Tue Oct 13; done Oct 11; the full-universe run waits on the Tardis quota)
• v1 features: queue imbalance, spread, mid, realized vol, momentum → features/v1.py, labels in features/labels.py (Fri Oct 9; done Oct 11)
• Point-in-time query path via DuckDB or polars lazy scans → features/query.py (Tue Oct 13; done Oct 11)

Gate

• QE: golden-snapshot test passes with 0 mismatches. Tardis top-25: 299,559 snapshots / 15.0M levels; live exchange audits: 18 / 233,764 levels; both 0 mismatches (docs/data.md §6).
• QE: v1 features match QR's hand calculations (tests/test_features_v1.py)
• QR: EDA reviewed together at the sync

Week 3: Signal (due Tue Oct 20)

QR — Derek

• OFI, multi-level OFI and microprice signal study (Tue Oct 20)
• HAC (Newey-West) regressions for H1 (Fri Oct 16)
• IC by horizon and liquidity for H3; nested-model tests for H2 (Tue Oct 20)
• Hand-computed examples for each new feature, for QE's unit tests (Wed Oct 14)

QE — Josh

• Full feature set: OFI, multi-level OFI, depth imbalance, trade-flow imbalance, microprice, activity, time of day (Fri Oct 16)
• Streaming/incremental feature engine; batch mode = replay through it (Tue Oct 20)
• Trailing-only normalization; labels computed in a separate, clearly marked step (Fri Oct 16)
• Multi-symbol batch runs (Tue Oct 20)

Gate

• QE: leakage test passes (perturbing future events changes nothing at t)
• QE: throughput benchmark recorded (events/sec per stage)
• QR: H1–H3 preliminary results written up

Week 4: Modeling (due Tue Oct 27)

QR — Derek

• Model ladder rungs 1–4: univariate IC → OLS/HAC → Ridge/Lasso/Elastic Net → logistic (Fri Oct 23)
• Gradient boosting (LightGBM), kept only if it beats the regularized linear model out of sample (Tue Oct 27)
• Walk-forward split design with purge and embargo, written as a spec for QE (Fri Oct 23)
• Backtester acceptance tests, including a toy scenario with a known P&L (Wed Oct 21)

QE — Josh

• Event-driven backtester v1: event queue, strategy interface, order management, portfolio (Fri Oct 23)
• Explicit latency model (decision + order-to-exchange) (Tue Oct 27)
• Execution Level 1: market orders cross the spread at the touch (Tue Oct 27)
• Sanity strategies: random signal, always-long, "cheating" future-label strategy (Tue Oct 27)

Gate

• QE: determinism test passes (same inputs → byte-identical outputs)
• QE: toy scenario gives QR's expected P&L
• QR: first data/signals/{run_id}.parquet runs end to end through the backtester

Week 5: Strategy (due Tue Nov 3)

QR — Derek

• Combine signals and define portfolio logic: thresholds, holding period, max position (Fri Oct 30)
• configs/strategy.yaml v1 handed to QE (Fri Oct 30)
• Gross vs. net analysis, break-even cost, cost sensitivity grid (H4) (Tue Nov 3)

QE — Josh

• Execution Level 2: walk the book for size (Fri Oct 30)
• Execution Level 3: limit orders with queue-position modeling; Level 4 adverse selection if time allows (Tue Nov 3)
• Pluggable cost model with optimistic / realistic / pessimistic scenarios and square-root impact (Fri Oct 30)
• Cost-sweep mode (0 → N bps) (Tue Nov 3)

Gate

• QE: queue-fill unit tests pass
• QE + QR: cost sweep produces a break-even curve
• QR: H4 preliminary result written up

Week 6: Validation (due Tue Nov 10)

QR — Derek

• Full walk-forward and out-of-sample results, holdout still untouched (Tue Nov 10)
• Model ladder comparison with Diebold-Mariano tests (Tue Nov 10)
• Run the bias checklist (QR §6) on every result so far (Tue Nov 10)

QE — Josh

• Profile with cProfile / py-spy / scalene and identify the real hotspots (Fri Nov 6)
• Optimize them (polars → NumPy → Numba → C++ only if still needed) (Tue Nov 10)
• Parallel walk-forward fold runner with purge/embargo support, plus a results store (Fri Nov 6)
• docs/performance.md with before/after throughput (Tue Nov 10)

Gate

• QE: folds reproduce identically; N× speedup documented
• QR: bias checklist clean for all walk-forward results

Week 7: Robustness (due Tue Nov 17)

QR — Derek

• Evaluate the final holdout exactly once, and record that in the paper (Fri Nov 13)
• Deflated Sharpe using the true trial count from trials.csv; White's Reality Check / SPA (Tue Nov 17)
• Stationary block bootstrap CIs on IC and Sharpe (Tue Nov 17)
• Regime splits (volatility, spread, time of day) and structural-break tests (CUSUM, Chow, Bai-Perron) for H5 (Tue Nov 17)
• Decay analysis: IC vs. horizon, rolling 20-day IC (Tue Nov 17)

QE — Josh

• Dockerfile + Docker Compose (Fri Nov 13)
• CI runs unit, golden, leakage, determinism and sanity-strategy tests (Fri Nov 13)
• Data versioning and config-driven runs; parameter-sweep API (Tue Nov 17)
• Makefile targets: data, features, backtest, walkforward, paper (Tue Nov 17)

Gate

• QE: CI green on a fresh clone
• QR: H1–H5 each have a result, test statistic, and p-value or CI

Week 8: Finalization (due Tue Nov 24)

QR — Derek

• Paper draft (paper/), 8–15 pages, following QR §8 (Fri Nov 20)
• Every paper figure generated by a script in research/studies/ (Fri Nov 20)
• Final paper, conclusions, and the README research section (Tue Nov 24)
• Review QE's engineering sections of the paper (Mon Nov 23)

QE — Josh

• FastAPI: /runs, /runs/{id}, /runs/{id}/trades, /compare, /features/{symbol}/{date}, /health (Fri Nov 20)
• Dashboard: gross vs. net equity, drawdown, rolling IC/Sharpe, cost sweep, trade scatter, run comparison, data health (Tue Nov 24)
• docs/architecture.md, docs/data.md finalized; engineering sections of the paper (data, execution model, limitations) (Fri Nov 20)

Gate: Definition of Done

• Fresh clone → docker compose up → make paper reproduces every result
• Book reconstruction golden test: 0 mismatches
• Leakage, determinism and sanity-strategy tests pass in CI
• Execution Levels 1–3 and the cost sweep are implemented and documented
• Net-of-cost results reported with break-even cost
• Holdout evaluated exactly once; deflated Sharpe reported
• Bias checklist fully checked