streaming-market-analytics

An end-to-end streaming analytics platform for high-frequency market data: ingest from log-file sources, publish through NATS JetStream, persist into TimescaleDB, compute orderbook features and time bars in Go, run predictive models and impact calibration in Python. Full observability stack shipped (Prometheus, Grafana, Loki). Everything comes up with one docker compose up.

The repository is organized as a Go workspace with three services plus a shared library, and a Python analytics package that runs alongside on a scheduler.

About this codebase

I designed this system as a research platform to feed structured market data into an ML pipeline. The architectural decisions (choice of NATS JetStream over Kafka, TimescaleDB continuous aggregates over hand-rolled partitioning, orderbook feature schema, prediction horizons, impact model shape) are mine. Significant portions of the implementation were written with AI assistance under my architectural control -- a common pattern in 2026 that lets a single author operate a system that would otherwise require a team.

I run this system in production and monitor it daily through Grafana, so I can reason about failure modes, bottlenecks, and design trade-offs. I do not claim line-by-line authorship of every file. What I claim is architectural ownership and operational familiarity.

Architecture at a glance

┌─────────────┐    ┌─────────────┐    ┌──────────────┐    ┌─────────────┐
│ log source  │───▶│   ingest    │───▶│ NATS         │───▶│  features   │
│ (tailed)    │    │  (Go)       │    │ JetStream    │    │   (Go)      │
└─────────────┘    └─────┬───────┘    └──────────────┘    └──────┬──────┘
                         │                                        │
                         ▼                                        ▼
                   ┌─────────────────────────────────────────────────────┐
                   │      PostgreSQL + TimescaleDB                       │
                   │  hypertables · continuous aggregates · compression  │
                   └─────────────────────────────────────────────────────┘
                                            │
                        ┌───────────────────┼───────────────────┐
                        ▼                                       ▼
                 ┌─────────────┐                       ┌─────────────────┐
                 │  api (Go)   │                       │ analytics (Py)  │
                 │  REST + WS  │                       │ predictor,      │
                 │  JWT        │                       │ impact calib,   │
                 └─────────────┘                       │ feature store   │
                                                       └─────────────────┘
                        ▲                                       │
                        └───────────────────┬───────────────────┘
                                            ▼
                              Prometheus + Grafana + Loki

Ingest tails append-only log files, parses each line as a canonical market event, publishes to NATS JetStream, and batch-writes into TimescaleDB. Transitions between order snapshots (an order appearing, changing price, disappearing) are computed and stored separately for audit-style queries.

Features subscribes to NATS, computes per-item orderbook snapshots (best bid/ask, spread, mid, depth at 110/120/150% of best, hidden wall risk score), time bars (5m/1h/1d), decay rates, and cross-market kill pressure signals. Writes to TimescaleDB hypertables and caches in Redis.

API exposes REST endpoints and WebSocket streams over the aggregate tables, with JWT auth.

Analytics runs on an APScheduler entrypoint. Every 5 minutes the predictor produces predictions on five horizons (15m, 1h, 6h, 24h, 72h) using an ensemble of baseline + seasonal models. An impact calibrator periodically refits a market-impact model against actual trades. A feature store manages joined feature snapshots for downstream models.

The stack

• Go 1.22 with workspaces (go.work) — services/ingest, services/features, services/api, services/_shared
• Python 3.12 — analytics/ (Polars, sklearn, APScheduler, LightGBM in the predictive engine)
• PostgreSQL 16 + TimescaleDB 2.14 — 21 migrations building hypertables, continuous aggregates, compression policies, retention
• NATS JetStream — durable message queue
• Redis — hot orderbook cache
• Prometheus + Grafana + Loki — metrics, dashboards, log aggregation
• Docker Compose — full stack up in one command

Running it

# 1. Env
cp deploy/.env.example deploy/.env
$EDITOR deploy/.env        # set POSTGRES_PASSWORD, JWT_SECRET, HOOK_LOG_DIR

# 2. (optional) point at your own log sources / cities
$EDITOR configs/items_to_track.yaml

# 3. Put the log source somewhere the containers can read it
mkdir -p hook-logs
# your log producer writes here: market_hook_*.log

# 4. Bring the stack up
make up

# 5. Watch it come alive
make ps
make logs

Reachable services:

• PostgreSQL / TimescaleDB — localhost:5432
• NATS — localhost:4222 (monitoring on :8222)
• Redis — localhost:6379
• REST API — http://localhost:8080
• Grafana — http://localhost:3000 (admin password in .env)
• Prometheus — http://localhost:9090

See docs/DESIGN.md for the architectural rationale (why NATS JetStream, why TimescaleDB continuous aggregates, why the impact model looks the way it does, why the ingest service does its own transitions).

Repository layout

streaming-market-analytics/
├── services/                        Go monorepo (workspaces)
│   ├── ingest/                      log tail → NATS → Postgres
│   ├── features/                    orderbook / bars / decay / kill_pressure
│   ├── api/                         REST + WebSocket + JWT
│   └── _shared/                     canonical event schema, decimal helpers
├── analytics/                       Python package
│   ├── common/                      db, nats, redis, logging, metrics
│   ├── feature_store/               joined snapshots, catalog, builder
│   ├── predictive_engine/           5-horizon ensemble predictor
│   ├── impact_calibrator/           market impact regression
│   ├── parsers/                     auxiliary data parsers
│   └── schedulers/                  APScheduler entrypoint
├── db/
│   └── migrations/                  golang-migrate, TimescaleDB features
├── deploy/                          docker-compose + prometheus + grafana
├── configs/                         top-level YAML config
├── docs/
│   └── DESIGN.md                    architectural rationale
├── scripts/                         smoke tests and helpers
└── Makefile

License

MIT. See LICENSE.