MicroSentinel

Real-time market-manipulation surveillance over live L2 order books

A low-latency, event-driven engine that ingests live multi-exchange L2 order books, reconstructs them, and detects spoofing, layering, quote-stuffing, wash trading, and cross-venue dislocations as they happen — on a Rust → Kafka → ClickHouse pipeline, deployed on Kubernetes.

The problem

Market manipulation — placing large orders you intend to cancel (spoofing), stacking fake one-sided liquidity (layering), flooding the book with churn (quote stuffing), and wash / ignition trading — is a real, adversarial problem that exchanges and regulators (SEBI, SEC) staff whole teams to police. The signal lives in microstructure: the lifecycle of individual order-book levels at sub-millisecond resolution, across many venues at once.

This project surveils live, free public L2 feeds from multiple crypto exchanges (Binance, Coinbase; no paid API key), reconstructs every book, and runs streaming detectors that flag manipulation-like patterns in real time — each alert carrying the exact feature vector that triggered it, fully replayable from ClickHouse history.

Crypto venues are used because they expose free, high-volume, 24/7 L2 websockets, so the system genuinely runs on real data and is demoable on day one. The contracts and detectors are venue-agnostic and map directly onto equity microstructure.

Why this architecture (not decorative)

Forcing requirement	Choice
Hundreds of thousands of book/trade msgs/sec, allocation-light hot path	Rust ingestor + detector
Decouple ingest / detect / store / serve; replay history to tune detectors	Kafka (Redpanda)
Store every tick/delta/alert and answer OLAP questions fast	ClickHouse
Low-latency internal RPC for live book snapshots	gRPC / protobuf
Operate many always-on services	Kubernetes + Helm, Terraform, Prometheus/Grafana/OTel, k6

Architecture

flowchart LR
    EX["Binance / Coinbase<br/>(free public L2 + trades)"] -->|websocket| ING
    subgraph RUST["Rust"]
      ING["ingestor<br/>normalize · rebuild L2 books"] -->|protobuf| K[("Kafka / Redpanda<br/>md.book · md.trades")]
      K --> DET["detector<br/>spoof · layer · stuffing · wash · cross-venue"]
    end
    ING -->|gRPC book snapshots| GW
    DET -->|surveillance.alerts| GW
    DET --> CH[("ClickHouse<br/>trades · book_deltas · alerts · candles")]
    GW["Go gateway<br/>REST + WebSocket"] --> WEB["Next.js desk<br/>book heatmap · alert tape"]
    CH --> GW
    PY["Python<br/>offline backtest · AI explainer"] --> CH

Loading

Service	Language	Responsibility
services/ingestor	Rust	Per-exchange WS clients, L2 reconstruction, Kafka producer, gRPC book snapshots
services/detector	Rust	Streaming detectors → alerts to Kafka + ClickHouse
services/md-core	Rust	Pure-domain order book + detectors (unit + property tested, benched)
services/gateway	Go	gRPC→REST/WebSocket bridge, Kafka alert consumer, ClickHouse queries
app/surveillance	TypeScript / Next.js	Live order-book heatmap + alert tape
src/microsentinel	Python	Offline detector backtests + Azure-OpenAI alert explanations

Detectors

Detector	Signature
Spoofing	Large resting level (≫ rolling z-score) cancelled within T ms, away from the touch, never traded
Layering	≥ N fleeting large cancels clustered on one side
Quote stuffing	Book add/cancel message rate exceeds its rolling z-score
Wash / ignition	Rapid alternating (or one-sided) prints in a tight band that revert
Cross-venue dislocation	Same asset's mid diverges across venues beyond a fee band

Quickstart

# 1. Data plane (Redpanda + ClickHouse + Prometheus + Grafana)
make data-plane-up

# 2. Run the pipeline (each in its own shell)
make run-ingestor      # GS_EXCHANGES=binance,coinbase  (or 'synthetic' offline)
make run-detector
make run-gateway

# 3. The desk
npm install && npm run dev      # http://localhost:3000/surveillance

No internet / exchanges down? Run the ingestor with --exchanges synthetic — it generates a believable book with injected spoofing so the whole pipeline and desk light up.

What's verified

• md-core: unit tests for the book + detectors and a property test proving the reconstructed book never crosses, for any delta sequence (cargo test).
• Detector quality: offline backtest on synthetic data with labelled injections — precision 1.00 / recall 1.00 (python -m microsentinel.backtest).
• Cross-language parity: the Python detector port reproduces the Rust behaviour (pytest).
• Deploy: helm lint + helm template + kubeconform validate the chart and manifests.
• CI builds and tests Rust, Go, Python, and TypeScript plus the Helm chart on every push.

Repo layout

proto/                 shared protobuf contracts (market data + surveillance)
services/              Rust workspace (md-core, md-proto, ingestor, detector) + Go gateway
app/ components/ lib/  Next.js surveillance desk
src/microsentinel/      Python offline backtest + AI explainer
deploy/                docker-compose data plane, Helm chart, k8s/kind, Grafana, OTel
infra/terraform/       GKE + Artifact Registry
tests/load/            k6 load test for the gateway

See docs/ARCHITECTURE.md for the deep dive and RESUME.md for résumé-ready bullets.

License

MIT © Lalit Kumar — research/education; not investment advice.