FPGA Arb - Cross-venue arbitrage and Kalman market making on FPGA

FPGA market-data and order-generation prototype on a PYNQ-Z2 / Zynq-7020.

The system replays historical multi-venue market data, pushes normalized market records through a real FPGA datapath, emits IOC taker and limit maker order frames, and matches those frames on the server against timestamp-merged MBO data. The main result is not a PnL claim. The main result is a measured low-latency hardware decision path with enough surrounding infrastructure to show where the real bottlenecks move once the strategy logic is in RTL.

Executive Summary

• Hardware strategy path in RTL at 166.67 MHz.
• Fixed-point FPGA order path: no floating point in the decision/OMS datapath.
• Multi-venue replay, FPGA order generation, host forwarding, MBO fill simulation, and static trader dashboard are all wired end to end.
• Best observed total FPGA trigger-to-egress latency is 24 ns, exactly the expected 4-cycle path at a 6 ns clock.
• Median total FPGA trigger-to-egress latency is 84 ns.
• Strategy decision latency is stable: p50 42 ns, p99 42 ns, max 378 ns.
• Tail latency is dominated by PYNQ/simple-mode DMA egress, AXI ready backpressure, Linux/userspace forwarding, and TCP/server path, not by the strategy compare path.

This is a systems prototype. It is intentionally instrumented so a reviewer can separate:

• strategy decision latency,
• OMS/risk/egress latency,
• total FPGA-resident latency,
• non-FPGA host/server latency,
• replay/matching behavior.

Architecture

Databento DBN/bin data
        |
        v
server replay
  - venue feed replay
  - timestamp-merged MBO simulator
  - TCP order receiver
        |
        | UDP multicast market records
        v
PYNQ-Z2 ARM client
  - joins replay multicast groups
  - batches market records
  - drives AXI DMA MM2S
        |
        | AXI DMA / control domain @ 142.86 MHz
        v
AXI clock converter
        |
        | strategy stream domain @ 166.67 MHz
        v
FPGA PL core @ 166.67 MHz
  decoder
      |
      +--> ES-band taker / maker strategy
      |
      +--> cross-venue lock detector
      |
      v
  OMS / risk / order serializer
      |
      +--> taker AXI-Stream order lane
      |
      +--> maker AXI-Stream order lane
        |
        | strategy stream domain @ 166.67 MHz
        v
AXI clock converter
        |
        | 32-byte order frames over AXI DMA S2MM @ 142.86 MHz
        v
PYNQ-Z2 ARM client
  - drains order DMA
  - forwards frames over a persistent TCP connection
        |
        v
server MBO matcher
  - consumes timestamp-merged MBO events
  - previews taker IOC fills without mutating historical liquidity
  - tracks maker quote resting/fills
        |
        v
fills.log -> dashboard/data/session.js -> dashboard/index.html

FPGA Blocks

Block	Role
rtl/decoder.sv	Parses packed replay records into venue/symbol/price/size/timestamp fields.
rtl/kalman.sv	Fixed-point fair-value, ES-band taker, maker quote, inventory-aware quote gating.
rtl/lock_detector.sv	Cross-venue locked/crossed-book detector that emits paired IOC legs.
rtl/oms.sv	Taker priority lane, maker lane, position/risk checks, 32-byte frame serialization, final latency stamping.
rtl/pl_core.sv	Top-level strategy wrapper, AXI-Lite config registers, AXI-Stream interfaces.
scripts/fpga/build_pynq_z2.tcl	Vivado block design: AXI DMA, clock converters, AXI interconnect, PS/PL clocks.

Clocking And Transport

• FPGA strategy clock: 166.67 MHz, about 6 ns per cycle.
• DMA/control side uses the PS clock domain and AXI DMA.
• AXI clock converters bridge market-data ingress and order egress across PS/PL clock domains.
• Taker and maker orders use separate AXI-Stream order lanes.
• The order frame is 32 bytes and carries action, type, venue, symbol, price, quantity, trigger timestamp, trigger sequence, strategy cycles, OMS cycles, and total FPGA cycles.

Strategy Logic

The RTL contains two taker sources:

• ES-band taker: compares ETF bid/ask levels against a fixed-point ES-implied reference adjusted by an EMA premium.
• Cross-venue lock detector: tracks per-symbol best bid/ask across venues and emits paired IOC legs when the book is locked/crossed.

The maker path emits protected limit quotes and cancel/new events. The current server simulator tracks maker resting orders separately and fills them from MBO trade prints.

The order path is deliberately simple and inspectable:

• prices are raw 1e-9 scaled integers,
• state and thresholds are AXI-Lite configurable,
• OMS has a taker priority lane,
• simple risk gates reject impossible/oversized takers,
• latency fields are stamped in RTL and carried to the server log.

Latency Theory

At 166.67 MHz:

1 cycle ~= 6 ns
4 cycles ~= 24 ns

The log has multiple latency fields because one number would hide the real bottleneck.

Field	Definition	What It Answers
fpga_strategy_ns	Market trigger beat to strategy order-valid.	How fast did the RTL strategy decide?
fpga_oms_ns	Strategy order-valid to order transmitter accept.	How much time was spent in OMS/risk/egress wait?
fpga_ns	Market trigger beat to final FPGA order egress stamp.	How long was the order resident inside FPGA logic/egress?
server_path_ns	Server replay send time to server receive time for the returned order.	What did the whole PYNQ/Linux/client/TCP/server path cost?
non_fpga_ns	server_path_ns - fpga_ns.	How much of end-to-end latency was outside FPGA?
maker_rest_ns	Maker quote arrival to observed trade fill.	Quote lifetime/resting time, not a taker latency metric.

The key point: fpga_strategy_ns is the strategy datapath. fpga_ns also includes OMS/egress waiting. On PYNQ-Z2, the S2MM DMA path is simple-mode and small order frames can be delayed by re-arming, AXI ready backpressure, and bursty order output. That is why the minimum and median are nanoseconds while the p99/p99.9 tail can be milliseconds.

fpga_oms_ns is stored as a saturated 16-bit cycle count:

65535 cycles * 6 ns = 393,210 ns

So any OMS/egress wait beyond about 393 us is visible in fpga_ns, while the separate fpga_oms_ns field clamps at 393,210 ns.

Current Latency Measurement

Source: current full-day fills.log in this repository.

Taker Latency Percentiles

All values below are from taker fills only.

Metric	Samples	Min	Avg	P50	P90	P99	P99.9	Max
Strategy decision, fpga_strategy_ns	3,983,085	6 ns	39.2 ns	42 ns	42 ns	42 ns	42 ns	378 ns
OMS/egress, fpga_oms_ns	3,983,085	18 ns	9.429 us	42 ns	4.404 us	393.210 us	393.210 us	393.210 us
Total FPGA, fpga_ns	3,983,085	24 ns	75.905 us	84 ns	4.446 us	2.653 ms	9.906 ms	29.217 ms
Full server path, server_path_ns	3,983,085	376.819 us	11.748 ms	9.354 ms	21.807 ms	41.416 ms	68.895 ms	3.737 s
Non-FPGA path, non_fpga_ns	3,983,085	374.052 us	11.672 ms	9.308 ms	21.777 ms	41.370 ms	68.783 ms	3.737 s

Interpretation

The minimum total FPGA latency is exactly the expected value:

4 cycles * 6 ns = 24 ns

The median total FPGA latency is 84 ns, so the common filled-order path is still only 14 cycles on the 166.67 MHz strategy clock. The fastest path still appears as the expected 4-cycle case, while the strategy decision field sits at 42 ns from p50 through p99 on this full-day run.

The high average/tail of fpga_ns is not the strategy compare taking microseconds or milliseconds. It is total trigger-to-egress time, and it grows when the order transmitter cannot immediately hand the frame to the downstream AXI/DMA path. The p99/p99.9 outliers are a prototype egress artifact:

• simple-mode S2MM DMA on PYNQ,
• small 32-byte frames,
• re-arming/order-drain overhead on the ARM side,
• AXI-Stream tready backpressure,
• bursts of generated orders,
• userspace/Linux scheduling outside the PL.

The full server path is much larger because it includes market replay, PYNQ ARM userspace, DMA servicing, TCP forwarding, and server receive. It is useful for end-to-end replay accounting, but it should not be presented as the hardware decision latency.

Maker Quote Lifetime

Maker rest time is not taker latency. It measures how long a simulated passive quote sat before an MBO trade print crossed it.

Metric	Samples	Min	Avg	P50	P90	P99	P99.9	Max
Maker rest, maker_rest_ns	1,908	0 ns	2.247 s	29.219 ms	1.287 s	36.644 s	229.913 s	1168.437 s

Matching And PnL Model

The server uses MBO data for replay matching. The FPGA still consumes compact top-of-book style records because that is what the hardware strategy needs for fast decision logic. The server keeps the heavier MBO state where memory and software flexibility are available.

Current replay summary:

Metric	Value
Market events processed	334,377,768
Order frames received	22,543,367
Total fills	3,984,993
Taker fills	3,983,085
Maker fills	1,908
Fill/order rate	17.6770%
FPGA-time PnL	$652,861,858.0475
Strategy-time PnL	$652,864,817.6275
FPGA vs strategy PnL delta	$2,959.5800

Fill source attribution:

Source	Fills
taker/cross-venue	2,027,529
taker/es-band	1,955,556
maker	1,908

Important details:

• Market MBO actions add/modify/cancel external book state.
• Taker IOC orders preview crossed MBO liquidity for fills.
• Taker IOC orders do not remove historical external liquidity from the book.
• Maker quotes are tracked as our own resting state and filled from MBO trade prints.
• The log reports both FPGA-time PnL and strategy-time PnL:
  • pnl_fpga matches orders at trigger timestamp plus total FPGA cycles.
  • pnl_strategy matches the shadow path at trigger timestamp plus strategy cycles.

This lets the replay answer two different questions:

• How did the actual FPGA-output timing perform?
• What would the same signal have looked like if only the RTL strategy decision latency mattered?

Dashboard

After a run:

python3 scripts/dashboard/build_dashboard_data.py \
  --fills fills.log \
  --out dashboard/data/session.js

Then open:

dashboard/index.html

or serve it locally:

cd dashboard
python3 -m http.server 8080

The dashboard shows replay timeline, PnL, fills, per-symbol behavior, per-venue attribution, latency percentiles, and recent fill tape.

Build And Test

Run RTL tests:

scripts/run/run_sv_tests.sh

Build server and PYNQ client:

scripts/run/build_all.sh

The build is split by target. The replay server is compiled natively for the laptop/workstation, while the client is cross-compiled for the PYNQ-Z2 ARM core with the board sysroot. That keeps the hot FPGA-facing receiver binary ABI-compatible with the board, but still lets the heavier replay/matching server run on the development machine.

This produces:

• build/server/bin/hftarb_server
• build/arm/bin/hftarb_client

If the ARM cross-toolchain/sysroot is missing, the script still builds the server and reports what is missing for the PYNQ client.

Run A Replay

Network Setup

Use a direct Ethernet link or isolated switch between the laptop/server and the PYNQ board. Replace <laptop-eth-iface> with the interface name from ip link such as enp3s0 or eno1.

On the laptop/server:

sudo ip addr flush dev <laptop-eth-iface>
sudo ip addr add 192.168.2.1/24 dev <laptop-eth-iface>
sudo ip link set <laptop-eth-iface> up

On the PYNQ board:

sudo ip addr flush dev eth0
sudo ip addr add 192.168.2.99/24 dev eth0
sudo ip link set eth0 up

Check the link:

ping 192.168.2.99

On the server/laptop:

scripts/run/run_server.sh --mcast-iface 192.168.2.1 --session market-open

On the PYNQ path:

scripts/run/run_client.sh --host 192.168.2.99 --mcast-iface 192.168.2.99

The client script deploys the ARM binary, calibration registers, bitstream/HWH, loads the FPGA through PYNQ Overlay(), and starts the receiver/order forwarder.

MBO Data

Download the test-day MBO data with a Databento API key:

export DATABENTO_API_KEY=...
python3 scripts/data/download_test_mbo.py --estimate-only
python3 scripts/data/download_test_mbo.py

Convert DBN MBO files into the binary format used by the server:

python3 scripts/data/mbo_to_bin.py data/test_mbo

The server expects converted *_mbo_*.bin files under data/test_mbo.

Repository Layout

Path	Purpose
rtl/	SystemVerilog strategy core, OMS, lock detector, AXI-Lite wrapper.
rtl/tb/	Self-checking Icarus Verilog testbenches.
client/	PYNQ-side UDP receiver, DMA transport, order forwarder.
server/	Replay server, MBO matcher, order receiver, fill logger.
include/	Shared packet/config/util headers.
scripts/run/	Build, server run, client deploy/run, RTL test scripts.
scripts/data/	Databento download and DBN-to-binary conversion scripts.
scripts/dashboard/	fills.log to static dashboard dataset builder.
dashboard/	Static HTML/CSS/JS trader dashboard.
docs/images/	README screenshots and board photo.
bitstream/	Current PYNQ-Z2 bitstream, HWH, and XSA.
third_party/orderbook/	Order book engine used behind the MBO simulator.

How To Present This Project

Strong claims:

• Real FPGA datapath, not only a software simulator.
• Nanosecond-level strategy decision instrumentation stamped in RTL.
• Clear separation of hardware latency, OMS/egress latency, host path latency, and replay matching latency.
• Realistic MBO replay/matching path instead of a naive L1-only fill model.
• Professional post-run dashboard and log pipeline.
• Honest identification of the bottleneck: PYNQ/simple-mode DMA and host path, not the strategy compare logic.

Claims to avoid:

• Do not present the replay PnL as live tradable alpha.
• Do not claim colocated gateway performance from a PYNQ/Linux/TCP prototype.
• Do not call server_path_ns FPGA latency. It is end-to-end replay path latency and is dominated by non-FPGA components.

Known Limitations

• PYNQ-Z2 is a development board, not a production NIC-attached trading FPGA.
• The current egress path uses simple-mode DMA and small order frames, which creates queue/backpressure tails under bursts.
• Userspace Linux and TCP forwarding are intentionally visible in the measurement; they are not hidden or smoothed out.
• The matching simulator is MBO-based but still not an exchange-grade model: queue priority, fees, hidden liquidity, cancels racing our order, and exchange gateway behavior are simplified.
• Current strategy behavior can accumulate inventory; the project is a low-latency architecture and measurement prototype, not a production risk engine.

Target Hardware

• Board: PYNQ-Z2
• FPGA: Zynq-7020 xc7z020clg400-1
• Tool: Vivado 2024.1
• Top: system_wrapper
• Bitstream: bitstream/hftarb_pynqz2.bit