DriftQ-Core is a durable message broker (v1) that also contains the DriftQ v2 foundations: a replayable workflow runtime with a persistent run/event log, deterministic DAG scheduling, and debugging primitives.
- v1 (stable): broker API under
/v1/*(produce/consume/ack/nack, topics, leases). - v2 foundations (evolving): workflow runtime exposed via
/debug/*anddriftqctl runs ...(replay, timelines, diffs, rollback primitives).
If you only want the broker, you can ignore v2. If you want "Temporal-like" durability + replay, v2 is where this is going. 🙂
Building reliable backend systems today means choosing between two painful options:
Option A: Managed infrastructure overkill. You need a message queue, so you spin up Kafka (plus ZooKeeper or KRaft), or RabbitMQ (plus Erlang cluster management), or pay for SQS/Pub-Sub. You need durable workflows, so you add Temporal (plus a Cassandra or PostgreSQL cluster). Suddenly your "simple pipeline" requires 4+ services, a Kubernetes cluster, and a platform team to keep it all running. Most of the time, your actual workload is a few hundred messages per second — nowhere near justifying this complexity.
Option B: Roll your own. You wire up Redis lists, cron jobs, and Postgres-as-a-queue hacks. It works until it doesn't: messages get lost during deploys, retry logic is scattered across 15 files, debugging a failed pipeline means grepping through logs from three different services, and "replay that failed job from step 3" is a fantasy.
Neither option is great when you're a small team shipping fast, or when you're building AI agent pipelines where the real complexity is in the logic — not the plumbing.
Single binary. Zero external dependencies. DriftQ-Core is one Go binary that gives you both a Kafka-style message broker and a Temporal-style workflow runtime. No ZooKeeper. No etcd. No Redis. No separate database. You run one process, it writes to one WAL file, and you're done. docker run and you have durable messaging + workflow orchestration in seconds.
Built for the workloads most teams actually have. Not every project needs to process a million messages per second. Most teams need reliable delivery for a few hundred to a few thousand messages per second, with proper retries, dead-letter queues, and the ability to see what went wrong when something breaks. DriftQ-Core is built for exactly that sweet spot — where you need real durability guarantees without the operational tax of distributed infrastructure.
Designed for AI and agent workflows from day one. The v2 runtime isn't a generic workflow engine that happens to work for AI — it was built with AI pipelines in mind. Budget controls track tokens and dollars across a run so a runaway agent can't burn through your OpenAI bill. Concurrency throttles prevent "500 parallel embedding calls" accidents. Replay lets you re-run a pipeline from step 3 without re-calling the expensive LLM steps that already succeeded. Artifacts store large intermediate outputs (embeddings, generated documents) without bloating your event log.
Debuggable by default. Every run produces an append-only event log. Every step records its input, output, timing, and attempt number. You can inspect a failed run, diff two attempts of the same step, time-travel replay to reproduce issues, and see exactly where your budget was spent — all through the CLI or HTTP API. No more "what happened to that job?" mysteries.
- Small teams building AI/LLM pipelines who need durable execution without managing Temporal + Kafka + PostgreSQL
- Backend developers who want a lightweight message broker with proper retry semantics, DLQ routing, and consumer groups — without running a Kafka cluster
- Solo developers and startups who need production-grade messaging and workflow orchestration that runs on a single $5/month VPS
- Anyone tired of gluing together 5 services to get reliable message processing with retry and observability
- It's not a distributed system (yet). It runs as a single process with file-based durability. If you need multi-node replication and horizontal scaling today, use Kafka + Temporal.
- It's not a general-purpose database. The WAL is append-only and optimized for message/event storage, not arbitrary queries.
- It's not trying to replace Kafka at 10 million messages per second. It's built for the 99% of workloads that don't need that scale.
- Topics / partitions (Kafka-style offsets)
produce, streamingconsume(NDJSON),ack/nack- Consumer groups with round-robin dispatch
- Consumer leases (
lease_ms) with automatic redelivery - Idempotency keys (at-least-once with dedupe)
- Retry policies with exponential backoff
- Dead Letter Queue (DLQ) routing (with DLQ-of-DLQ prevention)
- Backpressure via configurable partition buffer limits
- Configurable broker limits (max partition bytes, max partition messages, max in-flight)
- WAL-backed durability
- Prometheus metrics
- Run contract + append-only run/event log (inspectable execution history)
- Deterministic DAG engine (step dependencies, fan-out/fan-in, retries)
- Validates against duplicate/empty node IDs at spec parse time
- Replay controls
- time-travel replay: reuse recorded outputs/artifacts (don't re-run expensive steps)
- live replay: re-execute from a chosen step
- Durable delay primitive (timers + resume loop after restart)
- Artifact store + replay cache (store big outputs, reuse on replay)
- Budget/throttle controls (max attempts, tokens, dollars, wallclock timeout)
- Debug endpoints for inspection/control (run state, timelines, diffs, replay)
- Minimal rollback primitive via an "active index" pointer (promote/rollback)
- Handler panic recovery (panicking handlers do not crash the server)
Recommended (pinned tag):
docker run --rm -p 8080:8080 -v driftq_data:/data ghcr.io/driftq-org/driftq-core:1.2.0Development / tracks main:
docker run --rm -p 8080:8080 -v driftq_data:/data ghcr.io/driftq-org/driftq-core:latestThen hit:
curl http://127.0.0.1:8080/v1/healthzTip: In production, pin the image tag (reproducible deploys).
latestis for dev.
Using docker-compose:
docker-compose up -dDocker with custom flags:
docker run --rm -p 8080:8080 -v driftq_data:/data ghcr.io/driftq-org/driftq-core:1.2.0 \
-addr :8080 \
-wal /data/driftq.wal \
-engine-store file \
-engine-wal /data/engine.wal \
-artifacts-dir /data/artifacts \
-max-partition-bytes 8388608 \
-max-inflight 4 \
-log-level info \
-log-format jsonBuild from source:
go build -o driftqctl ./cmd/driftqctl
./driftqctl --helpv1 broker examples:
# List topics
./driftqctl --base-url http://127.0.0.1:8080 topics list
# Create a topic
./driftqctl --base-url http://127.0.0.1:8080 topics create --name my-topic --partitions 4
# Peek at messages
./driftqctl --base-url http://127.0.0.1:8080 topics peek --topic my-topicv2 foundations examples:
# List runs
./driftqctl --base-url http://127.0.0.1:8080 runs list
# Get run status
./driftqctl --base-url http://127.0.0.1:8080 runs status --run-id <RUN_ID>
# View timeline
./driftqctl --base-url http://127.0.0.1:8080 runs timeline --run-id <RUN_ID>
# Time-travel replay (reuse recorded outputs)
./driftqctl --base-url http://127.0.0.1:8080 runs replay --run-id <RUN_ID> --from-step <STEP_ID> --mode time-travel
# Live replay (re-execute steps)
./driftqctl --base-url http://127.0.0.1:8080 runs replay --run-id <RUN_ID> --from-step <STEP_ID> --mode live
# Cancel a run
./driftqctl --base-url http://127.0.0.1:8080 runs cancel --run-id <RUN_ID> --reason "stopping"
# View artifacts
./driftqctl --base-url http://127.0.0.1:8080 runs artifacts --run-id <RUN_ID>
# Start a demo run
./driftqctl --base-url http://127.0.0.1:8080 runs demoAll stable broker endpoints are under /v1/*:
| Method | Endpoint | Description |
|---|---|---|
| GET | /v1/healthz |
Health check |
| GET | /v1/version |
Version info |
| GET | /v1/topics |
List topics |
| POST | /v1/topics?name=T&partitions=N |
Create topic |
| POST | /v1/produce |
Produce message (JSON body or query params) |
| GET | /v1/consume?topic=T&group=G |
Streaming consume (NDJSON) |
| POST | /v1/ack |
Acknowledge message |
| POST | /v1/nack |
Negative acknowledge (trigger retry) |
| GET | /metrics |
Prometheus metrics |
Full reference: docs/v1/v1-README.md
These endpoints are under /debug/* and are meant for development, demos, and iteration:
| Method | Endpoint | Description |
|---|---|---|
| POST | /debug/run-spec |
Start a run from JSON spec |
| GET | /debug/runs |
List all runs |
| GET | /debug/run?run_id=ID |
Get run details |
| GET | /debug/run-state?run_id=ID |
Get run state |
| POST | /debug/run-replay |
Time-travel or live replay |
| POST | /debug/run-cancel |
Cancel a run |
| GET | /debug/run-artifacts?run_id=ID |
List run artifacts |
| GET | /debug/artifact-meta?run_id=ID&node_id=N |
Artifact metadata |
| GET | /debug/artifact-get?run_id=ID&node_id=N |
Download artifact |
| POST | /debug/run-demo |
Start demo workflow |
| GET | /debug/index/active |
Get active index pointer |
| POST | /debug/index/promote |
Promote index pointer |
| POST | /debug/index/rollback |
Rollback index pointer |
| GET | /debug/topics |
List topics (debug) |
| GET | /debug/topics/peek |
Peek topic messages |
| GET | /debug/topics/lag |
Consumer lag info |
| GET | /debug/metrics |
Engine metrics |
Full reference: docs/v2/v2-README.md
go run ./cmd/driftqdServer flags:
| Flag | Default | Description |
|---|---|---|
--addr |
:8080 |
HTTP listen address |
--wal |
driftq.wal |
Path to broker WAL file |
--reset-wal |
false |
Reset WAL by moving existing file aside (creates a .bak.<ts> file) |
--engine-store |
memory |
Engine store: memory or file |
--engine-wal |
driftq.engine.wal |
Path to engine WAL (when --engine-store=file) |
--artifacts-dir |
driftq.artifacts |
Artifact store directory (empty = in-memory) |
--log-level |
info |
Log level: debug, info, warn, error |
--log-format |
text |
Log format: text or json |
--max-partition-bytes |
0 |
Max bytes buffered per partition (0 = broker default: 4 MB) |
--max-partition-msgs |
0 |
Max messages buffered per partition (0 = broker default: 100) |
--max-inflight |
0 |
Max in-flight messages per (topic, group, partition) (0 = broker default: 2) |
Broker defaults (when flags are 0 or omitted):
| Limit | Default | Description |
|---|---|---|
| Max partition bytes | 4 MB | Per-partition byte buffer limit (backpressure) |
| Max partition messages | 100 | Per-partition message count limit (backpressure) |
| Max in-flight | 2 | Max unacknowledged messages per (topic, group, partition) |
Examples:
# Default (in-memory engine, file-based broker WAL)
go run ./cmd/driftqd
# Custom address and log level
go run ./cmd/driftqd --addr :9090 --log-level debug
# JSON structured logging (for production / log aggregation)
go run ./cmd/driftqd --log-format json
# Durable engine with file-based storage
go run ./cmd/driftqd --engine-store file --engine-wal ./data/engine.wal --artifacts-dir ./data/artifacts
# Tune broker backpressure limits
go run ./cmd/driftqd --max-partition-bytes 8388608 --max-partition-msgs 500 --max-inflight 4
# Reset WAL on startup (safe: moves old WAL to .bak)
go run ./cmd/driftqd --reset-walUnit tests:
go test ./... -count=1Integration tests (requires the integration build tag):
# Basic integration test run
go test -tags=integration ./... -count=1
# Multiple iterations to catch flaky behavior
go test -tags=integration ./... -count=5
# Race detection + shuffled test ordering (recommended for CI)
go test -tags=integration ./... -race -count=1 -shuffle=on
# Stress test: race + shuffle + multiple iterations
go test -tags=integration ./... -race -count=5 -shuffle=onUsing gotestsum (nicer output, recommended):
go run gotest.tools/gotestsum@latest --format pkgname -- -count=1 -tags=integration ./...A load test script is included at scripts/loadtest.sh. It uses hey to measure produce throughput, burst handling, and sustained load:
# Start the server first
go run ./cmd/driftqd &
# Run load tests (defaults: 100 req/s for 60s)
./scripts/loadtest.sh
# Custom rate and duration
RATE=500 DURATION=120 ./scripts/loadtest.sh
# Against a remote server
BASE_URL=http://remote:8080 ./scripts/loadtest.shThe load test covers: produce throughput, burst testing (500 concurrent), sustained low-rate load, health check throughput, and concurrent workflow demo execution.
go build -o driftqd ./cmd/driftqd
go build -o driftqctl ./cmd/driftqctlBuild with version info:
go build -ldflags "-X main.buildVersion=1.2.0 -X main.buildCommit=$(git rev-parse --short HEAD)" -o driftqd ./cmd/driftqdWAL is forward-compatible only: once you write WAL entries with newer ops, you can't safely downgrade to an older binary that doesn't understand them.
- Message Queue MVP (Completed ✅)
- Replayable Workflow Runtime (Completed ✅)
- Multi-Agent Runtime & Real-Time AI
- DriftQ Cloud
DriftQ-Core/
├── cmd/
│ ├── driftqd/ # Server binary
│ └── driftqctl/ # CLI client
├── internal/
│ ├── broker/ # v1 broker core (dispatch, redelivery, idempotency, DLQ)
│ ├── engine/ # v2 workflow runtime (runner, DAG, replay, artifacts, timers)
│ ├── storage/ # WAL implementation
│ └── httpapi/ # HTTP types and helpers
├── scripts/
│ ├── loadtest.sh # Load testing script (uses hey)
│ └── demo_v1.ps1 # PowerShell demo script
├── docs/
│ ├── v1/ # Broker documentation
│ └── v2/ # Workflow runtime documentation
├── docker-compose.yml # Local development
├── Dockerfile # Multi-stage distroless build
└── Makefile
For copy/paste starter repos and runnable demos, see DriftQ-Starters:
- GitHub: driftq-org/DriftQ-Starters
- v1 broker docs:
docs/v1/v1-README.md - v2 foundations docs:
docs/v2/v2-README.md
See LICENSE.