A Kafka-compatible message broker in a single Go binary.
Your disk plus any S3 bucket. No ZooKeeper, no KRaft, no JVM, no cluster to operate.
Real Kafka clients connect to it. kcat, kafka-python, KafkaJS, franz-go, Sarama,
confluent-kafka, and the Apache Kafka Java client all speak to it without a shim, a
proxy, or a patched driver, because it implements the Kafka wire protocol rather than
imitating it.
It is one process and one directory. There is no coordination service, no broker cluster, no separate schema component, and nothing to install before it runs.
┌──────────────┐ Kafka wire protocol ┌──────────────────────────────────┐
│ any Kafka │ ───────────────────────► │ kafka-wire │
│ client, any │ ◄─────────────────────── │ wire → log → segments on disk │
│ language │ └──────────────┬───────────────────┘
└──────────────┘ sealed + old │
▼
┌──────────────────────────────────┐
│ cold tier (optional) │
│ none · a directory · any S3 │
│ AWS MinIO R2 B2 Ceph Wasabi … │
└──────────────────────────────────┘
docker run -p 9092:9092 -v kw:/data \
-e KAFKA_WIRE_AUTH_ALLOWANON=true \
ghcr.io/csnyder256/kafka-wire:latestThat is the whole setup. Now use it from anywhere:
# with the binary's own client, no libraries to install
kafka-wire topic create demo
echo hello | kafka-wire produce demo
kafka-wire consume demo --from-beginning# or with a normal Kafka library
from kafka import KafkaProducer, KafkaConsumer
KafkaProducer(bootstrap_servers="localhost:9092").send("demo", b"hello")
for msg in KafkaConsumer("demo", bootstrap_servers="localhost:9092",
auto_offset_reset="earliest"):
print(msg.value)No configuration file was written, and nothing was configured. Run
kafka-wire config init when you want one.
- One binary.
kafka-wire serveis the broker. The same binary is also the producer, the consumer, the topic admin, and the diagnostic tool. - Kafka on the wire. 19 protocol APIs, consumer groups, SASL/SCRAM, TLS, ACLs, and Prometheus metrics. Your existing client library and your existing code work.
- Opaque to your data. Keys, values and headers are bytes. JSON, Avro, Protobuf, MessagePack, CBOR, images, PDFs, gzip blobs, and invalid UTF-8 all round-trip byte-identically, and the test suite asserts exactly that.
- Cold storage that is not AWS-shaped. Sealed segments can tier to a directory or to any S3-compatible store, and consumers fetching old offsets get them transparently restored.
This is the section to read before adopting it.
- Not replicated. One node, one copy. If the disk dies, the data that had not been archived dies with it. There is no failover and no quorum. See docs/durability.md for what that does and does not cost you.
- Not a Kafka feature replica. No transactions, no exactly-once semantics, no Kafka Streams, no Kafka Connect runtime, no log compaction, no KRaft.
- Not a distributed system. If you need multi-node durability or throughput beyond one machine, run Apache Kafka or Redpanda. That is not a hedge; it is the correct answer for that requirement.
If your workload fits on one machine and you would rather own it than operate a cluster, this is built for you. If it does not, it is not.
Yes, with one caveat that has a one-line fix.
kafka-wire has no transaction coordinator, so it does not implement
InitProducerId. Any client that turns idempotent producing on by default
needs it switched off. That currently means the Apache Kafka Java client
(default since 3.0) and kafka-python 3.x:
enable.idempotence=false # Java, Spring, Quarkus, MicronautKafkaProducer(..., enable_idempotence=False) # kafka-python 3.xClients built on librdkafka default it off already, as do kafka-python 2.x, KafkaJS and Sarama. franz-go turns it on by default but degrades gracefully when the broker does not advertise the API, so it works either way. Everything else works unchanged.
What you give up is producer-side deduplication of retries. It is not emulated,
because a broker that accepted InitProducerId and then ignored sequence
numbers would be claiming a guarantee it does not provide.
| Client | Works | Note |
|---|---|---|
kcat / kafkacat |
yes | |
| kafka-python, most clients | yes | verified in CI |
| confluent-kafka (Python, Go, .NET), librdkafka, node-rdkafka | yes | idempotence already defaults off |
| KafkaJS | yes | |
| franz-go | yes | verified in CI |
| Sarama, segmentio/kafka-go | yes | |
| Apache Kafka Java client 3.x / 4.x | yes | enable.idempotence=false |
| Spring for Apache Kafka, Quarkus, Micronaut | yes | same producer setting |
| Kafka Connect, Debezium | partial | source connectors work; anything requiring transactions does not |
| Kafka Streams, ksqlDB | no | needs transactions and internal topic management |
Yes, and none of them are a special case. The cold tier is one interface with three
implementations: none, fs, and s3. The same conformance suite runs against the
filesystem backend and against a live S3 server in CI, so they cannot drift.
archive:
backend: s3
s3:
bucket: my-bucket
endpoint: http://minio:9000 # omit entirely for AWS
addressing: path # MinIO, Ceph and SeaweedFS want path| Store | Endpoint | Region | Addressing |
|---|---|---|---|
| AWS S3 | omit | your real region | auto |
| MinIO | http://host:9000 |
us-east-1 |
path |
| Cloudflare R2 | https://ACCOUNT.r2.cloudflarestorage.com |
auto |
auto |
| Backblaze B2 | https://s3.REGION.backblazeb2.com |
e.g. us-west-004 |
auto |
| Ceph RADOS Gateway | your gateway URL | site-defined | path |
| SeaweedFS | http://host:8333 |
us-east-1 |
path |
| Wasabi | https://s3.REGION.wasabisys.com |
e.g. us-east-1 |
auto |
| Garage | your node URL | operator-defined | path |
| Google Cloud Storage | https://storage.googleapis.com |
auto |
auto |
| Hetzner, OVH, Scaleway, DigitalOcean Spaces, Storj, Tigris | provider URL | provider region | auto |
| A directory, an NFS mount, a NAS | backend: fs |
Credentials come from the config, or from the standard AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY variables that every S3-compatible vendor documents, or
from the instance credential endpoint. Nothing here assumes an AWS account.
There is a deliberate engineering choice behind this: the S3 driver is built on
minio-go, not the AWS SDK. Since aws-sdk-go-v2/service/s3 v1.73.0 the AWS SDK
computes a checksum on every upload by default and switches the request to a
streaming aws-chunked trailer. Stores catch up at their own pace, and several
did not implement it for a long time. Rather than track which release of which
provider is currently compatible, the driver simply never sends the trailer.
No. There is nothing to elect, because there is one node. That is the entire tradeoff of this project stated in one sentence.
Anywhere that gives you a persistent disk and a raw TCP port.
| Docker / Compose | deploy/docker |
| Kubernetes | deploy/kubernetes (StatefulSet, PVC, probes, PDB) |
| systemd on a VM | deploy/systemd (hardened unit) |
| Nomad | deploy/nomad |
| Fly.io, Railway, Render, Koyeb, Hetzner, EC2 | deploy/paas |
| Raspberry Pi / ARM64 | multi-arch images and binaries |
Platforms with an ephemeral filesystem or HTTP-only routing (Cloud Run, Heroku,
DigitalOcean App Platform, Vercel) cannot run this, and
deploy/paas says so plainly rather than letting you find
out in production.
The one setting people get wrong everywhere: a Kafka client connects to your bootstrap address, then throws it away and reconnects to whatever the broker advertises. Behind NAT, a port mapping, or a load balancer, set:
listeners:
advertisedhost: kafka.example.com
advertisedport: 9092Otherwise the client connects, gets told to go to an address that means something
different on its side of the network, and hangs. kafka-wire doctor prints the
address clients will be given, so you can check it before your users do.
kafka-wire starts with no configuration at all. When you want some, there is one canonical file and a mechanical mapping to environment variables:
storage.datadir -> KAFKA_WIRE_STORAGE_DATADIR
archive.s3.bucket -> KAFKA_WIRE_ARCHIVE_S3_BUCKET
Uppercase the dotted path, replace . with _, prefix with KAFKA_WIRE_.
Environment beats file, because the file is usually baked into an image and the
environment is what your orchestrator injects. Secrets can come from
KAFKA_WIRE_ADMIN_TOKEN_FILE style indirection so they never appear in a process
listing.
kafka-wire config init # a starter file, twelve lines
kafka-wire config reference # every setting, its type, default, and meaning
kafka-wire config print # what is in effect, and where each value came from
kafka-wire config validate # fail before you deploy, not after
kafka-wire doctor # check ports, disk, advertised address, object storeThere is no giant .env.example listing every possible integration. That
pattern gets copied wholesale, and settings nobody chose end up in production.
config print shows the resolved value and its source for every setting instead,
which is the question people actually have.
Bad values are startup errors that name the variable at fault, not silent fallbacks to a default. Several configurations that would corrupt or lose data are refused outright: a segment-to-part ratio above the 10,000-part limit every S3 store enforces, an archive retention shorter than the archive age, half-configured TLS, and a non-loopback listener with authentication disabled.
Full reference: docs/configuration.md.
Advertised via ApiVersions, so clients negotiate correctly rather than discovering
gaps at runtime.
| API | Versions | API | Versions | |
|---|---|---|---|---|
| Produce | 3-9 | JoinGroup | 0-7 | |
| Fetch | 4-11 | SyncGroup | 0-5 | |
| ListOffsets | 0-5 | Heartbeat | 0-4 | |
| Metadata | 0-9 | LeaveGroup | 0-4 | |
| OffsetCommit | 0-8 | FindCoordinator | 0-3 | |
| OffsetFetch | 0-6 | DescribeGroups | 0-5 | |
| CreateTopics | 0-6 | ListGroups | 0-4 | |
| DeleteTopics | 0-5 | DescribeConfigs | 0-4 | |
| ApiVersions | 0-3 | SaslHandshake / SaslAuthenticate | 0-1 / 0-2 |
Anything not in that table is not implemented. Notably absent: InitProducerId,
AddPartitionsToTxn, EndTxn and TxnOffsetCommit (no transactions),
DeleteGroups, CreatePartitions, OffsetForLeaderEpoch, DescribeLogDirs,
DescribeCluster, AlterConfigs, IncrementalAlterConfigs, DeleteRecords,
OffsetDelete, and the ACL and SCRAM admin APIs. An unimplemented API returns a
correctly-typed response carrying UNSUPPORTED_VERSION, so a client fails
cleanly instead of desynchronizing.
The default fsyncmode: interval acknowledges a write once the operating system has
it, and fsyncs on a timer. This is the same tradeoff Apache Kafka makes by default,
where safety comes from replication instead. kafka-wire has no replication, so:
- A process crash loses nothing. The log is recovered and validated on restart.
- A machine crash can lose the last few seconds of writes. Set
fsyncmode: alwaysto trade throughput for that window. - A disk failure loses everything not yet archived. The cold tier is the answer: archived segments carry a SHA-256 that is verified on every restore.
Do not use this as the only copy of data you cannot regenerate. docs/durability.md has the full accounting.
Measured on the machine below with the command below. Reproduce it on yours:
go test ./e2e/ -run XXX -bench Throughput -benchtime 1x| Payload | Records/s | Throughput |
|---|---|---|
| 256 B | ~2,200,000 | ~537 MiB/s |
| 4 KiB | ~180,000 | ~704 MiB/s |
4 concurrent producers, 200,000 records per run, incompressible random payloads,
compression disabled, default fsyncmode: interval, loopback, on an Intel i9-11900K
with NVMe storage running Windows.
Read that as a shape, not a capacity plan: these are page-cache-backed writes on one fast machine with no consumers competing, and your disk, network and payload will dominate. The point is that a single-node broker is not the bottleneck for the workloads it targets.
I built this as ClarusStream, the message broker for the Clarus contract platform, to replace a managed Kafka service that cost more than the rest of the infrastructure combined. It has carried that platform's production traffic since May 2026.
This repository is a rebuilt, vendor-neutral version of that broker: the proven protocol and storage core, with every assumption about my stack removed and replaced with a choice. Making it general surfaced real defects that a single-client deployment never exercised, including a request-header parsing bug that broke every modern client, a response field whose Go zero value silently redirected consumers to a nonexistent broker, and a consumer-group coordinator that delivered every record to every member of a group. All three are fixed here and covered by tests.
| Getting started | first broker, first message |
| Configuration | every setting |
| Cold storage | object stores, tiering, restore |
| Security | SASL, TLS, ACLs, exposure |
| Durability | what survives what |
| Operations | metrics, backup, upgrades, troubleshooting |
| Examples | Go, Python, Node, Java, shell |
| Adversarial harness | the attack loop that runs against every push |
Beyond the usual tests, the repository ships the harness that was built to
attack its private ancestor, in chaos/. It generates a randomized
universe of principals, topics, groups and ACL grants, runs baseline traffic
through it, and then tries to break out: reading another principal's topic,
joining its consumer group, replaying its offsets, brute-forcing the admin
API, corrupting metadata mid-read, and feeding the wire protocol malformed
frames. A successful breakout halts the run and dumps a forensic snapshot with
a seed that reproduces the exact topology.
pip install -r chaos/requirements.txt
bash chaos/run.sh 120 4It runs nightly rather than on every push, because the daemon does not yet
exit reliably on its own duration; chaos/README.md says so plainly. Building
it is what surfaced the fact that
authorization was enforced on Produce and Fetch but not on the topic-admin
APIs, so a principal with no grants could enumerate and delete every topic in
the cluster. That is fixed, and invariant 9 is why.
Issues and pull requests are welcome. See CONTRIBUTING.md. Security reports: SECURITY.md.
MIT. See LICENSE.
Apache Kafka is a registered trademark of the Apache Software Foundation. kafka-wire is an independent project, is not affiliated with or endorsed by the Apache Software Foundation, and the name is used only to describe the wire protocol this software is compatible with.