This workload is intended for use with any system that uses the Apache Kafka protocol to verify common Kafka properties that should always hold true, even in the presence of faults.
This workload uses the rdkafka client library; as of August 2025, it's using version 0.36.2, which is compatible with librdkafka v1.9.2+.
As of August 2025, this workload expects to run against a 3-node Kafka cluster. The cluster configuration (set via the producer) strives for the strongest consistency guarantees- the number of in-sync replicas and the replication factor are both 3, with the producer expected acknowledgements from all nodes to consider a write to be successful.
As you can see in the image's entrypoint, the workload executable will be executed some number of times in the background. The validation executable will be continually executed as well.
The workload does a series of producing and consuming against the cluster.
Each invocation of the workload will create a random number of producers. Each producer produces a sequence of messages to random test topics, with configurable, random delay between, retrying each send until its successful. Producers update shared global state to track topic offsets. All the while, it's logging it's successful attempts.
Each invocation of the workload will create a random number of consumer groups with a random number of consumers in each. Each consumer joins a group; subscribes to topics; handles rebalances (partitions assigned/revoked). It then polls for messages, with configurable processing delays and commits offsets (manual or auto).
Consumers similarly updates global state so the test harness knows which offsets have been consumed and then stop consuming once all producer messages have been read (or if no messages were ever written). Similar to the producer, the consumer logs structured events at every step.
Based on the logged output from the producers and consumers, a series of validation steps are performed. These validation steps are described in more detail below.
As of August 2025, there are five properties being asserted via the Antithesis SDK:
- There are no "message integrity" violations (specifically, there are no "lost" messages or messages that were read but never written).
- Messages don't change partitions.
- Previously-committed consumer offsets are never seen again.
- The producer doesn't "double-write" messages.
- Sequential messages are in sequential offsets.
This workload is intended to run in any environment with a Kafka-compatible cluster. Try it locally with your own Kafka-compatible system!
You can build the workload image from within the workload directory by running the following command:
docker build . -t antithesis-kafka-workload:latestYou can then run the entire Kafka cluster from within the config directory by running the following command:
docker-compose up -d