distributed-log-pipeline

A distributed pipeline for moving high volumes of log and event records across services: producers write records, a partitioned log carries them, and a set of C++ consumer services process, transform, and forward them, with ordering, replication, and delivery semantics handled deliberately. It runs as containerized services on Kubernetes and is built to keep the data path fast and predictable under load.

The interesting part is not "read a log line" but the distributed-systems side: partitioning, ordering guarantees, consumer coordination, and what happens when a service dies mid-stream.

What it does

• Producers publish records into a partitioned, replicated log.
• A partitioned commit log (Kafka) carries records durably, with partitions for parallelism and replication for durability.
• C++ consumer services read from the log in consumer groups, process records (parse, enrich, aggregate), and forward results downstream.
• Coordination: partitions are balanced across consumer instances; offsets are committed so processing survives restarts.
• Delivery semantics: at-least-once by default, with idempotent downstream writes so duplicates from a redelivery are harmless; ordering preserved per partition key.

The distributed-systems core

• Partitioning and ordering. Records are keyed so everything for a given key lands on the same partition and is processed in order. Cross-partition there is no global order by design; the key choice is what makes per-entity ordering hold.
• Consumer groups and rebalancing. Consumers share partitions in a group; when an instance joins or dies, partitions rebalance. Processing is written so a rebalance mid-flight does not drop or double-count committed work.
• Offset management. Offsets are committed after processing, not before, so a crash replays the in-flight record rather than skipping it. That is the at-least-once contract.
• Replication and durability. The log is replicated across brokers; a broker loss does not lose committed records. Producers wait for the required acks before a write counts.
• Idempotency downstream. At-least-once means occasional duplicates, so downstream writes are keyed and deduplicated and a replay is a no-op. This is what makes at-least-once safe without paying full exactly-once cost everywhere.
• Backpressure. If consumers fall behind, the log absorbs the lag (it is the buffer); consumers scale out horizontally rather than the producers stalling.

Running on Kubernetes

• Consumers as a scalable deployment - replicas map onto log partitions; scaling the deployment scales throughput up to the partition count.
• Health and readiness probes - a consumer reports ready only once it is connected and assigned partitions, so traffic and rebalancing behave during rollouts.
• Resource limits and requests - CPU/memory bounded so the scheduler places them sanely and one service cannot starve the node.
• Rolling updates - updates roll through consumers without stopping the pipeline; the group rebalances around instances as they cycle.
• Config and secrets - broker endpoints and credentials injected via config/secret rather than baked in.

Keeping the data path fast

• C++ for the hot path - per-record processing is C++, with attention to allocation, copying, and cache behavior on the busy path.
• Multithreaded consumers - each instance processes partitions concurrently, with ownership arranged so threads do not contend on shared state.
• Batching - records are fetched and committed in batches to amortize per-record overhead, sized against latency targets.
• Measured, not guessed - throughput (records/sec), end-to-end latency, and consumer lag are instrumented so the effect of batching, partition count, and replica count is visible.

Stack

C++17 for the producer/consumer services and the processing path, Apache Kafka (librdkafka) as the partitioned replicated commit log, Kubernetes for deployment, scaling, health, and rollout. Linux throughout, container images per service, benchmarks for throughput, latency, and lag.

Structure

distributed-log-pipeline/
├── producer/       C++ producer service (keyed publish, acks)
├── consumer/src/   consume.cpp (fetch, offsets, rebalance-safe loop)
│                   process.cpp (per-record hot path)
│                   forward.cpp (idempotent downstream write)
├── deploy/         Kubernetes manifests (deployments, probes, limits, config)
├── bench/          throughput / latency / lag measurement
└── CMakeLists.txt

Building and running

# build
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build

# produce
./build/producer/producer --brokers $KAFKA_BROKERS --topic records

# consume (one of N group members)
./build/consumer/consumer --brokers $KAFKA_BROKERS --group pipeline --topic records

# deploy
kubectl apply -f deploy/

# measure
./build/bench/throughput --brokers $KAFKA_BROKERS --topic records

Broker endpoints and credentials come from the environment (ConfigMap/Secret under Kubernetes), never from the source tree.

What it demonstrates

• Distributed systems in practice - partitioning, per-key ordering, consumer-group coordination, replication, and at-least-once with idempotent downstreams.
• Kafka - partitions, consumer groups, offset and delivery semantics, durability via replication and acks.
• Kubernetes - scaling consumers against partitions, probes that respect group membership, rolling updates without dropping the stream.
• C++ at scale - a multithreaded, allocation-aware processing path with batching, measured for throughput and latency.

Notes

• A reusable pipeline pattern; the record schema, the processing logic, and the downstream sink are what a concrete deployment supplies.
• Delivery semantics and batching are explicit knobs, so they can be tuned to a real workload's latency and durability needs rather than hard-coded.

MIT licensed.