DistKV

DistKV — a Raft-based, fault-tolerant, linearizable distributed key-value store.

What It Does

• Raft consensus — leader election with randomized timeouts, log replication, snapshotting + log compaction, joint-consensus membership changes
• Linearizable reads via ReadIndex — no log append per read; the leader confirms quorum via a heartbeat round then reads locally
• Durable log + state — bbolt-backed persistent log, HardState, and snapshot storage (internal/raftstore)
• gRPC service — inter-node Raft RPCs and client-facing KV RPCs on a single listener
• Leader-aware client — pkg/client transparently follows leader hints and retries with bounded exponential backoff
• Live TUI dashboard — cmd/distkv-tui renders a real-time cluster status table, highlights the leader, and accepts ad-hoc Put/Get commands
• Prometheus metrics + Grafana dashboard — role, term, commit lag, operation counts; provisioned in docker compose up
• Docker / Kubernetes / Helm — multi-stage distroless image, StatefulSet + headless service manifests, Helm chart
• CI/CD — GitHub Actions: test, vet, lint, image build + push to GHCR

---

Architecture

cmd/distkvctl          one-shot CLI (put / get / del / cas / status)
cmd/distkvd            daemon: one Raft node + gRPC server (KV + Raft RPCs)
cmd/distkv-tui         live terminal dashboard (tview)
cmd/distkv-bench       throughput / latency / failover benchmark

pkg/client             leader-aware gRPC client with transparent retry + redirect

internal/server        RaftNode driver: ticker, apply loop, Propose, LinearizableGet
internal/transport     gRPC Raft transport (real network)
internal/simnet        in-process network simulator: partition, drop, delay, crash/restart

internal/raft          Raft consensus core — election, replication, snapshots, membership
internal/raftstore     bbolt-backed durable log + HardState + snapshot storage
internal/store         in-memory KV state machine
internal/wal           write-ahead log primitives

internal/chaos         fault-injection chaos harness
internal/lincheck      Wing-Gong-Liu linearizability checker
internal/bench         benchmark harness (log-scale histogram, closed-loop runner)
internal/metrics       Prometheus metrics

**Design decisions

• Pure Raft core with interface seams. internal/raft depends on Storage and Transport interfaces only. The real gRPC transport and the in-process simnet both implement Transport, so the entire chaos suite runs without any network I/O — deterministic, reproducible, and fast.
• Simnet for deterministic testing. internal/simnet wires Raft nodes together in-process with a software network that supports partition, message drop, message delay, and node crash/restart. Every chaos scenario is seeded and fully reproducible.
• ReadIndex for cheap linearizable reads. A Get does not append a log entry. The leader captures commitIndex, sends a heartbeat round to confirm it still holds a quorum, waits for the state machine to apply up to that index, then reads locally. This gives ~10k reads/sec versus ~80 writes/sec on the same loopback cluster.
• Joint-consensus membership. Adding or removing a node goes through a two-phase joint configuration appended as a log entry. The new configuration takes effect on the leader as soon as the entry is appended (not after commit), matching the paper's apply-on-append rule and avoiding the availability window where two separate majorities could exist.

---

Quickstart

# Build and start a local 3-node cluster
./scripts/cluster-up.sh

# Basic operations
./bin/distkvctl put hello world
./bin/distkvctl get hello
./bin/distkvctl status

# Live TUI dashboard (requires a real terminal)
./bin/distkv-tui --endpoints 127.0.0.1:9001,127.0.0.1:9002,127.0.0.1:9003

# Tear down
./scripts/cluster-down.sh --clean

---

Live Web Showcase

make demo-web        # builds bin/distkv-web and serves http://localhost:8080

A single self-contained binary (cmd/distkv-web) runs a 5-node cluster in-process — the real internal/raft consensus core over the internal/simnet software network — and serves a terminal-styled browser console that makes the protocol visible and pokeable. No external cluster, no dependencies; all assets are go:embeded.

What you can watch and do live:

• Consensus in motion — nodes on a ring with role/term, the leader glowing, and animated RPC dots (heartbeats, vote requests, AppendEntries) flying along the edges; a narrated event log ("n3 → Candidate (term 5)", "n1 became Leader").
• Fault injection — kill/restart any node, split the network into partitions, or crank a message-drop slider, and watch elections and recovery happen.
• Key-value ops — put/get/del from the UI; writes replicate and commit across the per-node Raft log columns (committed entries turn green); reads are linearizable via ReadIndex.
• Speed control — slow / normal / fast tick pacing so you can follow a single RPC or stress the cluster.

Kill the leader and watch a new one get elected in well under a second, without losing a committed write.

Healthy cluster	Leader failover	Network partition
Leader glowing, heartbeats in flight, writes committed (green) on all nodes	Kill the leader → a new one is elected in well under a second	Split-brain prevented: only the quorum side makes progress

Architecture note: the consensus code running is identical to the production path; only the transport between nodes is the deterministic in-process simnet (the same harness the chaos suite uses), which is what makes the whole thing a single reliable binary.

---

Raft Design Notes

Leader election

Each server starts with a randomized election timeout in [ElectionTimeoutMin, ElectionTimeoutMax] ticks. If no heartbeat arrives in that window, the server increments its term, converts to Candidate, and sends RequestVote RPCs to all peers. A Candidate wins when it collects votes from a strict majority.

§5.4.1 election restriction: a RequestVote is rejected if the voter's log is more up-to-date than the candidate's (compared first by last log term, then by last log index). This guarantees the elected leader holds all committed entries — without it a new leader could be missing entries that were already committed on a quorum.

Log replication

The leader appends each proposal as a new log entry, then sends AppendEntries to every follower. When a quorum acknowledges (including the leader itself), the entry is committed. If a follower's log diverges, the leader walks back its nextIndex for that follower until it finds the conflict point and sends the missing suffix.

§5.4.2 current-term commit rule: the leader only advances commitIndex for entries from its current term. A leader that takes over after a partial replication by a previous leader must wait until it appends at least one entry in the new term before those earlier entries become safe to commit. Violating this rule would let a majority from a previous term overwrite a majority from an even earlier term that had already been committed.

Snapshots + log compaction

When the applied index advances past a configurable threshold, RaftNode serializes the state machine into a snapshot and calls Storage.SaveSnapshot. Followers that have fallen far behind receive an InstallSnapshot RPC carrying the full snapshot plus the snapshotIndex / snapshotTerm that anchors the truncated log. After installation the follower resumes normal AppendEntries replication.

Joint-consensus membership

Adding or removing a node uses a two-phase joint configuration. The leader appends a ConfChange entry describing the joint config (old ∪ new). As soon as the entry is appended — before it commits — the leader begins using the joint majority rule (a majority of the old config AND a majority of the new config must agree). Once the joint entry commits, a second entry for the final config is appended. This two-phase approach prevents any point in the reconfiguration where two separate majorities could elect two separate leaders.

ReadIndex for linearizable reads

When a client issues a Get, the leader:

1. Records the current commitIndex as the read index.
2. Sends a heartbeat to a majority of peers and waits for their acknowledgements — confirming it has not been deposed by a new leader whose writes the client has not yet seen.
3. Waits until the state machine has applied up to the read index.
4. Reads from the state machine locally.

This avoids appending a log entry for every read, keeping read throughput about two orders of magnitude above write throughput on the same cluster.

---

Testing — How Correctness Is Shown

Unit and property tests

Every Raft sub-system has targeted unit tests. The WAL and Raft log invariants are verified with property-based tests using pgregory.net/rapid: the fuzzer generates arbitrary append/truncate/read sequences and asserts that index-term consistency, durability, and compaction boundaries always hold.

Deterministic in-process chaos suite

internal/chaos builds a cluster of in-process Raft nodes wired through internal/simnet. Each scenario seeds the network simulator with a fixed integer so the fault sequence is 100% reproducible. The suite covers:

Scenario	Fault pattern
FollowerCrash	crash one follower, recover it mid-workload
SymmetricPartition	3|2 split for several seconds, then heal
AsymmetricPartition	4|1 isolation, then heal
RollingRestart	restart every node in turn, one at a time
DropDelayStorm	25% message drop + random jitter for several seconds

Each scenario runs closed-loop clients concurrently with the fault injection, records the full operation history (invocation time + response time + value), and then verifies the history with the WGL linearizability checker.

LeaderCrash is skipped — not because the system fails it, but because of a documented harness recording limitation under idempotent Put retry + concurrent writes. See internal/chaos/scenarios_test.go for the full explanation.

Linearizability verification

internal/lincheck implements the Wing-Gong-Liu algorithm. It takes a recorded history of KV operations with overlapping time intervals and checks whether any sequential ordering of those operations is consistent with the semantics of a linearizable key-value store. Every chaos scenario must pass this check — it is the ultimate correctness criterion.

The headline number

152,563 client operations recorded under 60 seconds of continuous, seeded fault injection on a 5-node in-process cluster (partitions + follower restarts + drop+delay storms), 100% verified linearizable by the WGL checker.

go test -timeout 180s -run TestChaos_LargeRandomized ./internal/chaos/

---

Measured Results

These numbers were measured on a 3-node localhost cluster on this machine.

Hardware / environment:

• Apple M2, Darwin 24.6.0 (arm64)
• Go 1.26.3 darwin/arm64
• 3-node cluster on 127.0.0.1:9001/9002/9003 (all on one machine, loopback only)
• Concurrency: 32 closed-loop goroutines per run
• Key space: 1000 keys, value size: 128 bytes
• Warmup: 2s discarded before measurement window (10s)

Throughput and latency

Workload	Throughput (ops/sec)	P50 (ms)	P95 (ms)	P99 (ms)
write-only	82	418.66	608.99	656.38
read-only	9,868	3.21	5.42	6.79
mixed (70% get / 25% put / 5% CAS)	270	117.10	229.87	287.82

Failover time

820 ms — time from SIGKILL to the leader process until the first successful Put completes on the new leader (measured on the same loopback cluster).

Interpretation

• Writes are serialized through the Raft commit pipeline. Each Put or CAS requires a full Raft round-trip: the leader appends the entry, sends AppendEntries to both followers synchronously, and waits for a quorum ACK before applying. On this single-machine cluster the Raft pipeline commits ~80 writes/sec. This is expected for a correctness-first implementation (50ms tick, 150ms heartbeat interval, synchronous per-follower dispatch). A production system would batch proposals and pipeline AppendEntries asynchronously.
• Reads are fast because linearizable Get uses ReadIndex (leader confirms quorum via heartbeat, then reads locally without a log append), giving ~10k ops/sec at sub-5ms P95.
• Mixed workload latency is dominated by the write fraction (30% of ops) queuing behind the commit pipeline, raising P50 to ~117ms.
• Failover (820ms) covers Raft election timeout (10–20 ticks × 50ms = 500–1000ms randomized) plus the client's leader-discovery retry. No leader-hint persistence across failover; the client retries all known endpoints.

These numbers are from a loopback cluster, not a LAN or WAN. Network latency dominates real deployments.

Reproducing

make bench

This builds the bench binary, starts a 3-node local cluster, runs all three workloads, then tears down cleanly.

---

Observability

Every distkvd node exposes a Prometheus /metrics endpoint (default 127.0.0.1:9101). Tracked metrics include:

• distkv_raft_role — current role as a gauge (0=Follower, 1=Candidate, 2=Leader)
• distkv_raft_term — current Raft term
• distkv_leader_elections_total — count of leader elections won by this node
• distkv_kv_operations_total — KV operation counts by type and status
• distkv_replication_lag_entries — approximate replication lag (entries behind)

A pre-built Grafana dashboard JSON lives at deploy/grafana/. Running docker compose up starts the 3-node cluster, Prometheus, and Grafana with the dashboard provisioned automatically. Navigate to http://localhost:3000 (credentials: admin/admin).

---

Deployment

Docker

Multi-stage, distroless image. The first stage builds with the official Go image; the final stage copies only the distkvd binary into gcr.io/distroless/static.

docker build -t distkv:local .

Kubernetes

Static manifests under deploy/k8s/:

• statefulset.yaml — 3-replica StatefulSet with stable network IDs
• service-headless.yaml — headless Service for stable DNS per pod
• service-client.yaml — ClusterIP Service for client traffic
• configmap.yaml — peer address configuration

kubectl apply -f deploy/k8s/

Helm

helm install distkv deploy/helm/distkv/

License

MIT License — Copyright 2026 Neil Patel.