RaftKV

A distributed key-value store built from scratch in Go, implementing the Raft consensus algorithm without any external consensus libraries.

Goal

Most general-purpose distributed databases (etcd, Cassandra, CockroachDB) are optimized for steady-state throughput. This project takes a different angle: it's optimized specifically for fast failover, meaning how quickly the cluster recovers when a node crashes. I benchmarked that recovery time directly against etcd under the same conditions.

Demo

failover.mp4

Three real OS processes running in separate terminals. I kill the leader live and a new one gets elected automatically within milliseconds, while the surviving nodes keep serving writes the whole time.

Status: Day 20 of 20, complete

• Leader election, using randomized timeouts, terms, and majority voting
• Log replication with per-follower nextIndex/matchIndex tracking and majority-based commit
• Real TCP transport via net/rpc, with persistent reused connections
• Standalone multi-process binary (--id/--addr/--peers flags)
• KV application layer (Set/Get/Delete backed by the committed log)
• Election Restriction safety fix (see the bugs section below, this one was a big deal)
• Persistence: write-ahead log to disk, atomic writes, survives a crash
• Fast-failover tuning: 100-200ms election timeout, 20ms heartbeat
• Benchmarks vs etcd (see below)
• Final polish: gofmt cleanup, go vet clean, gitignore fixed, code and README swept for leftover merge artifacts

Running it

Local demo, one process, 3 simulated nodes over real TCP loopback:

go run . --demo

Real multi-process mode, run each of these in its own terminal:

go run . --id 1 --addr 127.0.0.1:9001 --peers "2=127.0.0.1:9002,3=127.0.0.1:9003"
go run . --id 2 --addr 127.0.0.1:9002 --peers "1=127.0.0.1:9001,3=127.0.0.1:9003"
go run . --id 3 --addr 127.0.0.1:9003 --peers "1=127.0.0.1:9001,2=127.0.0.1:9002"

Commands you can type at any node's prompt: set <key> <value>, get <key>, del <key>, status, quit. Each node writes its state to raftstate-<id>.json, so if you kill and restart a node it picks its term, votedFor, and log back up from disk.

Benchmark: failover speed vs etcd

I measured both the same way: kill the current leader process, poll until a new leader is confirmed, and record the elapsed wall-clock time. Full reproduction steps are in bench/BENCHMARKING.md, and the scripts themselves are bench/bench_etcd.sh and bench/bench_raftkv.sh.

System	Failover time
RaftKV (tuned, 100-200ms election timeout)	116 to 152ms, averaged about 129ms across 5 runs
etcd (default config)	1011 to 1382ms, averaged about 1200ms across 3 runs

One honest caveat here: this isn't really "RaftKV beats etcd." It's a tuned-for-local-speed setup compared against etcd's deliberately conservative production defaults, which are built to handle real-world unpredictable networks. A fully fair comparison would also involve retuning etcd's own heartbeat-interval and election-timeout flags to match. I'm stating that plainly because that nuance is the actual interesting part, not something worth hiding.

Interview revision notes

The core algorithm, in one paragraph

Raft elects a single leader per term using randomized election timeouts (100-200ms here, tuned down from the standard 150-300ms) so followers don't all start elections at the same moment. Only the leader accepts writes. It appends them to its own log and replicates to followers through AppendEntries RPCs. An entry only counts as committed, meaning safe and durable, once a majority of nodes have it, not all of them. Followers match their log to the leader's using a PrevLogIndex/PrevLogTerm consistency check, and only truncate from the actual point where they disagree. A separate apply step then runs committed entries against an in-memory state machine, which in this case is the KV map. currentTerm, votedFor, and the log all get written to disk on every change, so a node that crashes can safely rejoin later without breaking any of Raft's safety guarantees.

Concepts I should be able to explain out loud

• Why randomized timeouts? They stop every follower from timing out at the same instant and splitting the vote forever.
• Term: a counter that only ever goes up. Any node that sees a higher term than its own immediately steps down to Follower, this is what lets the cluster heal itself after a partition or a stale leader.
• Committed vs proposed: proposed just means it's written to the leader's own log. Committed means a majority has it and it's durable even if that leader crashes right after. An uncommitted entry can actually be lost if its leader dies before replicating it, and that's correct behavior, not a bug (I proved this with chaos testing, see below).
• Why majority and not everyone? So the cluster can keep working through up to (N-1)/2 node failures while staying fully safe.
• Election Restriction: a voter has to refuse a candidate whose log is less up to date than its own, no matter what term the candidate claims. Without this check, a candidate could win an election purely by shouting a higher term number even with stale or empty data (this is the bug story below).
• Why persist before replying to an RPC, not after? If a node says "yes, I voted for you" but crashes before that hits disk, it could restart and vote for someone else in the same term, which breaks the one-vote-per-term rule elections depend on.

Where things live in the code

• node/node.go has all the Raft logic: election (startElection, runElectionTimer), replication (runLeader, HandleAppendEntries), safety (advanceCommitIndex, the Election Restriction check), the state machine (applyCommitted, applyCommand), persistence (persist, LoadNode), and transport (Serve, callPeer, the persistent client cache).
• main.go has two entry points: --demo for the all-in-one local test, and single-node mode (--id/--addr/--peers) which runs as a real separate process with an interactive prompt and real persistence to raftstate-.json.
• bench/ has bench_etcd.sh, bench_raftkv.sh, and BENCHMARKING.md.

Bugs I found while building this (this is genuinely the best interview material)

1. Double-close panic (Day 1). Calling Stop() twice on a node panicked because you can't close an already-closed channel. Fixed with sync.Once.

2. Zombie RPC responses (Day 1). A "killed" node kept answering RequestVote and AppendEntries calls, because Stop() only stopped the node's own loop, not its RPC handlers. Fixed by checking a killed flag at the top of both handlers, which makes a stopped node actually unresponsive, like a real crash would be.

3. Stale state reads on a dead node (Day 1). After Stop(), a node's state field just freezes wherever it last was (so it can still say Leader even though it's dead), since nothing updates it after that. Test code had to explicitly skip killed nodes so it wouldn't try to target a zombie leader.

4. Blind log truncation instead of real conflict detection (Day 4). HandleAppendEntries used to wipe and rewrite a follower's log on every single heartbeat, even when the entries already matched. Fixed it to only truncate from the actual point where terms disagree, which is what the Raft spec actually calls for.

5. TCP connection churn (Day 5-6). Every heartbeat (every 75ms at the time) was dialing a brand new TCP connection to each peer, and I ended up with hundreds of sockets stuck in TIME_WAIT (you can see this with netstat). Fixed by caching one persistent rpc.Client per peer and reusing it, only redialing if it actually fails.

6. The big one: a stale or empty node could steal leadership (Day 9). HandleRequestVote only ever checked term numbers, never how complete a candidate's log actually was. That meant a node that restarted with a totally empty log could keep retrying elections, climb past the real cluster's term, and win a vote purely because its term number was higher, even against a leader that was holding real committed data. I actually reproduced this live: killed a follower, wrote several keys through the real leader, restarted the follower, and watched it become leader with an empty log while another node was sitting on 6 committed entries. That's a genuine data-loss scenario. Fixed it by implementing Raft's Election Restriction properly: a voter now refuses any candidate whose log is less up to date (lower last-log-term, or same term but a shorter log), no matter what term it's claiming.

7. Single-node cluster couldn't elect itself (Day 10-12). The majority-reached check only ran inside the per-peer RPC callbacks, so with zero peers that code path never fired at all, meaning a lone node could never become leader even though its own self-vote already met the majority threshold. Fixed with an explicit check right after the self-vote.

Known limitations, and I'd rather mention these upfront than have someone find them

• Reads aren't linearizable. Get reads whatever's been applied locally, so a lagging follower can return stale data. Real Raft implementations fix this with a read-index protocol, which I've documented in the code but haven't built yet.
• The command encoding is very simple, just a plain "SET key value" string, so keys and values can't contain spaces. A real system would use something structured or binary instead.
• No cluster membership changes. The peer list is fixed at startup, there's no way to add or remove nodes from a running cluster.
• No incremental snapshotting. A really long log gets replayed in full when applying, whereas real Raft implementations compact old entries into snapshots.
• No security at all right now. No TLS between nodes, no authentication, so any process that can reach the ports can vote, propose writes, or read everything. That's fine for a local or LAN demo but not something to expose to the internet as it stands.
• The benchmark isn't fully apples to apples, see the caveat above the results table.

Design notes

• Consensus: core Raft, meaning leader election, log replication, and the safety rules that stop split-brain or losing committed data.
• Transport: real TCP sockets through net/rpc, with persistent per-peer connections.
• State machine: the log is the actual source of truth, the KV map is just a derived cache that could always be rebuilt by replaying the log from index 0.
• Persistence: writes go to a temp file first and then get renamed into place, so a crash mid-write never leaves a corrupted state file behind. currentTerm, votedFor, and the log get persisted; commitIndex and the KV map deliberately don't, since they're rebuilt through replay anyway.

Why I built this

This started as a deep dive into distributed systems fundamentals, consensus, replication, and fault tolerance, built from scratch instead of leaning on a managed database or an existing consensus library.