Site: consensus/Raft docs section + SEO refresh

apps/site/content/docs/architecture.mdx

@@ -24,8 +24,8 @@ result is a complete, smaller system — never a broken half of a larger one.
 ┌──────────────────────────────────────────────────┐
 │  MVCC transaction layer  (Rust)                   │        Phase 3
 ├──────────────────────────────────────────────────┤
-│  Raft consensus  (Rust)                           │        Phase 2
-│  election · replication · read-index · snapshot · membership
+│  Raft consensus  (Rust)                           │        Phase 2 — core shipped
+│  election · replication · read-index ✓ · snapshot · membership ← next
 ├──────────────────────────────────────────────────┤
 │  Custom LSM storage engine  (Rust)                │        Phase 1 — shipped
 │  WAL · memtable · SSTables · leveled compaction · bloom
@@ -49,27 +49,34 @@ through every write so the future MVCC layer can store multiple versions per
 key. Depends on the filesystem only. Full internals: [LSM Storage
 Engine](/docs/lsm-engine).
 
-### 2. Raft consensus (Rust)
+### 2. Raft consensus (Rust) — core shipped
 
 Replicates a log of commands across a group so every replica applies the
 same commands in the same order, tolerating a minority of node failures.
-Leader election with pre-vote, log replication with a commit index,
-read-index for linearizable reads without a log write, log compaction via
-storage-engine snapshots, and joint-consensus membership changes. Depends on
-a dedicated Raft log store (append-oriented, index-keyed — not the LSM KV
-engine; see [Raft-over-Paxos](/docs/decisions/raft-over-paxos) and
+**Built:** leader election with pre-vote, log replication with the
+consistency check and conflict back-up, commit-index advancement under
+Raft's current-term rule, and read-index for linearizable reads without a
+log write — as `RaftCore`, a pure, synchronous, I/O-free step function,
+proven against four safety invariants by a deterministic multi-node
+simulation. **Next:** log compaction via storage-engine snapshots and
+joint-consensus membership changes. Depends on a dedicated Raft log store
+(append-oriented, index-keyed — not the LSM KV engine; see
+[Raft-over-Paxos](/docs/decisions/raft-over-paxos) and
 [log-store vs. LSM reuse](/docs/decisions/lsm-over-btree)) and a pluggable
 transport trait so a deterministic in-memory implementation can drive
-multi-node simulation in tests.
+multi-node simulation in tests. Full internals:
+[Raft Consensus](/docs/consensus).
 
-### 3. Transport / RPC (Rust)
+### 3. Transport / RPC (Rust) — shipped
 
 Node-to-node messaging for Raft (AppendEntries, RequestVote, snapshots) and
-client request routing. Async send/receive of typed messages, pluggable so
-the chaos harness can inject partitions, drops, delays, and reordering.
-Transport is kept deliberately dumb — length-prefixed messages over TCP —
-because the pluggable seam exists for the chaos harness, not for production
-networking features.
+client request routing, behind a `Transport` trait with two
+implementations: a deterministic, seeded in-memory transport for
+simulation (fault injection: partition, drop, delay) and a length-prefixed
+framed TCP transport for a real cluster. Transport is kept deliberately
+dumb by design — the pluggable seam exists for the chaos harness, not for
+production networking features. Wiring it into an async node event loop is
+still ahead. See [Transport](/docs/consensus/transport).
 
 ### 4. MVCC transaction layer (Rust)
 

apps/site/content/docs/consensus/election-and-replication.mdx

@@ -0,0 +1,82 @@
+---
+title: Election and Replication
+description: Pre-vote leader election and log replication with the consistency check and conflict back-up, as a pure step function.
+---
+
+## Roles and time
+
+`RaftCore` tracks four roles — `Follower`, `PreCandidate`, `Candidate`,
+`Leader` — and advances purely on logical time: `tick()` drives every
+timeout, and there is no `Instant`/`SystemTime` anywhere in the core.
+Election timeouts are randomized per node, drawn from a seeded PRNG held
+inside the core, so the same construction seed reproduces the same election
+timing run after run — a requirement for the
+[deterministic simulation](/docs/consensus/safety-and-simulation) to be
+meaningful at all.
+
+## Pre-vote: elections without disruption
+
+A naive Raft election lets any follower that stops hearing from a leader
+increment its term and start soliciting votes. That's a problem for a node
+that's merely partitioned, not actually needed: it keeps incrementing its
+term on every timeout, and the moment it rejoins the cluster, its inflated
+term forces a legitimate, working leader to step down for no real reason.
+
+`RaftCore` runs a **pre-vote** round first: on election timeout, a follower
+sends `RequestVote { pre_vote: true }` *without* bumping its own term. Peers
+answer honestly — would they vote for this candidate, hypothetically — using
+the same up-to-date-log check a real vote uses, but the responder doesn't
+persist anything or mutate `voted_for` for a pre-vote. Only if a *majority*
+grants the pre-vote does the node become a real `Candidate`: increment its
+term, vote for itself, persist that hard state, and broadcast a real
+`RequestVote`. A partitioned node's pre-votes simply never reach a quorum,
+so it never disrupts a healthy cluster. Any message carrying a higher term
+than the node's own — pre-vote or real — steps it down to `Follower` and
+persists the observed term.
+
+## Vote granting
+
+A peer grants a real vote iff the candidate's log is at least as
+up-to-date as its own (compared by `last_log_term`, then `last_log_index`
+as a tiebreak) **and** it hasn't already voted for a different candidate
+this term. `voted_for` is persisted *before* the grant is sent — reversing
+that order would let a crash between "decide to grant" and "persist the
+vote" produce a double vote across a restart, which is exactly what the
+core's restart test pins down.
+
+## Replication and the consistency check
+
+`AppendEntries` carries `prev_log_index`/`prev_log_term` so a follower can
+check that its log agrees with the leader's immediately before the new
+entries. On a mismatch, the follower rejects and returns a `conflict_index`
+hint rather than making the leader retry one index at a time — that hint is
+what lets a badly lagging follower catch up in a handful of round trips
+instead of one per missing entry. On a match, the follower truncates any
+conflicting suffix (`truncate_suffix` — see [Log Store](/docs/consensus/log-store))
+before appending the new entries. Heartbeats are just empty `AppendEntries`
+calls, reusing the exact same consistency-check path as a real replication
+round.
+
+The leader tracks `next_index`/`match_index` per peer and updates them from
+each ack — advancing on success, backing up using `conflict_index` on
+rejection.
+
+## Commit advancement: the current-term rule
+
+The leader advances its commit index to the highest index that's
+replicated on a majority of `match_index` values — but only counting
+entries **that belong to the current term**. This is Raft §5.4.2's
+guardrail against a classic bug: committing an *older*-term entry purely by
+replica count can be undone by a later leader that legitimately overwrites
+it, which would mean a "committed" entry silently disappears. `RaftCore`'s
+simulation pins this down with a dedicated scenario — a leader appends an
+entry, gets partitioned before it commits, and a new leader is elected and
+overwrites the uncommitted entry — asserting the old entry never gets
+counted as committed by count alone. Followers adopt
+`min(leader_commit, last_index)` as their own commit index once they've
+matched the leader's log up to that point.
+
+See [Read-Index](/docs/consensus/read-index) for how commit advancement
+feeds linearizable reads, and
+[Safety Invariants & Simulation](/docs/consensus/safety-and-simulation) for
+how all of this is checked under fault injection.

apps/site/content/docs/consensus/index.mdx

@@ -0,0 +1,73 @@
+---
+title: Overview
+description: The Phase 1 consensus layer — log store, transport, and RaftCore — and what's actually proven about it so far.
+---
+
+## Status: core shipped
+
+Raft consensus is `crates/raft` (`cairn-raft`), built bottom-up in three
+cycles, all merged: a dedicated **log store**, a pluggable **transport**
+layer, and **RaftCore** — a pure, synchronous, I/O-free consensus step
+function covering pre-vote leader election, log replication, commit
+advancement, and read-index linearizable reads. **104 library tests plus 9
+deterministic simulation tests, green.** The simulation drives a multi-node
+cluster through partition, drop, reorder, and crash-restart fault injection
+and checks four safety invariants after every scenario — see
+[Safety Invariants & Simulation](/docs/consensus/safety-and-simulation).
+
+What's left before this backs a live cluster: `InstallSnapshot` handling and
+joint-consensus membership changes, the async node driver that wires
+`RaftCore` to a real clock and the real transport, and the chaos/Jepsen
+harness. See [Roadmap](/docs/roadmap) for what's next and in what order.
+
+## What Raft turns the LSM engine into
+
+The [LSM storage engine](/docs/lsm-engine) is a durable, crash-recoverable
+*local* key-value store — it has no idea any other node exists. Raft
+replicates a log of commands across a fixed set of nodes so every replica
+applies the same commands in the same order, which is what turns that local
+store into a linearizable, fault-tolerant one. See
+[Raft over Paxos](/docs/decisions/raft-over-paxos) for why this project's
+consensus protocol is Raft specifically, and
+[Architecture](/docs/architecture#components) for where it sits in the
+overall stack.
+
+## Module map
+
+| Module | Responsibility |
+| --- | --- |
+| [Log store](/docs/consensus/log-store) | Durable, index-addressed storage for log entries, hard state, and snapshot metadata |
+| [Transport](/docs/consensus/transport) | The `Transport` trait, a deterministic in-memory implementation, and a framed TCP implementation |
+| [Election and replication](/docs/consensus/election-and-replication) | Pre-vote leader election and log replication as a pure step function |
+| [Read-index](/docs/consensus/read-index) | Linearizable reads without a log write |
+| [Safety invariants & simulation](/docs/consensus/safety-and-simulation) | The four properties RaftCore is checked against, and how the deterministic simulation proves them |
+
+## RaftCore's interface
+
+`RaftCore<S: RaftStorage>` is a synchronous struct — no async, no wall
+clock, no filesystem handle. It takes inputs and buffers outputs for a
+driver to send elsewhere:
+
+```rust
+fn tick(&mut self);                                    // advance logical time
+fn step(&mut self, from: NodeId, msg: Message) -> Result<()>; // process one inbound RPC
+fn propose(&mut self, command: Vec<u8>) -> Result<LogIndex>;  // leader appends a command
+fn read_index(&mut self, token: ReadToken);             // register a linearizable read
+
+fn ready(&mut self) -> Ready; // drains { messages, apply, reads }
+```
+
+Keeping persistence behind a synchronous `RaftStorage` trait call (rather
+than an async one) lets the core itself enforce Raft's persist-before-act
+ordering — append then send, persist a vote then reply — and keeps
+simulation fully deterministic, since the storage backing it is
+synchronous too. Phase 1 ships `MemStorage`, an in-memory implementation
+for the core's own tests and simulation; a `RaftLog`-backed adapter that
+plugs the real [log store](/docs/consensus/log-store) into `RaftStorage` is
+part of the node-driver work still ahead.
+
+The state machine this cycle applies committed entries to is a simple
+in-memory KV — wiring the real LSM engine in as the applied state is also
+node-driver work, not part of `RaftCore` itself. Keeping that boundary
+narrow is what let the consensus core be built and proven correct without
+dragging the whole storage engine into every test.

apps/site/content/docs/consensus/log-store.mdx

@@ -0,0 +1,78 @@
+---
+title: Log Store
+description: Durable, index-addressed storage for the Raft log, hard state, and snapshot metadata — the foundation the consensus core is built on.
+---
+
+## Purpose
+
+Raft needs a durable, ordered log of entries indexed by a dense integer, not
+an arbitrary key — and it needs to be able to throw away a suffix of that
+log when a follower's history conflicts with a new leader's. That's a
+different access pattern from a KV store, which is why the log store is its
+own component in `crates/raft` rather than another consumer of the
+[LSM engine](/docs/lsm-engine). Full rationale:
+[LSM over B-tree](/docs/decisions/lsm-over-btree#why-the-raft-log-store-is-not-another-lsm-consumer).
+
+It was also the first thing built in the consensus cycle — everything else
+(transport, `RaftCore`) is testable against it without any networking or
+election logic in the picture.
+
+## What it durably tracks
+
+- **The log itself** — `LogEntry { term, index, command }`, appended in
+  order, addressed by index.
+- **Hard state** — `current_term` and `voted_for`, the two fields Raft
+  requires survive a crash before a node can safely take further action
+  (grant a vote, become a candidate). Persisted atomically and CRC'd.
+- **Snapshot metadata** — the index and term the latest snapshot covers, so
+  the log store knows what a compacted prefix stood for even after the
+  entries themselves are gone.
+
+## Interface
+
+```rust
+fn append(&mut self, entries: &[LogEntry]) -> Result<()>;
+fn entries_from(&self, index: LogIndex) -> Vec<LogEntry>;
+fn entry(&self, index: LogIndex) -> Result<Option<LogEntry>>;
+fn last_index(&self) -> LogIndex;
+fn last_term(&self) -> Term;
+fn truncate_suffix(&mut self, from_index: LogIndex) -> Result<()>; // conflict resolution
+fn compact_prefix(&mut self, up_to_index: LogIndex, meta: SnapshotMeta) -> Result<()>;
+fn save_hard_state(&mut self, hs: &HardState) -> Result<()>;
+fn load_hard_state(&self) -> HardState;
+```
+
+`truncate_suffix` and `compact_prefix` are the two operations that don't map
+cleanly onto an LSM engine's append-only, newest-wins model — a follower
+that discovers its log diverges from the leader's needs to drop everything
+after the conflict point outright, not shadow it with tombstones.
+
+## Durability: CRC records, torn-tail replay, truncate-on-open
+
+The log store reuses the ideas proven in the [WAL](/docs/lsm-engine/wal) —
+CRC-checksummed records and crash-tolerant replay — without inheriting its
+data structure. Each op-log record is length- and checksum-framed; on
+replay, the reader stops cleanly at the first corrupt or short record and
+returns everything valid read so far, the same torn-write discipline the
+storage engine's WAL uses.
+
+One thing the Raft log adds on top of that pattern: **truncate-on-open**. A
+crash can leave a torn tail behind even when nothing was actively being
+truncated — replay has to reconcile the on-disk op-log with the last known
+good state at open time, not just at the moment of a live write, since a
+`truncate_suffix` call itself has to be crash-safe.
+
+Hard state is persisted separately from the op-log, atomically and CRC'd,
+and fsync'd on every change — a node must never be able to lose or corrupt
+its current term or vote across a restart, since replaying a stale vote
+could let it grant two votes in the same term.
+
+## What's still a soft invariant
+
+`truncate_suffix`'s snapshot-boundary check — refusing to truncate at or
+below the last compacted index — is currently a `debug_assert` rather than
+a returned `Error::Corruption`, unlike `append` and `compact_prefix`, which
+were promoted to real errors. It's harmless until snapshotting
+([Roadmap](/docs/roadmap)) actually starts compacting log prefixes in
+practice, but it's the kind of thing that needs to become a hard error
+before this backs a live cluster.

apps/site/content/docs/consensus/meta.json

@@ -0,0 +1,11 @@
+{
+  "title": "Raft Consensus",
+  "pages": [
+    "index",
+    "log-store",
+    "transport",
+    "election-and-replication",
+    "read-index",
+    "safety-and-simulation"
+  ]
+}

apps/site/content/docs/consensus/read-index.mdx

@@ -0,0 +1,63 @@
+---
+title: Read-Index
+description: Serving linearizable reads without a log write, and proving they're actually safe.
+---
+
+## The problem
+
+The simplest way to make a read linearizable in Raft is to route it through
+the log like any other command: propose a no-op, wait for it to commit,
+then read. That's correct, but it means every read pays for a log append
+and a disk fsync — expensive for a read-heavy workload. **Read-index** gets
+the same linearizability guarantee without writing anything to the log.
+
+## How it works
+
+`read_index(token)` registers a read request. The leader:
+
+1. **Snapshots its current `commit_index`** as the read's floor — nothing
+   committed after this point needs to be visible for the read to be valid.
+2. **Confirms it's still the leader** by forcing a heartbeat round and
+   waiting for a majority to acknowledge it — a stale or partitioned former
+   leader must not serve a read that a newer leader has already diverged
+   from.
+3. **Releases the read** — the token is emitted in `Ready.reads` — once
+   `last_applied` has caught up to the floor from step 1.
+
+The leadership-confirmation step doesn't use wall-clock "last heard from
+this peer at time T" bookkeeping, which is fragile under message reordering
+— an ack that arrives late could be mistaken for a fresh one. Instead each
+peer has a send/ack counter pair: the read snapshots the current send count
+per peer as a barrier, forces a broadcast, and only counts an ack toward
+the quorum once a peer's ack count has advanced *past* that snapshotted
+barrier — a pigeonhole argument that the ack corresponds to a message sent
+after the read was registered, not a stale one. That holds under arbitrary
+drop and reorder, which is exactly the condition the
+[transport](/docs/consensus/transport) and
+[simulation](/docs/consensus/safety-and-simulation) are built to exercise.
+
+## The freshly-elected-leader gap
+
+A newly elected leader's `commit_index` may still reflect entries from a
+*previous* term — it hasn't necessarily committed anything in its own term
+yet, since commit advancement only counts current-term entries (see
+[Election and Replication](/docs/consensus/election-and-replication#commit-advancement-the-current-term-rule)).
+Serving a read against that stale commit index would violate
+linearizability. `RaftCore` closes this the standard way: a freshly elected
+leader appends a no-op entry immediately, and won't release any read until
+that no-op has committed — which forces its commit index to be current
+before the first read can be trusted.
+
+## What the tests pin down
+
+Unit tests cover the release conditions directly: the floor is computed
+correctly, a read isn't released before quorum confirmation, a read isn't
+released before applied state catches up, and no read is served across the
+no-op gap on a freshly elected leader. The
+[deterministic simulation](/docs/consensus/safety-and-simulation) is what
+currently exercises this under realistic node behavior (elections,
+partitions, restarts); driving actual *reads* through the same
+arbitrary-reorder fault injection the simulation applies to writes is
+tracked as follow-on work for the chaos harness, not yet built — the
+simulation today asserts safety invariants over applied writes, not read
+outcomes.